HARD12 LEADERBOARD
12 hard, hand-checked coding tasks from Terminal-Bench 4, DeepSWE, SWE-rebench, SWE-bench Pro V2 and Longtail-SWE. Single models and multi-agent setups, 3 runs per task. Updated 2026-11-03. ---
ALL RUNS
pass@1 is the mean solve rate over 12 tasks × 3 runs, with a 95% bootstrap CI over tasks. pass@3: task solved in at least one of 3 runs. pass-all-3: solved in all 3 runs. $/task covers every model in the setup.
| Setup | Type | pass@1 | 95% CI | pass@3 | pass-all-3 | $/task | Time/task |
|---|---|---|---|---|---|---|---|
| Opus 5.5 medium | Single model | 80.6% | 61–97 | 91.7% | 66.7% | $1.20 | 9:18 |
| GPT-6.1 Terra medium (pi) new | Single model | 77.8% | 58–94 | 91.7% | 58.3% | $0.17 | 7:36 |
| GPT-6.1 Sol medium (pi) | Single model | 75% | 53–94 | 83.3% | 66.7% | $0.17 | 11:24 |
| GPT-6.1 Terra high (pi, pre-fix task version) | Single model | 75% | 53–94 | 91.7% | 66.7% | $0.30 | 10:00 |
| GPT-6.1 Sol medium (pi) + Sol self-review + fix | Review | 72.2% | 50–92 | 83.3% | 58.3% | $0.32 | 11:24 |
| GPT-6.1 Sol high (Codex) | Single model | 72.2% | 50–92 | 83.3% | 58.3% | $0.64 | 12:18 |
| GPT-6.1 Sol low (Codex) | Single model | 69.4% | 44–92 | 75% | 58.3% | $0.33 | 6:12 |
| GPT-6.1 Sol medium (Codex) | Single model | 69.4% | 47–89 | 83.3% | 50% | $0.51 | 6:54 |
| Opus 5.5 orchestrator + DeepSeek V4.1 Flash (OpenCode) | Orchestrator | 69.4% | 44–92 | 75% | 58.3% | $1.31 | 17:48 |
| Opus 5.5 orchestrator + Qwen3.8 27B (pi) | Orchestrator | 69.4% | 44–92 | 75% | 58.3% | $1.39 | 21:24 |
| Opus 5.5 + DeepSeek V4.1 Flash 1017, delegation by prompt | Orchestrator | 66.7% | 44–86 | 83.3% | 50% | $0.71 | 12:42 |
| Opus 5.5 orchestrator + DeepSeek V4.1 Flash 1017 (OpenCode) | Orchestrator | 66.7% | 50–83 | 91.7% | 33.3% | $1.29 | 17:06 |
| DeepSeek V4.1 Flash 1017 (OpenCode) + Sol review loop | Review loop | 63.9% | 42–83 | 83.3% | 41.7% | $0.28 | 16:30 |
| GPT-6.1 Sol medium (pi) + Opus cross-review + fix | Review | 63.9% | 42–86 | 83.3% | 50% | $0.51 | 11:24 |
| Opus 5.5 orchestrator + GPT-6 Luna (pi) | Orchestrator | 61.1% | 39–83 | 75% | 41.7% | $1.29 | 18:18 |
| Opus 5.5 low + Sol cross-review + fix | Review | 61.1% | 39–83 | 83.3% | 41.7% | $1.39 | 17:24 |
| Opus 5.5 low | Single model | 58.3% | 36–81 | 75% | 41.7% | $0.58 | 6:36 |
| Opus 5.5 orchestrator + Haiku 5.5 (Claude Code subagent) | Orchestrator | 58.3% | 36–81 | 75% | 41.7% | $1.75 | 25:54 |
| DeepSeek V4.1 Flash 1017 (OpenCode), loop run without review | Review loop | 55.6% | 33–78 | 75% | 33.3% | $0.15 | 16:30 |
| GPT-6 Luna (pi) | Single model | 52.8% | 31–75 | 75% | 33.3% | $0.06 | 12:00 |
| DeepSeek V4.1 Flash 1017 (OpenCode) | Single model | 52.8% | 31–75 | 75% | 33.3% | $0.14 | 9:30 |
| Opus 5.5 low, review run without review | Review | 52.8% | 33–72 | 75% | 25% | $0.58 | 17:24 |
| PAUSE: Opus 5.5 spec + GPT-6 Luna | Team | 52.8% | 28–78 | 66.7% | 41.7% | $0.58 | 20:12 |
| Kimi K3 (pi) new | Single model | 52.8% | 31–75 | 75% | 33.3% | n/a | 11:42 |
| Opus 5.5 low + Opus self-review + fix | Review | 50% | 31–72 | 75% | 25% | $1.71 | 17:24 |
| Advisor: GPT-6 Luna + Opus 5.5 high | Team | 47.2% | 25–72 | 66.7% | 33.3% | $2.14 | 53:18 |
| PAUSE: Opus 5.5 spec + Qwen3.8 27B | Team | 44.4% | 25–67 | 75% | 25% | $0.51 | 22:12 |
| Pipeline: Opus 5.5 medium + GPT-6 Luna | Team | 44.4% | 25–67 | 66.7% | 16.7% | $0.85 | 30:24 |
| DeepSeek V4.1 Flash (OpenCode) | Single model | 41.7% | 19–64 | 58.3% | 16.7% | $0.07 | 15:18 |
| Haiku 5.5 (Claude Code) | Single model | 41.7% | 19–64 | 66.7% | 25% | $0.37 | 12:36 |
| Pipeline: Opus 5.5 high + GPT-6 Luna | Team | 41.7% | 22–64 | 66.7% | 25% | $1.22 | 36:12 |
| Qwen3.8 27B (pi) + Sol review loop | Review loop | 38.9% | 22–58 | 66.7% | 8.3% | $0.07 | 22:24 |
| Advisor: GPT-6 Luna + Opus 5.5 medium | Team | 38.9% | 19–61 | 66.7% | 16.7% | $2.02 | 48:42 |
| GPT-6 Luna (Codex) | Single model | 30.6% | 14–50 | 58.3% | 8.3% | $0.08 | 10:42 |
| Qwen3.8 27B (pi), loop run without review | Review loop | 27.8% | 11–47 | 50% | 8.3% | n/a | 22:24 |
| Qwen3.8 27B (team harness) | Single model | 27.8% | 8–50 | 41.7% | 8.3% | n/a | 14:36 |
| Qwen3.8 27B (pi) | Single model | 19.4% | 6–36 | 33.3% | 0% | n/a | 16:06 |
| DeepSeek V4 Flash 0731 (pi) | Single model | 13.9% | 3–28 | 33.3% | 0% | $0.10 | 12:18 |
| reflection/beam (pi) | Single model | 11.1% | 0–25 | 25% | 0% | n/a | 15:30 |