PERSONAL CODING EVALS
Coding agents evaluated on tasks derived from my day-to-day engineering work. Not a general "best coding model" ranking. Updated 2026-10-17. How this benchmark works → ---
MODELS
Suite core10-v1.3: 10 tasks × 3 runs per config. Harbor 0.21.0; harnesses: Pi 0.85.1, Codex 0.118.0, Claude Code 2.1.281, Oh My Pi 18.2.11, DeepSeek Harness (dsh) 0.2.1-rc.1.
| Harness | Model | Effort | pass@1 | pass@3 | pass-all-3 | $/task | Time/task | Steps | Tokens/task | Cache hit |
|---|---|---|---|---|---|---|---|---|---|---|
| Pi | GPT-6 Luna | xhigh | 100% | 100% | 100% | $0.0145 | 2:46 | 14.2 | 290k | 90.7% |
| Pi | Haiku 5.5 | high | 100% | 100% | 100% | $0.0208 | 2:08 | 9.2 | 287k | 86.9% |
| Pi | DeepSeek-V4.1-Flash-1017 | high | 100% | 100% | 100% | $0.0450 | 2:38 | 15.3 | 547k | 97.9% |
| Pi | GPT-6.1 Sol | medium | 100% | 100% | 100% | $0.0704 | 1:07 | 8.0 | 55k | 82.9% |
| Pi | GPT-6 Sol | medium | 100% | 100% | 100% | $0.0715 | 0:53 | 8.0 | 58k | 84% |
| Pi | Sonnet 5.5 | high | 100% | 100% | 100% | $0.0971 | 0:46 | 4.3 | 45k | 72.7% |
| Pi | Opus 5.5 | low | 100% | 100% | 100% | $0.1168 | 0:37 | 3.7 | 31k | 75.2% |
| Pi | GPT-6.1 Sol | high | 100% | 100% | 100% | $0.1334 | 2:23 | 9.5 | 103k | 85.1% |
| Codex | GPT-6.1 Sol | medium | 100% | 100% | 100% | $0.1375 | 1:37 | 10.4 | 185k | 87.2% |
| Codex | GPT-6.1 Sol | high | 100% | 100% | 100% | $0.1472 | 1:35 | 11.0 | 194k | 87.2% |
| Claude Code | Sonnet 5.5 | high | 100% | 100% | 100% | $0.1473 | 0:52 | 4.6 | 196k | 92% |
| Codex | GPT-6 Sol | medium | 100% | 100% | 100% | $0.1612 | 1:43 | 13.0 | 238k | 89.9% |
| Pi | Opus 5.5 | medium | 100% | 100% | 100% | $0.2207 | 1:12 | 5.7 | 65k | 79.7% |
| Pi | GPT-6 Astra | low | 100% | 100% | 100% | $0.2618 | 0:55 | 6.5 | 35k | 78.7% |
| Pi | Opus 5.5 | high | 100% | 100% | 100% | $0.3034 | 1:42 | 6.7 | 98k | 82.2% |
| Claude Code | Opus 5.5 | high | 100% | 100% | 100% | $0.3546 | 1:47 | 6.5 | 227k | 91.6% |
| Pi | GPT-6 Luna | medium | 96.7% | 100% | 90% | $0.0061 | 1:02 | 11.0 | 113k | 88.1% |
| DeepSeek Harness | DeepSeek-V4.1-Flash-1017 | low | 96.7% | 100% | 90% | $0.0326 | 1:53 | 14.6 | 424k | 96.3% |
| Pi | GLM-5.3-Flash | max | 96.7% | 100% | 90% | $0.1999 | 7:01 | 23.2 | 1,196k | 0% |
| Pi | GPT-5.6 Sol | medium | 96.7% | 100% | 90% | $0.2017 | 1:24 | 7.9 | 69k | 82% |
| Pi | Opus 5 | high | 96.7% | 100% | 90% | $0.5161 | 2:26 | 9.7 | 177k | 88.7% |
| Pi | Haiku 5.5 | low | 93.3% | 100% | 90% | $0.0069 | 0:42 | 5.4 | 71k | 78.9% |
| Pi | GPT-6 Luna | high | 93.3% | 100% | 90% | $0.0112 | 2:04 | 14.0 | 228k | 90.6% |
| Pi | DeepSeek-V4.1-Flash | max | 93.3% | 100% | 80% | $0.0425 | 4:30 | 16.6 | 426k | 90.8% |
| Pi | GPT-5.6 Luna | medium | 90% | 90% | 90% | $0.0095 | 1:01 | 9.2 | 72k | 85.7% |
| Pi | Haiku 5.5 | medium | 90% | 100% | 80% | $0.0098 | 1:00 | 5.9 | 96k | 77.5% |
| Pi | DeepSeek-V4.1-Flash-1017 | low | 90% | 90% | 90% | $0.0257 | 1:31 | 11.0 | 242k | 96.3% |
| Oh My Pi | GPT-6 Luna | xhigh | 90% | 90% | 90% | $0.0277 | 3:50 | 17.7 | 776k | 93% |
| Pi | Haiku 5.5 | xhigh | 90% | 100% | 80% | $0.0647 | 4:28 | 14.0 | 830k | 92.1% |
| Codex | GPT-5.6 Sol | medium | 90% | 90% | 90% | $0.4415 | 2:03 | 13.6 | 310k | 89.8% |
| Pi | GPT-5.6 Terra | medium | 86.7% | 100% | 70% | $0.0938 | 1:20 | 7.6 | 59k | 81.9% |
| Pi | Sonnet 5 | high | 83.3% | 90% | 70% | $0.2910 | 2:52 | 12.7 | 309k | 91.7% |
| Claude Code | Sonnet 5 | high | 83.3% | 90% | 80% | $0.3957 | 3:11 | 10.8 | 628k | 95.1% |
| Pi | Qwen3.8-27B | xhigh | 76.7% | 100% | 40% | $0.2301 | 2:16 | 16.8 | 694k | 97.3% |