PERSONAL CODING EVALS
Coding agents evaluated on tasks derived from my day-to-day engineering work. Not a general "best coding model" ranking. Updated 2026-09-27. How this benchmark works → ---
MODELS
Suite core10-v1.3: 10 tasks × 3 runs per config. Harbor 0.21.0; harnesses: Pi 0.85.1, Claude Code 2.1.281, Oh My Pi 18.2.11.
| Harness | Model | Effort | pass@1 | pass@3 | pass-all-3 | $/task | Time/task | Steps | Tokens/task |
|---|---|---|---|---|---|---|---|---|---|
| Pi | GPT-6 Luna | xhigh | 100% | 100% | 100% | $0.0145 | 2:46 | 14.2 | 290k |
| Pi | GPT-6 Sol | medium | 100% | 100% | 100% | $0.0715 | 0:53 | 8.0 | 58k |
| Pi | Opus 5.5 | low | 100% | 100% | 100% | $0.1168 | 0:37 | 3.7 | 31k |
| Pi | Opus 5.5 | medium | 100% | 100% | 100% | $0.2207 | 1:12 | 5.7 | 65k |
| Pi | GPT-6 Astra | low | 100% | 100% | 100% | $0.2618 | 0:55 | 6.5 | 35k |
| Pi | Opus 5.5 | high | 100% | 100% | 100% | $0.3034 | 1:42 | 6.7 | 98k |
| Claude Code | Opus 5.5 | high | 100% | 100% | 100% | $0.3546 | 1:47 | 6.5 | 227k |
| Pi | GPT-6 Luna | medium | 96.7% | 100% | 90% | $0.0061 | 1:02 | 11.0 | 113k |
| Pi | GLM-5.3-Flash | max (default) | 96.7% | 100% | 90% | $0.1999 | 7:01 | 23.2 | 1,196k |
| Pi | Opus 5 | high | 96.7% | 100% | 90% | $0.5161 | 2:26 | 9.7 | 177k |
| Pi | GPT-6 Luna | high | 93.3% | 100% | 90% | $0.0112 | 2:04 | 14.0 | 228k |
| Pi | DeepSeek-V4.1-Flash | off | 93.3% | 100% | 80% | $0.0425 | 4:30 | 16.6 | 426k |
| Oh My Pi | GPT-6 Luna | xhigh | 90% | 90% | 90% | $0.0277 | 3:50 | 17.7 | 776k |
| Pi | Qwen3.8-27B | xhigh | 76.7% | 100% | 40% | $0.2301 | 2:16 | 16.8 | 694k |