My mini-benchmark for coding agents

2026-09-26 · 2 min · eval

I'm Ibragim, and I work on coding agents, evaluations, and training datasets at Nebius. I use agents every day and want to understand which models and settings solve my tasks faster and cheaper while keeping the same quality. So I built a small personal benchmark from my everyday work.

I took around three months of agent chats, removed duplicates, and selected 10 tasks: trajectory analysis, map-reduce data processing, simple inference, backend, frontend, and deployment in Python and TypeScript. These are clearly defined tasks that would usually take me less than an hour. I converted them into Harbor tasks and added automated checks, which I refined with agents and manual review.

A simplified task example — import-jsonl-into-local-table-store v1.1:

• Task: import JSONL into a local disk-based table store, validate records against a typed schema, and report rejected lines.
• Expected result: a canonical data.jsonl and a schema.json with the row count and SHA-256 hash. In append mode, identical records are skipped and conflicting IDs are rejected. In replace mode, the new content is published atomically, only after the entire input has been parsed.
• Verification: around 20 generated scenarios, including conflicts, empty input, and a corrupted existing table. The verifier compares the exact data bytes, schema, and rejection report. It also checks that replacement is atomic and no third-party modules are used.

I run the evaluations through Harbor. The main model comparison uses Pi, and I have also tested Codex and Oh My Pi. Models are accessed through APIs. Each configuration runs three times on every task: 30 attempts per configuration. I track success rate, cost, time, steps, and token usage.

Results are on the leaderboard.

This benchmark helps me choose models and test changes to the agent, prompts, and plugins. Several configurations already pass every check, so comparing their cost and speed is especially useful. Still, 10 tasks is a small sample, and the same pass rate does not mean the same code quality. The tasks remain private for now. I'll gradually expand the task set, add more models and agent setups, and keep the results updated.

If you have a specific task where a model fails, email me at admin@ibragim.dev. These examples are very useful for evaluating models.