Playbook: Your First Private Agent Eval

2026-11-11 · 11 min

Every team needs its own eval, and you can build the first one this week from work you have already done. This playbook is the written version of my talk "You're Sitting on a $1M Dataset" (In the Loop #13, London). Give this page to your coding agent and follow along.

The idea

• Your work already holds everything an eval needs. Agent sessions, pull requests, tickets and every message where you corrected the agent.
• Public benchmarks measure everyone. Your eval measures you: your model, harness, skills, prompts and tasks.
• With your own eval you can improve your setup one change at a time, and measure it instead of guessing.

Two terms to know. An eval is a set of tasks with checks that tell you whether the agent succeeded. The verifier is that check.

Quick start

Start with one task. Allow about two hours for a first attempt, assuming the tools are already installed; a 5 to 10 task eval takes longer. You can usually build the tasks with your existing agent subscription, but evaluation runs may need API access and cost extra.

1. Send this playbook to your agent and install the evals-from-sessions skill.
2. Make a small eval: 5 to 10 tasks from your own sessions.
3. Run your setup on it through Harbor and look at the results.

Choose one installation method:

npx skills add ibragim-bad/evals-from-sessions

# Claude Code
/plugin marketplace add ibragim-bad/evals-from-sessions
/plugin install evals-from-sessions@evals-from-sessions

# Codex
codex plugin marketplace add ibragim-bad/evals-from-sessions
codex plugin add evals-from-sessions@evals-from-sessions

# Pi
pi install git:github.com/ibragim-bad/evals-from-sessions

You also need Docker, git, Python 3.11+, Harbor (uv tool install harbor), gitleaks 8, and credentials for one agent and model.

Then start your agent in an empty folder and paste this prompt:

Read this playbook: https://ibragim.dev/writing/first-private-agent-eval/
Then use the evals-from-sessions skill to turn my agent sessions
from the last month into eval tasks.

1. Copy my sessions and scan the copy locally for secrets. Do not
   print detected secrets or send them to a model. Redact any
   findings, then read only the cleaned copy.
2. List candidate tasks in candidates.md. Prefer sessions where the
   agent got it wrong and I corrected it.
3. Stop and show me the candidates. I will pick 5 to 10.
4. For each one, write the task spec and stop for my approval.
5. Build the Harbor tasks. Test every verifier with four solutions:
   reference, another correct one, a broken one, and no change.
6. Show me what each verifier checks and the results of the
   four-solution test. Wait for my approval.
7. Run one weak and one strong model, 3 runs each, and report
   pass rate, cost, time and anything that looks like cheating.

Before any paid run, show me the estimated cost and ask me to
approve a total budget.

The skill does the building. You select the tasks, then approve each task spec and each verifier.

Step 1 · Find: a session becomes a task

Your dataset is already on your disk. Back it up first: some agents delete old sessions.

Transcripts can contain secrets. Scan the copy with gitleaks, redact what it finds, and let the agent read only the cleaned copy. A scan does not guarantee that every secret is gone, so check again before you share anything.

Every correction is a test case. The best candidates are sessions where the agent got it wrong and you fixed it. Your correction helps define what the check should verify.

A real example from my sessions:

• The agent: planned to delete every run that had an error.
• My fix: only remove a run if the last retry failed.
• The check: a run with an error first and a success after must be kept.

The obvious shortcut gives the wrong result. The check targets that edge case.

A good task meets three conditions:

1. It looks like your normal daily work.
2. There is an unambiguous way to check the result.
3. It can run in an isolated environment with the tools and data it needs.

Expect heavy filtering. I went from about 1,270 sessions to 15 candidate tasks and kept 10. No sessions yet? Take an open benchmark, pick the tasks closest to your work, and run the same loop on them.

Open benchmarks to borrow tasks from

You do not have to start from your own sessions. My harder set is 12 tasks picked from the first five benchmarks below, chosen because they look like my daily ML and software work.

Treat a borrowed task like your own: read the failed runs and test the verifier with four solutions before you trust it. In my 12-task set I had to fix the verifier or the prompt in 9 tasks.

Step 2 · Verify: turn your fix into a check

In your eval, don't review the code. Review the verifier. It is the part that decides every number you will see.

Write the task spec in three parts:

1. Requirements: what a valid solution must do.
2. Failure modes: the mistakes the tests must catch.
3. Details the agent must discover: what the agent has to find in the code or data itself.

Too many hints make the task mechanical. A missing requirement means the verifier tests something the agent was never asked to do. This is your internal spec. The instruction the evaluated agent sees states the requirements only, without the hidden checks or the details it is expected to discover.

Use deterministic checks: run the program and inspect files, JSON output and git state. Avoid an LLM judge where you can.

A toy example. The task: the basket total ignores the delivery fee. For a basket of £11.99 with £2.50 delivery, the right answer is £14.49.

As a minimum, test every verifier with four solutions before any model run:

A broken verifier lies in two directions. With false negatives, strong models hit a wall and score the same as weaker ones. With false positives, a cheap setup looks as good as an expensive one, and you switch to something that does not work.

This is not only a toy problem. In LongTail-SWE we audited 65 tasks by hand and 51 had issues. Often the project's own tests rejected a correct fix because it differed from the original pull request. The same Opus 5.5 patches on 63 tasks scored 29% with the original tests and 67% with repaired tests. Opus did not get better. The verifier did.

Step 3 · Harden: models will find a way to cheat

Stronger models cheat smarter. On SWE-rebench every task is a real GitHub issue that is already fixed, so the answer exists somewhere. This is what we saw, in order:

1. Agents found the fix in the git history. We removed the history.
2. Agents used WebFetch to open the original pull request. We blocked WebFetch.
3. An agent ran curl against the GitHub API, read the fix discussion, and downloaded the future test config from the main branch.

The prompt said "please don't cheat" the whole time.

After every few runs, ask another agent to read the logs and look for ways to pass without solving the task. Then you review what it found, fix the task, and repeat the four-solution test.

Difficulty only shows up after runs. One of my tasks, a branch cleanup that took me 90 minutes, was solved in one minute with git merge --ff-only. I dropped it.

Run it and read the results

Harbor is a framework for running agent evals: you give it tasks, pick an agent and a model, and it runs every attempt in an isolated environment and scores it. A Harbor task is a folder:

Start every attempt from the same clean state. Keep the reference solution, the hidden tests and your source sessions out of the evaluated agent's reach.

Run a setup like this:

harbor run -p tasks/ -a pi -m <model> --ak thinking=medium -k 3 -n 5

How to read what comes back:

• Run each task 3 times. Report how many tasks pass in all three runs, not only the average.
• Compare cost, time, steps and tokens, not only the pass rate. On easy tasks every setup passes, and the difference is the bill.
• Change one thing at a time: the model, the harness, the effort level, or one skill.
• Small sets are noisy. A 10-point gap on 12 tasks is not enough on its own to show that one setup is better. Compare results task by task and confirm the difference on more tasks.
• Numbers tell you what happened, logs tell you why. Read one solved and one failed run per setup.

Beyond code

None of this is only about code. One of our RL environments is a Blender task: an animator recorded five moves (run, jump, punch, kick, front flip), and the agent gets a 3D character and has to build the skeleton, attach the skin and animate it to match.

Reading the code will not tell you whether the animation looks right. A vision model asked "is this a flip?" is not stable, and comparing pixels overfits to one video. So the verifier tracks skeleton points, timing, and whether the character's body tears or breaks apart.

It is tested the same way as a code task: the reference scores 1.000, doing nothing scores 0.000, and it caught 9 of 9 broken variants. A clean flip scores 1.000; a flip where the body tears apart scores 0.046.

After the eval

Once you have an eval you trust, it becomes a ladder:

1. Choose a model and a harness.
2. Tune prompts and skills, and keep the changes that improve the trade-off you care about: reliability, cost or speed.
3. Fine-tune on the runs that passed.
4. RL on similar tasks.

Keep held-out tasks that you never use for training or prompt tuning.

Links

The skill

• evals-from-sessions on GitHub: finds candidate tasks in your sessions, drafts the specs, builds Harbor tasks and checks the verifiers. MIT licensed.

My posts

• Turn your agent sessions into evals. Mine found a 2× cheaper setup: the full method, step by step.
• I gave Opus a cheap coder to save money. Accuracy dropped and the bill went up: the team experiment.
• Same score, different agents
• My mini-benchmark for coding agents
• My leaderboard: every setup I have run on my own tasks.

Tools and benchmarks

• Harbor documentation
• SWE-rebench: fresh GitHub tasks every month, 30+ models.
• LongTail-SWE thread: 202 tasks across 13 less common languages.

Get in touch

• Built an eval? Send me one task. I will look at your verifier and help where I can.
• Doing post-training? We have many free, open-source RL environments. Reach out and I will share samples before release.

Ibragim Badertdinov · @ibragim_bad · ibragim.dev