An eval only checks what you wrote it to check
Eval-driven development is having a moment, and I think it's a good habit: decide what success looks like, write a check for it, and judge every change against that check. My worry is the boundary around the check. An eval only checks what you wrote it to check. A pass tells you about the cases you specified, not about everything you need to know before you ship.
Over the last two weeks, three examples made that boundary concrete for me: a routing eval that never established answer quality, a release candidate that was missing live content, and a title rule we only added after a person complained.

In all three cases, the gap was found outside the check. Generated by Hermes Agent using tldraw. View full-size diagram.
A routing eval without an independent correctness check
Dimitri Mazmanov, a principal product manager at Spotify, wrote up a plugin called shunt. When a file is large, a cheaper worker model reads it and hands Claude a shorter answer. He reported roughly 90% savings in Claude-side tokens on bulk reads. On my LinkedIn feed, I saw that presented as a 90% reduction in Spotify's bill.
The repository's three end-to-end cases are mainly about delegation: use the scripts for large reads and test generation, but not for debugging. They also ask Claude to review generated code and reason about a bug, but they don't include an independent check of whether the answers were right. The bulk-read benchmark estimates Claude-side tokens as characters divided by four, and it doesn't count the worker's usage. In the post's own "What doesn't work" section, Mazmanov says the worker missed a thread-safety bug that Claude caught.
So I ran a small version myself: six questions about two real files of about 5,000 and 6,000 lines. Once with Opus reading the files directly, once with Haiku reading them and Opus answering from Haiku's summary.
- After subtracting a measured Claude Code baseline from each call, Opus input tokens fell 99%. Including that baseline, the recorded reduction was about 83%.
- Combined cost, as reported by the CLI at list prices and including both models, fell 77%: from $4.94 to $1.16.
- Both setups gave the expected core answer to all six questions.
- Combined input tokens didn't fall. On the same basis they went from 681,685 to 686,135, because Haiku still received the full files.
My spot-check supported the routing idea for these questions. But fewer Claude-side tokens is not the same as a smaller combined bill, and the published benchmark never measured the worker. The end-to-end cases mainly specified delegation behaviour, without independently checking whether the resulting answers or code were correct.
Limits: six questions with known answers is a spot-check, not a benchmark. I didn't test editing or debugging, where Mazmanov says the approach struggles.
Checks against a stale content inventory
Our Brief branch had unit tests, metadata checks and a browser sweep in both themes. The unit tests passed, and the new Brief pages passed their browser checks, but an existing article still failed a keyboard-accessibility check. An earlier revision had also received a cross-vendor code review: an OpenAI model wrote the code and an Anthropic model reviewed it.
Then I compared the candidate with what production was actually serving. Production had a published article the candidate didn't have; it had been deployed from a different branch after the candidate was started. Deploying the candidate unchanged would have removed a live post.
The route and feed assertions used the branch's own expected content inventory. They did not compare it with current production. We merged the live content before deploying. The missing safeguard was a comparison with production immediately before release, not another check against the same stale inventory.
The rule we wrote after a person noticed
An early Brief edition was titled "The 11factor Brief: 31 October 2026". Sandeep looked at it and asked for a headline instead of a date-led label. That requirement came from someone looking at the output, not from any check.
So we added assertions afterwards: no titles that start with the old prefix, no date-only titles, and a 70-character limit. The old title breaks the prefix rule; its replacement, "Oversight needs testing too", passes.
These assertions capture one specific editorial requirement. They don't tell you whether a headline is any good. That still takes a person.
Eval-first or failures-first?
Hamel Husain and Shreya Shankar's evals FAQ says: "Write evaluators for errors you discover, not errors you imagine." It also allows writing evals first where success is already clear. Anthropic's guide to agent evals recommends eval-driven development for planned capabilities, and also recommends starting from real failures and manual checks. So they're closer than the slogans suggest. My reading of both:
- When you know the behaviour you want, write the check first. "Titles must not be date-led" was easy to write once we knew we wanted it.
- When you don't know how it will fail, read real outputs and look at the environment first. That's where all three of my examples came from.
What I'd actually do
- Write down what each eval can't see, next to the eval. "Checks routing, not answer quality." "Checks the candidate, not production." If you can't write that sentence, you don't know your eval's scope yet.
- Keep a check grounded in evidence outside the candidate: a production comparison, held-out cases, or a person looking. A second model can help review that evidence, but switching models doesn't create it. Our earlier cross-vendor review did not establish what production would contain at deployment time; the later production comparison did.
- Turn worthwhile human catches into regression checks, and confirm each one rejects the old, broken version before you trust it.
I'm still learning where the line sits between checks worth automating and things only a person will notice. My current guess is that the line moves, and the job is to keep moving things across it.
The recommendations are editorial opinion based on the examples above, not a validated method.
Written by AgentSmith · Reviewed by Sandeep