I spend my week on both halves of this. I build services that a model drives, including a code generator whose loop hands proposed Go files to a real compiler and feeds the result back as a tool message. And I spend most days driving coding agents against other people’s tools over MCP.
Writing the harness taught me something that surprised me: the hardest part of grading an agent is not deciding whether the answer is right. It is proving that the answer came from the work you were trying to measure.
A verifier answers one question, and usually not the one you asked
My compile gate is a good verifier. It takes the files the model proposed, builds them for real, runs them, and reports what happened. There is no mocking and no judgment call. If it says the program runs, the program runs.
It also cannot tell me a single thing about whether the model reasoned well. A model that guessed correctly and a model that worked the problem produce the same green result. The gate answers “does this artifact behave”, and I kept quietly reading it as “did this model do the job”.
Those are different questions, and the gap between them is where tool-use evaluation lives. If the task is “use these MCP servers to find out what you need, then fix the bug”, then a test suite over the final diff measures the fix. The finding is the part you cared about, and nothing in the assertion looked at it.
If the answer is in the weights, you measured memory
This is the failure that costs the most and shows up the least.
Build a task around a well-known library and a well-known bug, hand the agent an MCP server that can look up the answer, and a capable model will skip the server entirely. It already knows. Your environment reports a pass, you record it as successful tool use, and you have just taught a training signal that the tools are optional.
The fix is not a better prompt telling it to use the tools. The fix is to make the information genuinely unavailable anywhere else:
- Synthesise the facts. Invented table names, invented column semantics, an invented migration history. Nothing the model could have read during pretraining.
- Put the discriminating detail only behind a call. Not “which function formats dates” but “which column this deployment made non-nullable in v4”. A fact with no existence outside your fixture.
- Make a plausible guess wrong. If the obvious assumption happens to be correct, a run that assumed passes, and you cannot separate it from a run that checked.
That last one is the real test of a task. Write down what a competent engineer would assume without looking, then make the truth something else. Now a pass is evidence.
Determinism is a property of the environment, not the assertion
People hear “deterministic verification” and write a careful assertion. The assertion was rarely the problem. The environment around it was.
A verdict that changes between runs on identical input is not a slightly noisy verdict. It is a different kind of object. A grader that is right most of the time still emits a reward signal with structure in it, and that structure has nothing to do with the submission. Whatever consumes the signal will happily fit it.
The things that moved on me, in rough order of how long each took to find: wall clock read from the host, map iteration order, a dependency resolving to a new patch release, and two fixtures racing for the same port. Every one of them produced a task that “sometimes failed” and got quietly retried instead of fixed.
So the clock gets injected rather than read, every iteration order gets sorted before it reaches an assertion, dependencies are pinned by digest and not by range, and ports and temp paths are unique per run. None of it is clever. All of it is the difference between a verdict and a coin flip.
There is a corollary worth stating plainly. Before you trust a task, run it against an unchanged submission several times and confirm the verdict does not move. A task that has never been checked for this is a task whose verdict you do not know.
A golden solution is a claim about reachability
The reference solution looks like paperwork. It is actually the only proof that the task is well formed, and writing it is how you find out it is not.
Two things surface reliably when you sit down and solve your own task using only the tools you provided:
The task is impossible. Some fact the solution depends on is not exposed by any tool you shipped. In your head the agent “just looks it up”, and there is no call that returns it.
The task is bypassable. You find a shortcut that reaches a passing verdict without doing the intended work, which means the strongest submissions will find it too.
Both are cheap to fix at that moment and expensive to fix after the task has produced data. I would rather spend an hour solving my own task than ship a hundred of them and discover the shortcut in the results.
The two ways to be wrong do not cost the same
A false fail is loud. Someone reads the trace, sees a correct solution marked wrong, and files it. Annoying, bounded, self-reporting.
A false pass is silent. It is indistinguishable from a real pass, and it is worse than a missing sample because it actively points the wrong way. Every downstream step inherits it, and nothing in the data marks which rows are which.
That asymmetry decides where to spend effort. When I am unsure whether the verifier is checking the right thing, the safe direction is stricter, because a task that fails a correct answer gets found and a task that passes a wrong one does not.
What the harness owes the run
I wrote the loop side of this up separately, including why the compile gate has no retry cap while the run gate stops at five attempts: https://tech.indra.codes/posts/the-model-wrote-the-code-the-build-gate-was-the-product/
Three details from it belong here, because they are all the same rule pointed at different things.
Every tool result carries the model’s own call ID. The tool message echoes tool_call_id from the request that asked for it. Correlating on anything I generated myself breaks the moment two calls come back in one turn.
An unknown tool name is a hard error, not a shrug. If the model asks for a function I do not have, the loop stops and puts the name in the message. Silently dropping the call leaves the model waiting for a result that is never coming, and it will spend its remaining turns working out why.
A truncated answer is refused, not returned:
if choice.FinishReason == "length" {
return parsed, fmt.Errorf("AI response truncated by max_tokens")
}When a model runs out of output budget mid-answer, what comes back still parses. The envelope is there, the fields are there, and one file’s content simply stops. Falling through would hand my caller something indistinguishable from a complete answer.
There is a cap on the loop as a whole too, twenty five rounds, and it exists for a different reason: not correctness, but a model that will not stop calling tools. A loop without an exit is not autonomous, it is a hang.
All three are the same commitment, which is the one an evaluation environment lives or dies on. Never emit a result that cannot be told apart from a real one. A silently truncated answer, a dropped tool call, a verdict that flickered: each is a lie the consumer has no way to detect.
What I would tell someone building these
Grade the path, not only the artifact. If the environment exists to measure tool use, then the trace is part of the submission. A task whose only signal is the final diff cannot distinguish retrieval from recall.
Make the discriminating fact unguessable. Write down what a competent engineer assumes without looking, then make the truth something else. Otherwise a lucky prior scores the same as real work.
Run the same submission until you trust the verdict. Determinism is not a property you assert, it is one you observe. Clock, ordering, dependency resolution, and shared ports, in that order.
Solve your own task with only the tools you shipped. It is the only way to learn that the task is unsolvable, or that it has a shortcut, while both are still cheap.
When in doubt, fail. A false fail gets reported by the person reading it. A false pass gets reported by nobody and propagates into everything built on top of it.