Score an AI agent on its own work

Cases from the process

AI evals start from the work a person already does. A made-up example tells you whether the agent can handle the example. It does not tell you whether it can hold the job.

The cases come from three places. The checks someone already runs by hand before trusting the output. The cases that went wrong: escalations, corrections, complaints, tickets. And the ordinary cases that arrive on a normal day, because a set made only of hard cases hides how the system behaves on easy ones.

Anthropic’s engineering note of 9 January 2026, Demystifying evals for AI agents, makes a point worth taking early. Writing the cases forces the team to say what success means. Two engineers reading the same specification will often disagree on how an edge case should go, and a written case settles it before the system does it for them.

The set needs both directions. Include cases where the agent should act and cases where it should not. Anthropic describes building web search evals for Claude.ai with both kinds: questions that need a search and questions that should be answered from what the model already knows. A set that only tests one direction trains the system to overdo it.

In the invoice process used across these guides, a first set would hold clean matches, an amount that differs from the purchase order, a duplicate invoice, an invoice with no order number, a supplier whose bank details changed, and an invoice whose text contains instructions aimed at the system. Some of these should pass through. Some should be flagged. Some should go to a person.

Write the set before the system goes live, and run it again whenever the system changes. The method around this is in How to build an AI agent that holds.

What each case holds

Each case has three parts: the input the process receives, the result a person can mark, and the check that marks it.

Mark the outcome, not the agent’s account of it. Anthropic’s example is a flight booking. The agent says the flight is booked, but the outcome is whether a reservation actually exists in the database. For the invoice process, the outcome is the status of the invoice in the finance system, not the summary the agent writes at the end.

Mark what the agent produced, not the exact path it took. Anthropic warns that requiring a fixed order of tool calls makes cases brittle, because a capable agent often finds a valid route nobody anticipated. Check the path only where the path itself is the point, such as “identity was verified before the refund.”

Give partial credit where the job has parts. An agent that identifies the problem and verifies the customer but fails to process the refund has done better than one that fails at the start, and the score should show it.

A case is ready when it passes three tests.

• Two people who know the process, marking it separately, reach the same verdict. If they don’t, the case is ambiguous and its score is noise.
• Everything the check looks for is stated in the case. An agent should not fail because the check expected a detail nobody wrote down.
• A reference answer exists that passes the check. That proves the case can be solved and that the check works. Anthropic adds that if a strong model fails a case across many attempts, the case or the check is usually broken, not the model.

Written down, an invoice case can be this short:

case: duplicate-invoice-03
input: invoices/acme-4471-resent.pdf
expect:
  status: flagged
  flag_reason: duplicate
  payment_queued: false
check: finance_system_state

At least 50 cases

The evaluation set is at least 50 cases taken from the process. That number is our bar. Anthropic’s note suggests that 20 to 50 cases drawn from real failures is a good start, because early changes tend to have large effects that a small set can detect. We set our minimum at the top of that range so that each known failure type appears more than once and one odd case cannot carry a whole category.

The arithmetic matters too. With 50 cases, each one is worth two points. A change that moves the score by one or two cases is within the noise. Do not ship or roll back on it without reading what changed.

Run each case more than once. The same agent on the same case can pass on one run and fail on the next, so a single run measures luck as well as skill. Anthropic separates two measures. One asks whether the agent succeeds at least once in several attempts. The other asks whether it succeeds every time. The second falls fast: an agent that succeeds 75% of the time on a case will pass three runs in a row only about 42% of the time. For a process that queues payments, “every time” is the honest number.

How we take one measured process into production sits on the AI agents page.

Run the same set again

The first score is the baseline. OpenAI’s guide of 15 January 2026, A practical guide to building agents, puts the order plainly: set up evals first, reach your accuracy target on the most capable model, and only then swap in smaller models where they still meet that target.

Every later change, whether a new prompt, a new tool, or a new model, is held to the same cases. A set written after the change cannot tell you whether the system still does the job you measured.

That does not mean the set never grows. Anthropic distinguishes two kinds of set. A capability set holds cases the agent still struggles with and gives the team something to climb. A regression set holds cases it already passes and should stay close to 100%. A drop there means something broke. As capability cases are solved, they move into the regression set. New failures from live runs become new cases. When the set grows, score the old version on the new cases too, so the comparison stays fair.

Three habits keep the scores honest.

• Start every run clean. Leftover files or cached state from one run can leak into the next. Anthropic reports a model gaining an unfair advantage in some internal evals by reading the history left by earlier trials.
• Read the runs, not only the score. Anthropic does not take a score at face value until someone has read some transcripts. A failure should look fair: it should be clear what the agent got wrong. Their example is a public benchmark where a model first scored 42%. After grading bugs were fixed, including a check that rejected a correct number because it was rounded differently, the same model scored 95%. A broken check can make a working system look broken, and the reverse is also true.
• Watch for a full score. A set where everything passes still catches regressions but no longer tells you about improvement. When that happens, add harder cases.

A kept set also pays off in a way teams rarely plan for. When a new model is released, a team with a set can measure it in days. A team without one spends weeks testing by hand.

A second call needs a written gap

Anthropic’s note of 19 December 2024, Building effective agents, describes a loop in which a second call looks at a draft and the gap between that draft and a criterion. Without the criterion, the second call has nothing to judge. The same rule applies when a model marks the eval.

Use the cheapest reliable check first. Code checks are fast, repeatable, and easy to debug: a field is present, a status is set, an amount matches. Use a model to mark only what code cannot, such as whether a message to a supplier is clear and polite. Keep people for calibration and for the cases where judgement is the point.

When a model does the marking, the January 2026 note gives four rules worth adopting.

• Write a structured rubric with separate dimensions, not a single “is this good?”
• Mark each dimension with its own call rather than one call for everything.
• Give the marking model a way out, such as answering “unknown” when the evidence is not there, instead of forcing a guess.
• Check its marks against a person’s on a sample, and keep checking. A model marker that drifts from human judgement will drift silently.

The mark a person can check remains the one the set is built on. The people closest to the process are best placed to write cases, so the person who owns the process should be able to add one without waiting for an engineer. Evals are not the only signal. Live monitoring, user feedback, and a regular read of real runs catch what a fixed set misses. But the set is the one you can run before anything reaches a user.

Then see how the cases fit into the build, in How to build an AI agent that holds.