Claude evals: designing a measurement the exam would accept

Claude evals are how you show a prompt got better rather than asserting it. The exams test the design of that measurement: a metric attached to a decision, a set that covers the cases that actually cost you, one variable changed at a time against a recorded baseline, and a bar fixed before you look at the result. A bar chosen afterwards is not a test at all: it records the outcome, and it passes every time.

Where claude evals sit across the four exams

The weighting varies more here than for any other subject, so check your own track before deciding how long to spend. On Architect Professional, Evaluation, Testing and Optimization is 16% — around ten of sixty-three questions, and the second-heaviest domain on that paper. Architect Foundations folds validation, retry and feedback loops into a 20% domain alongside prompt design.

On Developer Foundations, the Eval, Testing and Debugging domain is 2.6%, which works out to a single scored item. It is worth reading once and no more. The full weights for all four tracks put that in context.

A metric that decides nothing is not a metric

A number earns its place by changing what you do. Attach it to a decision and to a value set in advance, so it can be passed or failed rather than admired. Distractors in this area are frequently numbers that are real, easy to collect and connected to no decision at all — average response length, total requests served, a satisfaction score with nothing hanging on it.

The dimensions also pull against each other instead of stacking up. You cannot push accuracy, speed, spend, safety and security all upward together, and a design that claims to has usually measured none of them. Saying which one you sacrificed is part of the answer.

Representativeness beats size

An eval can be no better than the cases behind it, and what usually goes wrong is coverage rather than count. Sample from ordinary traffic and the easy majority swamps everything else, so a system posts a good number while still falling over on the cases that actually cost you money or reputation.

Build by category instead: the typical case, the edge case, the adversarial input, the one with a field missing, the policy-sensitive one. A category you leave out cannot be reported on, and that silence reads as a pass. Report by segment too — an aggregate hides the exact segment you needed to see, which is how a quarter of slow decline goes unnoticed while the headline number holds.

Three ways to grade, and when each fits

The examinable judgment is matching the method to the property. Anything a program can decide should be decided by a program — it is cheaper, exact, and never drifts. Reserve model-based grading for judgments where an exact match is meaningless, and calibrate it against labelled examples before its verdicts are allowed to gate anything.

One variable, one baseline, one bar set in advance

The loop is short and the discipline is the whole point. Record a baseline on the set before changing anything. Change one thing. Re-measure on that same set, and not on the case that provoked the change, which will pass because you aimed at it. Keep or revert against the bar you fixed beforehand, and note which variable moved the number.

Two variants of the same error are worth recognising. Adjusting a prompt after one bad output and checking it on that same output establishes almost nothing about anything else. And growing the evaluation set between rounds moves the instrument and the system together, so any difference could have come from either.

Knowing when to stop is part of it. When a round shifts the number by less than the noise in your own measurement, you cannot tell the shift from the noise, and further rounds spend effort on movements nobody can attribute.

Three causes that look identical from the outside

When output is wrong, three quite different things may have happened, and they have disjoint remedies. Telling them apart before acting is what the items reward.

A heavier configuration compensating for an unstated requirement is the classic version: a summariser keeps omitting a disclosure, a more capable tier reduces the omissions without removing them, and the prompt never listed the disclosure among the required elements. The cost of that misdiagnosis is permanent, and the actual fault took a sentence to fix.

The same care applies to optimisation. Fix the minimum acceptable quality first, then reduce spend and latency subject to it — never lower the floor once a cheaper result is in front of you. Count the cost end to end as well: halving per-request spend while doubling reviewer time has moved the cost, not removed it. On the Professional track the same diagnostic habit covers retrieval — see which layer of a pipeline broke. For the reliability half of this material, see context management, and for judging a single output rather than a system, how to evaluate Claude output. Official outlines are published on the Anthropic Partner Academy certification pages.

Key takeaways

• Check your track first. This is 16% of Architect Professional and 2.6% of Developer Foundations — ten questions against one.
• Fix the bar before you measure. A threshold set afterwards describes the result rather than testing it, and it is always met.
• Tie every metric to a decision. If nothing happens differently depending on the number, collecting it is not measurement.
• Build the set by category, not by sampling traffic. A category you omit cannot be reported on, and the silence reads as a pass.
• Let code decide anything code can decide. Reserve model-based grading for judgment, and calibrate it before it gates anything.
• Change one thing at a time. An improvement you cannot attribute is one you cannot repeat, and you will carry the useless parts forward.