Evaluation and testing on CCAR-P: five metric families, not one

Evaluation and testing on CCAR-P is a 16% domain — roughly 10 of 63 items, second only to Integration at 19% — spread across six objectives. The first of them is the one worth reading twice: it names accuracy, latency, cost, safety and security as the metrics an evaluation defines. Five families, and most evaluation work in practice measures the first.

Sixteen percent, six objectives

Two of the six are covered elsewhere. The diagnosis objective is the CCAR-P face of a skill every track examines, in troubleshooting on the Claude exams, and the underlying measurement discipline — a baseline, one changed variable, a threshold fixed beforehand — is in writing evals for Claude prompts. The other four are this domain’s own.

Five metric families, not one

The gap between what the objective names and what suites typically contain is where these items are written. An option proposing a thorough accuracy evaluation is correct about a fifth of the objective and reads as the diligent answer, which is exactly why it is offered.

Safety and security as measured properties are the least familiar of the five. Measuring them means the suite contains cases designed to fail — inputs that should be refused, content that should not be acted on, a boundary that should hold — and records whether they did. A suite made only of cases that should succeed cannot tell you anything about the ones that should not.

Mixed methodologies

The dataset objective asks for test frameworks using mixed methodologies, and the word doing the work is mixed. Different things being measured need different graders: a deterministic check where a right answer exists and can be compared, a model-graded judgment where correctness is a matter of degree, and human review where the consequence of being wrong justifies the cost.

The failure is applying one method everywhere. A model grading its own domain of ambiguity is the familiar version, but the reverse appears too — an exact-match assertion over output that legitimately varies, which fails on rewordings that were never wrong. Non-determinism is why the exams keep arriving at measurement rather than at string comparison, as covered in the model fundamentals material.

A/B testing needs a control

The objective pairs A/B testing with iterative improvement, and the examinable content is almost entirely in the first half. A comparison requires something to compare against, running at the same time, on comparable traffic.

What people do instead is change something, watch the numbers, and decide the numbers look acceptable. That is an observation, and it cannot separate the effect of the change from everything else that moved that week. On an exam that repeatedly rewards attribution, an option offering a before-and-after without a concurrent control is a reliable wrong answer.

Optimisation is a three-way trade

Tokens, latency and cost are named together in one objective because they move together. Trimming context reduces tokens and cost, and can reduce quality. Adding a verification step raises latency and cost, and raises reliability. Batching reduces cost per item and removes any per-item deadline.

So an answer that reports an improvement in one quantity without naming what moved in the others has described half a change. The exam-shaped version says which quantity was given up and why this system can absorb that — which is the same structure as every other trade on the paper, and the reason the integration domain’s accuracy-against-latency objective asks you to justify a configuration rather than just choose one.

Monitoring after it ships

The last objective moves the same question into production: logging and observability once the system is running and nobody is watching a particular request. The design decision is what gets recorded, because the information you did not capture cannot be recovered afterwards.

The distinction that decides these items is between health and diagnosis. A dashboard that is entirely green tells you nothing about why one specific answer was wrong, and a system can be available, responsive and cheap while producing poor output at a steady rate. Evaluation before launch and observability after it are the same discipline pointed at different moments.

Key takeaways

• Sixteen percent, ten items, six objectives. The second-largest domain on CCAR-P, behind Integration at 19%.
• Five metric families are named. Accuracy, latency, cost, safety and security — and a quality-only suite covers one.
• Safety needs cases that should fail. A suite of things that should succeed cannot measure a boundary holding.
• Match the grader to the measurement. Deterministic, model-graded and human review each fit different questions.
• No control, no A/B test. Before-and-after cannot separate your change from everything else that moved.
• Name what you gave up. Tokens, latency and cost trade against each other and against quality.