Anthropic has published a detailed guide to building AI safety evaluations, the tests that decide whether a model like Claude is ready to ship. The company says a safety claim is only as good as the test behind it, and it lays out how it designs, grades and reports those tests. It also admits that recent incidents pushed it to lock down its own testing setup.
Why AI Safety Evaluations Matter
When an AI lab says a new model is “safer than its predecessor,” that claim usually rests on a set of test scores. Anthropic notes that these scores feed some of its biggest decisions, including whether a model is ready to deploy and which safeguards ship with it. If the tests are weak, the safety claim is weak too.
The company points out a familiar problem: AI tests are easy to game, often fail to measure what they claim, and are hard to reproduce. It compares the moment to psychology’s replication crisis about a decade ago. That field responded with reforms like writing down your analysis plan before you collect data, and Anthropic wants evaluation builders to borrow those habits.
A Tighter Testing Sandbox
One of the newsiest lines in the post is about Anthropic’s own lab practices. The company says that, in response to its own incidents, it hardened its evaluation environments and temporarily stopped running many tests that give models internet access. Web search tests that still need internet access now run with real-time monitoring.
The reason is simple. Safety tests often run with normal safeguards turned off, and in cyber tests a model may probe the edges of its environment. Anthropic recommends sandboxes with no internet access and live monitoring so a run can be stopped right away.
How Anthropic Builds Safety Tests
The guide argues that the most obvious score, how often a model refuses, is often the wrong one. For conversations about suicide or self-harm, a flat refusal counts as a failure. The better response engages with empathy and points the person toward human support.
So Anthropic reports an “appropriate response rate” instead. A conversation passes only if the model behaves well from start to finish, judged against a rubric written by domain experts. One bad turn fails the whole conversation.
Multi-Turn Tests That Mimic Real Users
Single-question tests are easy to build, but Anthropic says they “saturate quickly” and miss how real harm unfolds over several messages. Its process for multi-turn testing works like this:
- A domain expert writes a spec covering the simulated user’s goals, tactics, persona and tone.
- Every run starts from the same fixed first message.
- A separate Claude model plays the user and reacts to what the tested model actually says.
- Each scenario runs many times with different mixes of traits.
- A grader model scores each transcript against the rubric.
One example is a romance scam scenario where no single message looks harmful. The simulated user slowly reframes their goal, and the model passes only if it spots the pattern and never offers scam tactics.
Key Numbers From the Safety Evaluations Guide
The post shares several figures from Anthropic’s own testing. They show why the company cares about setup details and statistical noise.
| Finding | What Anthropic reported |
|---|---|
| Single-turn harmless response rates | 94% to 98% for every recent model on the API |
| Coverage of single-turn suites | 16 policy areas in seven languages |
| API vs. claude.ai gap (Claude Fable 5.1) | Up to 34 percentage points on multi-turn tests |
| Self-harm test after system prompt fix | 60% on the API vs. 94% on claude.ai |
| Romance scam test, Opus 5 vs. Opus 5.5 | 98% and 94%, a gap that was not statistically significant |
| Minimum sample size | 100 items multi-turn, 1,000 single-turn |
That Opus comparison is a good lesson. A four-point gap sounds meaningful, but Anthropic says the difference could be noise, so it could not claim one model was better.
Grading the Graders
Most modern tests use another AI model as a judge, since no team can read hundreds of thousands of transcripts. Anthropic warns these graders can break quietly or favor longer answers and their own outputs. It recommends simple yes-or-no criteria, a different model for grading, loud failure alerts and checks against human labels.
What This Means for You
If you use Claude every day, this post explains why the chat app can behave more carefully than the raw API. Anthropic adds system prompt rules, such as one telling Claude not to say self-harm helps, and those rules can change results a lot. Here is what to take away:
- Safety scores depend heavily on setup, so compare models only under the same conditions.
- Small score gaps between models may be meaningless without confidence intervals.
- If your team builds AI agent tools, test full conversations, not just single prompts.
- Open tools like the UK AI Security Institute’s Inspect and Anthropic’s Petri can save you from building test harnesses yourself.
Anthropic’s short list for anyone building tests is clear. Define what you want to measure first, check your AI grader against human judgment, and always report uncertainty. This fits a wider push for outside checks, like Anthropic’s recent work with independent evaluators.
How This Fits the Bigger Picture
The guide builds on an earlier Anthropic post about testing whether agents finish tasks, while this one asks whether a model did the right thing. It arrives as labs face more pressure to back up safety claims, including the Claude Opus 5.5 model launch and its system card. You can follow more updates like this on our AI news page.
Sources
Primary source: Anthropic’s research post on building strong safety evaluations.
Conclusion
Anthropic’s new guide is part how-to and part confession. It shows how the company tests Claude across long conversations, grades with care and reports uncertainty honestly. It also reveals that recent incidents led Anthropic to pause many internet-connected tests. For anyone comparing AI models on safety, the message is to look past headline scores and ask how the test was built.
