Anthropic, the company behind Claude, wants California to let courts stop dangerous AI models before they cause harm. In a letter to Gov. Gavin Newsom, the company laid out how it thinks AI kill switches and outside safety auditors should work.
The letter answers ideas Newsom raised in an executive order last month. It also lands as state lawmakers prepare to return to Sacramento in about a month with a long list of AI safety bills.
What Anthropic’s Newsom Letter Asks For
Anthropic says independent evaluators working inside frontier AI companies should be able to report risks they find to the state Attorney General. If the Attorney General decides a model poses a near-term risk of catastrophic harm, the state could seek a court order. That order could suspend the model’s deployment, its internal use, or further development.
The company also wants large AI developers to publish risk reports at least every six months. Those reports would give the public a regular look at what the biggest labs think could go wrong.
- Courts could pause a risky model after a state request
- Outside evaluators would work inside AI companies
- Evaluators could report risks to the Attorney General
- Large developers would publish risk reports every six months
Why AI Kill Switches Are Not a Full Answer
Anthropic supports kill switches, saying every frontier AI company should be able to rapidly shut down models it serves or uses internally. But it warns that a law focused only on shutdown could give a false sense of reassurance. Pulling a whole model offline, it says, only makes sense in extreme situations.
Instead, the letter compares the idea to a circuit breaker. Companies need monitoring that spots problems and responds at the level where they happen. Anthropic says it already has processes for critical safety incidents and for rapid shutdown, and that it is updating them as AI capabilities grow.
How Outside AI Safety Auditors Would Work
Anthropic compares the access auditors should get to what banks give their outside accountants. It should be comparable to what a company’s own risk assessment staff have, with limits around sensitive customer data or technical details that do not matter for risk checks.
It also suggests a different way to pay them. Rather than paying auditors directly, companies would pay into a fund run by the state. State officials would rate auditors on the rigor of their reasoning and their track record of finding risks, and could require firms to use at least one highly rated evaluator.
| Question | Anthropic’s position |
|---|---|
| Who can stop a risky model? | A court, after the Attorney General asks |
| Who checks the labs? | Independent evaluators working inside companies |
| Who pays the evaluators? | Companies pay into a state-run fund |
| How often are risks reported? | At least every six months |
| Are kill switches enough? | No, they are only for extreme situations |
How the Newsom Letter Differs From OpenAI’s
OpenAI sent Newsom a similar letter last week, according to the report. It also backed shutdown controls and safety auditors embedded in AI companies, again using the circuit breaker comparison.
It did not go as far as Anthropic on court power. OpenAI instead proposed updating California law so companies must monitor models under development for cyber and other risks.
What This Means for You
If you use tools like Claude at work, none of this changes your account today. These are policy proposals, not laws. But California often sets the pattern for other states, so the rules that come out of Sacramento could shape how every major AI company operates.
- Watch for AI safety bills when the California Legislature returns
- Expect more public risk reports from large AI developers
- Ask vendors how they monitor and shut down AI agents that misbehave
- Read our earlier story on kill switch order
Anthropic’s ask is a proposal, and lawmakers may land somewhere different.
Sources
Primary source: none was published, because the letter was shared exclusively with one outlet. Reporting from Politico.
Conclusion
Anthropic is asking California for a court-backed way to stop risky AI models, paired with outside auditors and regular risk reports. Whether lawmakers follow that path or OpenAI’s lighter one will be a key AI policy story when the Legislature returns.
