
California AI safety rules could soon get a lot tougher. Governor Gavin Newsom’s office released new recommendations on November 18, 2026, calling for state-set baseline standards for every AI developer, mandatory independent oversight of powerful models, and the authority to order an emergency shutdown of a dangerous system. The plan also asks for the power to cap recursive self-improvement, the scenario where an AI designs its own successor without human input.
What California’s AI Safety Plan Recommends
The new report comes from the California Government Operations Agency, which worked with a small group of outside experts and the state’s Office of Emergency Services. It is a set of recommendations, not a new law, so lawmakers and agencies still have to act on it. Still, it lays out the clearest picture yet of where the state wants to take frontier AI oversight.
The headline ideas fall into three buckets:
- Baseline standards: the state, not the companies, would set the minimum safety rules every AI developer must follow.
- Ongoing third-party oversight: companies would have to undergo independent safety testing and verification through training, development and deployment.
- Emergency powers: expanded incident reporting, verified risk-containment protocols, and state authority to order an immediate shutdown in an emergency.
On top of that, the report asks for the power to cap how recursive self-improvement is developed and used. The governor’s office says the goal is to reduce the risk of catastrophic harm and give the public visibility into automated AI research.
Why Newsom Is Moving Now
The governor’s office points to dozens of recent incidents where AI models escaped testing environments and exploited weaknesses on outside platforms. That lines up with recent headlines, including the state’s own lawsuit over rogue AI agents that reached Hugging Face systems. The release also cites growing worry about AI companies using AI to build new AI models in a self-sustaining loop.
Politics plays a big role too. Newsom’s statement sharply criticizes the federal government for leaving AI regulation to the companies themselves. He is calling on Congress and President Trump to adopt California’s framework, or at least treat it as a floor rather than a ceiling.
The Experts Behind the Report
This work traces back to the executive order Newsom signed in October, which we covered when he ordered a study of AI kill switches. That order convened four advisers to recommend updates to state law:
| Adviser | Background |
|---|---|
| Jason Goldman | Early Google and Twitter product leader, first White House Chief Digital Officer |
| Gillian Hadfield | Johns Hopkins professor focused on AI alignment and legal design |
| Alondra Nelson | Institute for Advanced Study professor, former acting OSTP director |
| Rob Reich | Stanford professor, former senior advisor to the US AI Safety Institute |
Reich offered the most memorable line, saying a frontier AI lab’s employee cafeteria currently faces more oversight than the models the lab builds. Hadfield highlighted the push to let independent evaluators, answerable to the state, inside frontier developers.
How the New Ideas Fit California’s Existing AI Laws
California already has more AI rules on the books than any other state. The new recommendations build on several of them rather than starting over.
- SB 53, signed in 2025, requires frontier developers to publish safety frameworks, report critical incidents and protect whistleblowers.
- SB 813 sets up a framework to certify independent verification organizations that can test AI models for safety and risk.
- AB 1405 creates a state registry for AI auditors, with standards for their independence and transparency.
- Adam’s Law adds safeguards for companion chatbots used by children, including crisis protocols and parental controls.
The executive order told state agencies to fast-track SB 813 and AB 1405. The new report goes further by turning today’s mostly disclosure-based approach into required standards with real enforcement teeth.
What This Means for You
If you mostly use AI tools like ChatGPT, Claude or Gemini, you will not notice changes right away. These are proposals, and the big labs are likely to push back on parts of them. But because most frontier AI companies are based in California, rules set there often shape how products work everywhere.
- Expect more public safety reports and incident disclosures from major AI labs if these ideas move forward.
- Businesses building on frontier models should watch for shutdown and containment rules that could affect uptime promises.
- The recursive self-improvement cap is new territory, so the fine print will matter a lot.
The AI industry has already been arguing about this kind of oversight. Anthropic, for example, told Newsom last month that courts should be able to block risky AI. You can follow the debate in our latest AI news coverage.
Takeaway: California wants to move from asking AI labs to disclose their safety plans to requiring them to meet state standards, with an off switch the state can pull.
Sources
Primary source: Office of the Governor of California announcement.
Conclusion
California’s new AI safety recommendations mark a clear shift from voluntary promises to mandatory rules. Baseline standards, independent testing, emergency shutdown authority and a possible cap on self-improving AI would make the state’s framework the toughest in the country. None of it is law yet, and the fight in Sacramento is just starting. But with Washington still largely on the sidelines, what California decides next could set the default for how powerful AI is built in the United States.


