Anthropic has asked Governor Gavin Newsom to let courts block AI models that pose a serious risk. In an October 15 letter first reported by Politico, the company proposed that independent evaluators report risks to California's attorney general, who could seek a court order pausing a model's release, internal use, or further work on it.
Anthropic has told California Governor Gavin Newsom that courts should have the power to block AI models that pose a serious risk. The company made its case in a letter dated 15 October, first reported by Politico.
The letter responds to an executive order Newsom signed in September. That order created a working group tasked with recommending, within two months, how to strengthen California's AI safety laws. One open question is whether developers should be required to be able to switch their models off.
“When a review finds a model poses a serious risk, the government should be able to ask a court to block it,” Anthropic wrote. An Anthropic spokesperson confirmed the letter was genuine. Newsom's office did not immediately respond.
What would Anthropic's proposal let California courts do?
Under the company's proposal, independent evaluators would report risks to California's attorney general. If the attorney general saw a near-term risk of catastrophic harm, they could ask a court to pause a model. A court order could halt its release, its use inside the company, or further work on it, according to Politico.
Anthropic also wants outside safety auditors to work inside leading AI companies. Rather than paying the auditors directly, firms would pay into a state-run fund. Large developers would also publish risk reports at least every six months.
Does Anthropic support shutdown controls for AI models?
On shutdown controls, Anthropic agreed that every leading AI company should be able to shut down its models quickly. But the company warned that a law focused only on shutdowns could give false comfort. It instead backed narrower measures, such as turning off a single AI agent or cutting off one customer's access.
How does OpenAI's position differ from Anthropic's?
OpenAI sent its own recommendations to Newsom on 20 October. Politico reported that OpenAI backed shutdown controls and in-house auditors but did not support court orders to block risky models.
Anthropic is pressing a similar argument in Washington. A draft federal bill the company has circulated would let the Commerce Department ask a federal court to block a model's release in narrow cases. House lawmakers have also proposed an AI Kill Switch Act, which would let officials order AI firms to shut down dangerous models.
This article was written with the assistance of AI.
News Factory APP - agentic news to boost your SEO & AEO.