Anthropic to Newsom: let courts block 'risky' AI models; what it means in Merced

Key Takeaways

1. Anthropic’s Oct. 15 letter to Gov. Gavin Newsom backs court orders to block AI models deemed a “serious risk.”
2. Reuters reported the letter on Oct. 28, and the Merced Sun-Star carried it in the business section.
3. Any state power to pause models could affect UC Merced labs, public agencies, and local vendors using third‑party AI.
4. The letter echoes ongoing federal talks about restricting unsafe AI, with legal review as the backstop.

What Anthropic asked for

“When a review finds a model poses a serious risk, the government should be able to ask a court to block it.” That line sits at the heart of Anthropic’s Oct. 15 letter to Gov. Gavin Newsom, reported Wednesday by Reuters and published by the Merced Sun-Star. The company argued for targeted legal tools, not blanket shutdown switches, when a model could endanger the public.

The pitch lands as California agencies weigh how to police fast-moving AI releases. It matters here because a court order would not stop at the tech firms in San Francisco. It would bind anyone deploying the flagged model, which can include Valley schools, hospitals, and ag suppliers that buy software rather than build it.

Why this could matter in the Valley

UC Merced faculty run projects that rely on commercial AI for computing or as a baseline model before research fine-tuning. Merced College and local districts are testing AI tools for tutoring and back-office work. County IT shops buy procurement bots and document classifiers from vendors that ride on top of large models. If the Attorney General sought a court review and a judge agreed a model posed a serious risk, those deployments could be paused, even if the buyer did nothing wrong.

This is not a ban on research. It is a stop sign for specific models judged dangerous under a legal standard. For a Fresno or Merced employer, the near-term impact would look like a vendor email saying service is suspended pending a court order being lifted. In practice, the question becomes how fast courts move and how precise the orders read.

One detail that will matter to public agencies: notice. Who gets told first, and how specific is the description of the risk. Procurement language in Merced County and the City of Merced already allows termination for legal noncompliance, but most contracts don’t name AI pause orders yet.

What’s in the letter, and what isn’t

Anthropic’s letter echoes its broader safety stance, according to the reporting, and stops short of endorsing universal kill switches. The company points to courts as the venue to decide when a model can’t be released or must be pulled. That signals a preference for case-by-case calls with an evidentiary record and appeals path.

What’s not clear is the threshold. Who runs the initial “review” that would justify asking a court to block release. How a smaller vendor proves its system rides on a safe branch or version. Whether state guidance will define a process agencies in Merced County can follow without guessing. We’ve asked the Governor’s Office and the Attorney General for the mechanics and timeline, no answers yet.

A note from the newsroom table, where a lukewarm coffee sat by the scanner.

What local leaders should watch

• Contract clauses: add language that covers AI-related court orders so stoppages don’t trigger breach fights.
• Subprocessor maps: ask vendors to list which foundation models they use and the version history.
• Continuity plans: line up alternatives if a core tool depends on a model that gets paused.

For now, the clearest sentence in the record belongs to Anthropic’s letter: “When a review finds a model poses a serious risk, the government should be able to ask a court to block it.”

Central Valley AI is produced by the CVAI Newsdesk team and developed by Kaweah Tech, a regional firm that builds, deploys, and integrates AI solutions for businesses across California's Central Valley.

Source

https://www.mercedsunstar.com/news/business/article317811853.html