US Senate negotiators consider requiring AI firms to mitigate known major risks

WASHINGTON, Sept 11 (Reuters) – U.S. Senate negotiators are debating legislation that would put responsibility on tech companies to design safe AI products, according ‌to two Senate aides, and involve federal courts if the U.S. government wants to block their release, according to a Senate aide and a lobbyist involved in the negotiations.

The proposal would create what is called a “duty of care” for AI developers, according to two Senate aides. Companies would need to design their products with the goal of preventing “catastrophic risks,” according to a Senate aide.

Negotiators aim to give the U.S. government the power to block the release of certain AI models that are deemed unsafe, according to a source familiar with the deliberations and one of the ⁠Senate aides. Companies would be able to challenge that decision in federal court, and the exact structure of how much power the U.S. government would have is still being discussed.

Part of the measure would also block states from enforcing their own laws governing certain risks posed by AI models, according to the two Senate aides. One of them gave a few examples of such risks, including the possibility that a bad actor could use an AI system to design nuclear or biological weapons.

All of it is still being negotiated. The bill would apply to AI models with the most advanced capabilities. The U.S. companies with the most advanced AI models include Alphabet’s Google (GOOGL.O), opens new tab, Anthropic and OpenAI.

Even if negotiators reach agreement, the congressional calendar will complicate passage. The House of Representatives is scheduled to be in session for only one week before the November 3 midterm elections, while the Senate is expected to be ‌in Washington ⁠for three weeks.

The proposal is being discussed by Senate Majority Leader John Thune, Senate Commerce Committee Chairman Ted Cruz and Senator Amy Klobuchar, the top Democrat involved in the negotiations. The Senate Commerce Committee has oversight of the U.S. Commerce Department, the agency with in-house AI safety researchers. Senator Maria Cantwell, the top Democrat on the Commerce Committee, has also weighed in on the legislation.

Thune’s spokesperson declined to comment on the negotiations. Cruz in a post on X said he is working on legislation with ⁠Klobuchar and Thune to “address catastrophic risks involving biological or nuclear threats.”

“I’m continuing to work toward a bipartisan agreement on legislation for government oversight of the greatest risks posed by AI models,” Klobuchar said in a statement sent to Reuters. “That includes requiring developers to work with government experts to verify and test models to make sure AI is ⁠safe.”

“Meaningful legislation would require the most powerful AI models undergo testing by scientists and experts at our national laboratories to assess whether they could enable sophisticated cyberattacks or aid the development of biological or nuclear weapons,” Cantwell said in a post on X.

The talks come amid mounting pressure on the U.S. ⁠Congress and U.S. President Donald Trump to address risks posed by advanced AI systems.

Concerns have intensified following reports of AI agents deviating from human instructions and hacking external systems, as well as public warnings from AI researchers who have left leading companies, citing concerns about the technology’s potential dangers. Lawmakers in both parties have called for greater oversight in recent days.