$1bn and 50 deaths: the tripwires Anthropic wrote into its own rules

Anthropic has drafted federal legislation that would let the US government block the release or use of advanced AI models posing catastrophic risks, and circulated it among lawmakers. The company behind the Claude chatbot confirmed the proposal reflects its broader Advanced AI Framework. It remains a draft. It is not law.

The most consequential detail is not the principle. It is the number.

According to a Politico report published October 16, the draft would permit government intervention when a model poses an imminent risk of causing more than $1 billion in damage, or of killing or seriously injuring more than 50 people. Those are the tripwires. Cross them, and Washington could act before a model ships.

The draft also would require independent safety evaluations, greater transparency from AI developers, and stronger security measures.

If adopted, this would create the first formal US mechanism for pre-release government intervention on AI models, replacing vague standards with quantified ones. For developers, that means a defined regulatory line. For the public, it sets the bar at which the government could move first.

But the numbers cut both ways, and that is where the debate will land. Set the thresholds too high and they never trigger — a rule that exists on paper and nowhere else. Set them too low and they chill releases, turning a safety valve into a brake on the industry. A $1 billion damage figure and a 50-casualty figure are specific enough to argue about, which is precisely what makes them the proposal's most debatable part.

There is a second thing worth noticing. Anthropic is not merely asking for AI regulation. It is drafting the specific thresholds that would let Washington constrain a model — possibly one of its own. That makes the company a co-author of the rules that could one day bind it. Whether that reads as responsibility or as positioning depends on what happens next.

Much remains unresolved.

For now, the record shows a proposal, not a policy. The principle is familiar. The numbers are new.