Ex-Anthropic researcher warns of near-term extinction risk

Former Anthropic researcher Jacob Coxon said AI lab staff are “genuinely frightened” by the speed of frontier model progress and warned there is a “strong chance” humanity could die “in the immediate future” without a slowdown. Anthropic responded by stressing safeguards and “aggressive” testing while endorsing a lawful, verifiable way for the industry to pace releases, as prominent AI leaders split publicly on how seriously to take extinction-risk framing.

Key Takeaways

Former Anthropic researcher Jacob Coxon said AI lab staff are “genuinely frightened” about the pace of progress and what it could mean for humanity.
Coxon warned that without slowing development “there is a strong chance that we could all die in the immediate future,” and argued any real slowdown would need coordination with China to avoid an international race.
Anthropic said it builds models with “some of the strongest safeguards in the industry,” “aggressively” tests systems, publishes findings to aid scrutiny, and supports a lawful, verifiable mechanism to pace releases of powerful models.
Public reaction split between dismissal from some executives and endorsement from inside Anthropic, including Evan Hubinger’s claim that extinction risk is “>10% within the next decade.”

Ex-Anthropic Insider: Staff Are “Genuinely Frightened” as AI Progress Accelerates

Jacob Coxon, a 27-year-old former Anthropic researcher who previously worked at OpenAI, has pushed the AI-risk debate back into the mainstream with a blunt message: people building frontier AI models are scared, and they think the timeline is short. In an interview on “Sunday with Laura Kuenssberg,” Coxon said staff at AI companies were “genuinely frightened” about the speed of advances and the implications for humanity.

Coxon tied that fear to a call for regulation and pacing rather than a vague plea for “safety.” “I believe that if we don't slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future,” he said.

For markets, the immediate relevance is not whether Coxon’s personal timeline is right. It is that a former employee is making an extreme near-term claim at the same moment Anthropic’s leadership is publicly arguing for mechanisms to slow releases. That combination keeps “pacing,” oversight, and regulation narratives live, even without a new bill or enforcement action.

Coxon’s Case for a Slowdown—and Why He Says China Has to Be Part of It

Coxon’s argument is built around a race dynamic. He said AI workers are “completely serious” when they ask for regulation because they feel “trapped in a race” and “scared of the outcomes of that race.” The mechanism he is pointing at is competitive deployment pressure: if one lab believes another lab will ship a more capable model, restraint becomes individually costly even if it is collectively safer.

That is why Coxon framed any meaningful slowdown as geopolitical, not just corporate. “Sam Altman and Elon Musk have agreed that this is a great idea. But I think there'll need to be some sort of co-ordinated slowdown with China if we're going to avoid a race at an international scale,” he said.

He also offered a concrete failure mode, echoing a scenario referenced in Dario Amodei’s comments: a “swarm of bots acting like a supercomputer” that could take over the internet. Coxon said that scenario could be realistic in “six months to a year,” and added that peers feared danger could arrive within “the next two years.” Those are assessments rather than claims supported by technical evidence in the packet, but they are doing narrative work by putting a clock on the debate.

The catch is implementation. “Co-ordinated slowdown with China” is a high bar because it implies verification and enforcement across rival states. In practice, that tends to pull policy talk away from an outright pause and toward monitoring, standards, and disclosure regimes that can be audited.

Anthropic’s Reply: Safeguards, “Aggressive” Testing, and a Verifiable Pacing Mechanism

Anthropic’s response did not concede Coxon’s timeline, but it did validate the category of risk and the need for coordination. A spokesperson said the company has “always been transparent that AI will bring both enormous benefits and unprecedented risks,” and that it continues to build models with “some of the strongest safeguards in the industry.”

The company described a safety posture that is meant to be legible to outsiders: it “aggressively” tests models and publishes findings to aid scrutiny and prevent incidents of “AI misalignment,” the term for systems whose behavior diverges from what humans intended. Anthropic also said it was the first to publish a framework for mitigating risks posed by model development.

Most market-relevant is the explicit endorsement of pacing as an industry mechanism rather than a unilateral choice. “This work is also why we believe the world would benefit from the industry adopting a lawful, verifiable way to work together to pace how we release powerful models,” the spokesperson said.

That language lines up with CEO Dario Amodei’s essay, “We Must Pace the Frontier,” published Saturday. Amodei proposed a three-part plan: independent monitoring of AI models as they are developed, industry-wide regulation, and global regulation. Independent monitoring, in this framing, is third-party oversight during development rather than relying solely on the lab building the system.

Signals to Track: Credibility Split Among AI Leaders and the Next Regulatory Hooks

The public split among AI leaders is now part of the story, and it cuts both ways for narrative-driven positioning. Hugging Face CEO Clément Delangue criticized the framing of Coxon’s extinction-risk commentary, writing on X: “Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change.” He added: “Not saying it's necessarily uninteresting or wrong per se but let's keep things in perspective.” After Amodei’s essay, Delangue also offered to help with potential solutions.

Nvidia CEO Jensen Huang also pushed back. At a Goldman Sachs-hosted conference last week, multiple people in the group said Huang dismissed Coxon’s comments as untrue. Huang has previously called the notion that AI was “going to be the end of humanity” “complete nonsense.”

On the other side, Anthropic scientist Evan Hubinger publicly endorsed the core fear. “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” he wrote on X.

The next hooks are concrete, not rhetorical: whether Amodei’s independent monitoring concept turns into named third parties, standards, or pilots. Whether major labs or governments explicitly endorse or reject a “lawful, verifiable” pacing mechanism. And whether international coordination language, especially explicit references to China, starts appearing in policy speeches or AI safety summit outcomes. The other tell is whether more senior executives go on-record with Coxon- or Hubinger-style probabilities, which would either mainstream the claim or isolate it.

My Read: Why This Debate Can Still Spill Into AI-Token Sentiment Without Any New Policy

The part that decides this isn’t Coxon’s “immediate future” timeline. It’s the fact that Anthropic is simultaneously defending its safeguards and asking for a lawful, verifiable way to pace releases, which keeps third-party oversight and coordinated standards in the conversation even when no regulator has moved.

The real test is whether “independent monitoring” becomes a real object traders can point to, with named monitors and a standard that can be checked, rather than a general call for responsibility. If that turns into pilots or formal endorsements, the pacing narrative stops being a media cycle and starts looking like a constraint that can shape how frontier AI gets shipped and marketed.

Ex-Anthropic researcher warns of near-term extinction risk as AI pacing debate flares