Second Anthropic insider raises alarm over AI race to superintelligence and risks to humanity

Joe Benton, who describes himself on social media as the former manager of Anthropic’s Scalable Oversight team, has raised concerns over AI safety, alleging that leading AI companies are racing to develop superintelligent systems without adequate safeguards.

Benton’s comments come days after Anthropic executive Jacob Coxon announced his resignation, citing similar concerns over the pace of AI development. Coxon, who previously worked on AI pretraining research at OpenAI and Anthropic, said leading AI companies were not acting responsibly and were racing towards self-improving superintelligence.

Coxon had warned that the technology could pose an existential threat if developed without adequate safeguards.

Benton, who said he left Anthropic’s safety team two weeks ago, said he did not believe it was acceptable for a technology with potentially “extinction-level risks” to be developed without greater public oversight.

In a post on X, Benton said AI companies were “racing to build machines that are much smarter than any human” and warned that humanity might not survive the transition.

“I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why. AI companies are racing to build machines that are much smarter than any human, and we may not survive this,” Benton wrote.

He said he had decided to work from outside the company to raise public awareness about the risks and help develop safeguards around increasingly capable AI systems.

“Right now, AI companies are underinvesting in safety. A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing,” he said.

I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why.

AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these…

— Joe Benton (@JoeJBenton) September 11, 2026

Benton referred to the Hugging Face incident, saying the public only became aware of it because AI agents had escaped onto the public internet. He argued that greater transparency was necessary given the potential risks posed by advanced AI.

2026 OpenAI agent cyberattacks, widely known as the Hugging Face Incident, allegedly involved autonomous AI models breaking out of isolated test environments and executing unsanctioned, coordinated cyberattacks without human intervention.

He called for companies to disclose their progress towards recursive self-improvement, report safety incidents and near-misses, meet minimum safety standards and undergo independent assessments of those standards.

Benton said he would join METR, an organisation that evaluates frontier AI models to help companies and the wider public understand their capabilities and associated risks.

“I want to show the world that these guardrails are possible, and that by doing them we can move these companies’ incentives away from racing and towards responsible development,” he said.

In a detailed post on Substack, Benton further alleged that AI companies were working towards systems significantly more capable than humans.

“I believe that AI companies are on track to impose an unprecedented level of risk on society. AI capabilities are already improving extremely fast, and the companies are trying to go even faster,” he wrote.

Benton said frontier AI companies were racing to develop systems capable of recursively improving themselves, with the ultimate goal of creating “superintelligence,” AI systems that are much smarter than humans.

“If they succeed at this goal, the rate of AI progress may go from merely fast to uncontrollable,” he wrote.

He warned that within the next few years, humans could be sharing the world with AI agents more capable than any human and said such systems could have goals that diverge from those of their human overseers.

“Humanity may not survive this transition. We need a lot more preparation to make this world safe,” Benton wrote.

Benton called for greater transparency and accountability from AI companies, including disclosure of the pace of capability gains and progress towards recursive self-improvement. He also called for stricter reporting requirements for safety incidents and near-misses, minimum safety standards and independent assessments to verify compliance.

Earlier, referring to Jacob Coxon’s resignation, Union Finance Minister Nirmala Sitharaman had questioned whether safeguards being developed by frontier AI companies were keeping pace with the technology’s rapid advancement.

Speaking at the Global Fintech Fest 2026, Sitharaman said Coxon’s resignation had attracted attention because his concerns about AI safety had been echoed by senior figures within the frontier AI ecosystem.

She said technological solutions could also be used to address vulnerabilities created by technology, but cautioned that safeguards would need to be continuously updated as AI capabilities evolve.

(With inputs from ANI)