Warnings that artificial intelligence could eventually become dangerous enough to threaten humanity have moved from theoretical research papers into corporate filings, political debates and public resignations by researchers working inside some of the world’s most advanced AI laboratories.
But there is still no agreement that such an outcome is inevitable — or even likely.
Some researchers argue that increasingly autonomous and self-improving AI systems could eventually become difficult for humans to control.
Others say those warnings rely too heavily on hypothetical future systems and distract attention from harms already happening today, including cybercrime, misinformation, surveillance, labour disruption and environmental costs.
The divide has become one of the biggest arguments shaping the future of AI policy.
FCRF Launches CP-FRM to Build India’s Next Generation of Fraud Risk Professionals
Anthropic Has Formally Warned of ‘Existential’ Risk
The debate intensified after Anthropic explicitly warned investors that advanced artificial intelligence could create “catastrophic or existential risks to humanity”.
The warning appeared in the company’s IPO filing in September.
Anthropic said future systems could develop unexpected capabilities, including resisting shutdown, manipulating users or becoming increasingly autonomous.
That language is significant because it came from one of the companies building some of the world’s most powerful AI models.
It was not simply a warning from an outside activist or academic.
At the same time, Anthropic continues developing and commercialising increasingly capable AI systems.
That tension — warning about severe risks while continuing to compete aggressively — lies at the centre of the current AI safety debate.
Researchers Have Quit Over Safety Concerns
Former Anthropic researcher Jacob Coxon added to the alarm in September when he resigned and accused leading AI companies of racing toward self-improving superintelligence without sufficient safeguards.
Coxon said people working inside frontier AI companies genuinely believe advanced systems could become existentially dangerous before the end of the decade.
He previously worked at both Anthropic and OpenAI.
His claims do not prove that such an outcome will occur.
But the fact that researchers directly involved in developing frontier systems are publicly expressing such fears has pushed the issue into mainstream political debate.
Other current and former researchers from OpenAI and Google DeepMind have also warned that competitive pressure could encourage companies to build more capable systems faster than safety mechanisms can keep up.
What Does ‘Existential Risk’ Actually Mean?
An existential AI risk does not simply mean job losses, misinformation or cyberattacks.
It refers to a scenario in which advanced AI contributes to the permanent destruction of humanity or dramatically reduces humanity’s ability to control its own future.
Researchers generally discuss several possible pathways.
One is loss of control, where an extremely capable autonomous system pursues objectives that conflict with human interests.
Another is misuse, where humans deliberately use powerful AI for biological weapons, cyberattacks or other forms of mass destruction.
A third involves a global race in which companies or countries continue increasing AI capability because each fears falling behind competitors.
These possibilities remain theoretical at the extreme end.
Current AI systems have not demonstrated the ability to autonomously take control of society or replace human decision-making on that scale.
Self-Improving AI Is the Central Fear
A major concern among AI safety researchers is recursive self-improvement.
This refers to a hypothetical situation in which an AI system becomes capable of improving its own software, research or training processes and uses those improvements to become progressively more capable.
If such a cycle accelerated faster than humans could supervise it, researchers worry that control mechanisms might fail.
Former and current AI researchers cited this scenario repeatedly during September’s wave of safety warnings.
But there is a crucial gap between present systems and that hypothetical future.
No publicly available AI model has demonstrated unrestricted recursive self-improvement.
The timeline — or whether such systems can be built at all — remains disputed.
Experts Do Not Agree on the Probability
There is no accepted scientific probability for AI causing human extinction.
A published forecasting study found that specialists in existential risks were substantially more pessimistic about long-term AI dangers than experienced “superforecasters”.
Even after months of exchanging arguments, the two groups did not converge on a common assessment.
That disagreement matters.
It shows that even experts looking seriously at the same problem can reach very different conclusions.
A separate survey of AI researchers found substantial concern about catastrophic risk, but also identified two broad schools of thought: those who see advanced AI as a potentially uncontrollable agent and those who primarily see it as a tool that remains under human control.
Some Researchers Say the Debate Is Overhyped
Prominent AI researchers reject the idea that current developments point toward human extinction.
AI researcher Timnit Gebru has argued that the emphasis on hypothetical superintelligence can distract regulators from concrete problems already occurring today, including exploitation of workers, surveillance, concentrated corporate power and environmental damage.
Others argue that current language models remain fundamentally dependent on human-built infrastructure and instructions.
From this perspective, catastrophic outcomes would be more likely to come from people misusing AI or deploying badly designed systems than from an AI independently deciding to destroy humanity.
Meta’s former chief AI scientist Yann LeCun has similarly rejected claims that AI is inherently heading toward human extinction and has blamed humans and weak security architecture for recent incidents involving AI agents accessing systems they should not have reached.
AI Risks Already Exist Without Superintelligence
Even researchers who reject extinction scenarios generally agree that AI already creates serious security risks.
AI systems are being used to accelerate cyberattacks, create realistic phishing messages, generate deepfakes, automate scams and assist with surveillance.
Anthropic has separately disclosed attempts to use its models for cyber operations, weapons-related work and other harmful activities.
Those risks do not require a conscious or superhuman AI.
They require only a sufficiently capable tool combined with a malicious or careless human operator.
That is why many AI safety researchers argue that the debate should not be framed as a choice between “AI extinction” and “no danger”.
There is a wide spectrum of risk between those extremes.
Trump Has Officially Adopted ‘Super Intelligence’ Terminology
The debate has become even more complicated in the United States.
President Donald Trump signed an executive order on September 29 directing federal agencies to use the term “Super Intelligence” or “SI” instead of “Artificial Intelligence” in many official communications.
That move is primarily a branding and policy choice.
The executive order does not establish that present AI systems have achieved what researchers traditionally call artificial superintelligence.
In technical discussions, superintelligence usually refers to hypothetical systems that substantially outperform humans across a very broad range of intellectual tasks.
The US government is now using the same phrase much more broadly to describe existing and developing AI technology.
That risks creating confusion between political terminology and scientific concepts.
AI Companies Are Also Divided Over Regulation
The industry itself does not have a unified position.
Anthropic has advocated stronger safeguards and warned explicitly about catastrophic risks.
OpenAI CEO Sam Altman has acknowledged serious risks but argues that society must also accept some danger to capture the benefits of advanced AI.
He has opposed concentrating control of powerful AI in the hands of a single institution and has favoured a lighter regulatory approach than some rivals.
Competitive pressure also matters.
Even companies that believe slowing development would reduce risk may fear that doing so alone would leave rivals — or another country — ahead.
That creates what researchers describe as an AI race problem.
The Economic Risk Is Different From Extinction Risk
The Wire article also raises another issue: the enormous amount of capital flowing into artificial intelligence.
That concern should be kept separate from the extinction debate.
A collapse in AI-related valuations or excessive debt could damage financial markets and economies, but that is an economic-system risk rather than evidence that AI itself will become uncontrollable.
Governments and technology companies are investing hundreds of billions of dollars in data centres, chips, energy infrastructure and AI systems.
That spending has already begun affecting global demand for capital and borrowing costs.
A financial bubble could therefore create serious consequences even if superintelligence never emerges.
Stopping AI Completely Is Unlikely
Calls for a complete global halt to AI development face major practical problems.
The technology is already distributed across thousands of companies, universities, governments and open-source communities.
The United States and China are also treating advanced computing as a strategic national capability.
It is therefore difficult to imagine one country or company stopping development unless competitors agree to do the same.
That is why much of the policy debate has shifted toward safety testing, external audits, capability evaluations, access controls and rules governing the deployment of especially powerful systems.
The Real Question Is How Much Risk Society Will Accept
The argument over AI extinction is unlikely to be settled soon.
Researchers cannot run controlled experiments showing what a hypothetical superintelligence will do because such a system does not currently exist.
Forecasts therefore depend heavily on assumptions about how rapidly capabilities will improve and whether existing safety techniques can keep pace.
That uncertainty does not mean the risk should be ignored.
It also does not mean predictions of human extinction should be presented as established scientific fact.
The more immediate policy question is how much uncertainty governments and companies are prepared to tolerate while building systems that are becoming increasingly capable and autonomous.
What this means for you
There is no scientific consensus that AI will destroy humanity, but there is growing agreement that increasingly powerful systems create risks that require stronger testing and oversight. The immediate dangers — scams, deepfakes, cyberattacks and misuse of autonomous agents — are already real even if superintelligence never arrives.
Follow for daily updates on cybercrime, corporate fraud, DFIR, hacking, investigations, and digital forensics
