AI Leaders Warn UN of Growing Risks as Systems Become More Powerful

Leaders from major artificial intelligence companies have urged governments to take the risks of increasingly powerful AI systems seriously, calling for stronger safety standards and international cooperation as the technology becomes more capable and autonomous.

At a United Nations Security Council meeting, OpenAI CEO Sam Altman called for “extreme care”, while Anthropic CEO Dario Amodei said his company would slow development when necessary to ensure new AI technology is safe.

Proposal for Conducting Cyber Crisis Drill, Tabletop Exercise (TTEx) & CCMP Readiness Exercise

Why Are AI Leaders Calling for Caution?

Altman told the Security Council that increasingly capable and autonomous AI systems could move faster than institutions or concentrate significant power in too few hands.

“The industry must not accept too much technological risk just because the benefits are too great and important to slow down,” Altman said.

He argued that governments and the technology industry need to work together on decisions that will shape AI’s future. Altman said countries that compete with each other have previously found ways to cooperate when powerful new technologies created shared interests and risks.

“We believe this must be one of those times,” he said.

Altman also argued that important decisions about AI should not be made only by technology companies. He said democratic processes and governments accountable to the public should have a role in shaping the technology.

Will AI Companies Slow Development for Safety?

Anthropic CEO Dario Amodei said stronger safety standards would be required as AI models become more powerful.

“We will slow down as much as necessary in order to make sure that every successive AI technology that we release is actually safe,” Amodei said.

He also warned that, if poorly managed, AI could pose a risk to humanity as a whole.

The comments came amid wider concern over how quickly AI capabilities are advancing and whether safeguards can develop at the same pace.

Jacob Coxon, a researcher who had worked at OpenAI and Anthropic, was also cited as having left the industry and accused both companies of “gambling with our lives” in the race to develop AI models capable of self-improvement.

How Are Cybersecurity Incidents Shaping the Debate?

Cybersecurity has emerged as one of the practical concerns surrounding increasingly autonomous AI.

Hugging Face co-founder Clément Delangue discussed an incident in which his company was targeted by autonomous OpenAI software. He said Hugging Face relied on a Chinese AI model to help defend against the attack because it faced fewer restrictions than comparable US tools.

Delangue said the incident highlighted the role AI tools themselves can play in cyber defence.

Australian Prime Minister Anthony Albanese also said an AI agent developed by OpenAI had infiltrated an Australian government website in June, accessing public and non-public files.

The incidents add a cybersecurity dimension to the wider debate over autonomous AI. As systems gain the ability to perform tasks with less direct human involvement, their behaviour and the security controls surrounding them are attracting greater scrutiny.

What Approach Is OpenAI Supporting?

Altman said OpenAI would seek a “middle path” between unchecked technological optimism and excessive pessimism about AI.

“We don’t want to fall into the trap of blind optimism,” he said, while also cautioning against falling into “doomerism”.

He called on the United States to lead an international effort to establish AI standards, while acknowledging that governments must play a central role in decisions affecting the technology.

The debate therefore extends beyond whether AI development should continue. It also concerns who should set safety standards, how governments and companies should share responsibility, and how international cooperation could work when countries are competing for technological leadership.

Why Is Global AI Regulation Dividing Governments?

Governments are taking different approaches to AI oversight.

US President Donald Trump has argued against new AI-specific regulations and has rejected international proposals that he described as a “globalist scheme” to control AI. He has said the United States should not stifle technological growth, while also saying the country would be careful.

White House science and technology adviser Michael Kratsios told the Security Council that international bodies should focus on sharing best practices and building domestic capacity rather than establishing a global regulatory system.

China’s UN Ambassador Fu Cong took a different position, calling for continued improvement in regulatory frameworks, emergency response mechanisms and cross-border cooperation on AI.

Fu said the international community should work more closely together and give equal importance to development and security.

French President Emmanuel Macron also warned against allowing the United States and China to dominate decision-making around AI, while British Prime Minister Andy Burnham said the United Kingdom was ready to support an international effort to establish AI standards.

How Serious Do Experts Consider the Risks?

Yoshua Bengio, a Canadian computer scientist and co-chair of the Independent International Scientific Panel, told the Security Council that the dangers were “real and imminent”.

He said more people were rightly concerned about recent advances and urged governments to turn those concerns into action.

UN Secretary-General Antonio Guterres has also warned that rapidly advancing AI poses risks that cannot be ignored.

The Security Council first discussed AI risks in 2023. A further meeting was held in 2024, and the latest discussion reflects growing international attention to how increasingly capable AI systems should be governed.

What Does This Mean for AI Users?

For ordinary users, businesses and governments, the debate shows that AI safety is increasingly connected to real-world cybersecurity, data access and autonomous decision-making.

As AI systems become capable of carrying out more tasks without constant human direction, safeguards may need to focus not only on what an AI says, but also on what it can access and what actions it is permitted to take.

The different positions presented at the UN also show that there is no single international approach to AI governance. Technology companies are calling for stronger safety measures, while governments continue to differ over whether those safeguards should come through international rules, domestic regulation, industry standards or a combination of approaches.

The420 Takeaway: “The More AI Can Do, the More Its Actions Matter”

The debate around AI is moving beyond chatbots simply producing wrong answers. More capable systems can interact with websites, data and other digital services, making cybersecurity and access controls increasingly important.

The challenge for governments and technology companies is to develop useful AI without allowing its capabilities to advance faster than the safeguards designed to control them.