King Charles has warned leading figures in the artificial intelligence industry of the “existential dangers” that could arise if increasingly powerful AI technologies fall into the wrong hands.
The King issued the warning during a summit he convened at Dumfries House in Ayrshire to examine how AI can be developed and applied in ways that benefit society.
The meeting brought together senior figures from the technology sector and government, including UK AI Minister Kanishka Narayan, a papal adviser, and representatives of major AI companies such as Nvidia, OpenAI and Anthropic.
Addressing the executives, King Charles said those developing advanced AI systems were increasingly warning that the technology could acquire dangerous capabilities, including the potential to cause loss of life.
“Those who have created these technologies are now increasingly warning that AI risks developing darker capacities – perhaps even to take life,” the King said.
He stressed the urgency of addressing the risks associated with advanced AI falling into the wrong hands and being used in potentially catastrophic ways.
“The task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world,” he told participants.
The summit was convened to explore whether a “shared set of principles” could be developed to guide the future application of AI and ensure that its growing capabilities are accompanied by appropriate safeguards.
Nvidia Chief Executive Jensen Huang, one of the leading figures at the summit, said safety should be “paramount”, arguing that companies should hold back an AI product and continue developing it if it was not considered safe enough.
Huang also highlighted the potential benefits of open AI models, saying they could help ensure that people and countries are not left behind while enabling researchers, universities and start-ups to develop new applications.
He called for “responsible optimism”, urging the industry to build AI safely and securely while expanding access to the technology and using it to address major global challenges.
Sir Demis Hassabis, co-founder of DeepMind, offered a more cautious view on the pace of AI development. He said artificial general intelligence (AGI)—systems expected to perform as well as or better than humans across multiple tasks—could potentially be only a few years away.
Hassabis said such technology could have an impact “ten times that of the Industrial Revolution”, while acknowledging that the possibility of something going wrong was “definitely non-zero”.
He called for a balanced approach that would allow humanity to address emerging risks while continuing to develop the technology.
OpenAI Chief Financial Officer Sarah Friar similarly acknowledged AI’s potential to tackle some of society’s most difficult problems, but said its rapidly expanding capabilities raised important questions about safety and ensuring that the technology continues to serve people.
The summit comes amid growing debate within the AI industry over the pace of development, potential risks and the level of regulation required to keep increasingly advanced systems under control.
Researchers and technology executives have issued increasingly serious warnings about the possible consequences of advanced AI, while critics argue that excessive focus on hypothetical future threats could divert attention from more immediate concerns, including the technology’s environmental impact.
The debate has also intensified over whether AI companies should be trusted to regulate themselves or whether stronger government oversight is necessary.
As the technology continues to advance, the competing views have placed a central question before governments, researchers and technology companies: how to harness AI’s potential while ensuring that increasingly powerful systems remain safe and serve humanity.