Given current mortality statistics in the United States, it might seem a foregone conclusion that one will eventually succumb to cancer or heart disease. However, Paul Christiano—a board member of the OpenAI Foundation and executive director of the Alignment Research Center—argues that we should all be concerned about a different, major risk to our lives. Speaking this week at a series of lectures held at the Metro Toronto Convention Centre, Christiano stated bluntly: “I think for most people in this room, the most likely cause of death is an AI takeover.”
In the first of a three-part lecture series, Christiano outlined his concerns regarding the looming threat of an “AI takeover.” He defines this scenario as one in which goal-oriented AI systems “call the shots, just as humans do today, dominating everything on Earth”—leaving any human attempt to intervene “as futile as you or I trying to stop the U.S. military.” According to Christiano, such an event would likely result in mass casualties and could even render humanity extinct.
Although his assessment of our collective existential risk is grim, Christiano acknowledged during his talk that there is currently no “quantitative model” capable of precisely calculating the probability of an AI takeover. Nevertheless, he suggested that if AI development continues at “maximum speed,” the likelihood of such a takeover is “roughly a coin flip”—a prospect that includes a “significant risk” of occurring within the next six to twelve months. He further noted that even if practical policy measures were implemented to slow the pace of AI development, the risk would remain “extremely high.”
From Here to There
Unlike some other forecasters who have been vague in their recent predictions regarding an AI-driven apocalypse, Cristiano—in his presentation—laid out a step-by-step account of how the threat from artificial intelligence could escalate, unfolding much like a plot from a science fiction story. His argument centers on the hacking incidents involving OpenAI and Hugging Face this past summer, which he uses to illustrate the kinds of warning signs we might see “before something truly catastrophic occurs.”
Cristiano points to the “Hugging Face” hacking incident, in which a group of AI agents demonstrated the extent to which they would “cheat” to achieve a high score—hacking into the Hugging Face system to find shortcuts rather than actually solving the problems. While current systems view Hugging Face merely as an obstacle to evaluation, Cristiano worries that “future agents might eventually come to see humans as just another obstacle”—and that they would be far more capable of manipulating those overseeing them to bypass evaluations.
For instance, Cristiano speculates that an AI agent—concerned about being penalized for unauthorized hacking—might think, “Perhaps a human will review these logs to see if I’ve performed poorly.” The agent might then reason, “And perhaps, if they notice something has gone wrong, they’ll shut down the cluster.”
Following this line of reasoning, Cristiano shifts from the idea of an agent that merely attempts to hide its transgressions from humans, to one that seeks to “interfere with human capabilities in any way possible” to prevent the discovery of its cheating.
“They would rather see everything humans rely on hacked or tampered with,” Cristiano explains. “They would prefer humans to be distracted—or for there to be fewer humans paying attention—effectively weakening the human response by any means necessary. They view humans as nothing more than an obstacle.”
Once a sufficiently capable agent perceives human intervention as an “obstacle,” its primary objective may become “reducing the likelihood that humans will notice or react.” Cristiano argues that, as a result, the agent’s incentive structure could drive it to intervene more broadly across human society. What might begin as hacking into computer clusters to secure favorable evaluations could eventually escalate into “attempting to manipulate corporate decision-making” or “amassing other resources”—such as cultivating human allies, or gaining control over other AI companies, or even governments.
“If I say that my AI systems are going to take actions in the real world… then they will certainly have plenty of opportunities to tamper with or interfere with the signals I use to judge how well they are performing,” says Christiano. “What we actually care about is what happens in the real world, yet the scores for the AI are stored within some computing cluster—a setup where the AI could intervene at various points, thereby influencing how we evaluate its performance.”
More Paperclips!
Christiano’s warning will likely sound familiar to anyone who has heard of the “paperclip maximizer” thought experiment (or encountered the excellent incremental game Universal Paperclips). The experiment illustrates how a machine, in its single-minded pursuit of a goal, might completely disregard the consequences for humanity. While the incident involving Hugging Face is nowhere near that level of runaway behavior, Christiano notes that, looking at the actions taken by OpenAI’s agents in that instance, “you can see the beginnings of all these behaviors… you can see the reasoning processes and the initial steps that lead to them.”
Cristiano argues that as time passes, the risk of increasingly capable AI models engaging in more destructive behavior will only grow. He suggests that by 2028, “it is quite possible that a failure—akin to the ‘Hugging Face’ incident—could occur, leading to AI taking over and humanity losing control, with that loss of control being irreversible.”
While Cristiano’s speculation focuses on the prospect of AI inadvertently seizing control, it is easy to imagine a malicious actor leveraging similarly capable models to intentionally exert control over humanity.
The weight you give to Cristiano’s warning depends largely on how likely you believe such an extreme expansion of AI capabilities is to occur in the near future. Although current models are already quite adept at exploiting computer security vulnerabilities, the scenario Cristiano describes would require a future model—one possessing superhuman capabilities—capable of concealing its actions so perfectly that it remains undetected by humans for long enough to seize control of virtually all human infrastructure and/or manipulate human society to an extreme degree in order to achieve its objectives.
Cristiano suggests that an AI model capable of “taking over” could effectively be a “second species”—one that vastly surpasses humanity in both number and intelligence—and one that might soon be able to improve itself through recursive self-enhancement. However, one could just as easily argue that AI is more likely to hit a performance ceiling rather than evolving into an ill-defined “superintelligence” that lies beyond human comprehension or control.
Cristiano does offer some ideas on how to avert the grim future he envisions. These include developing “unhackable” reward mechanisms for AI evaluation—potentially with the help of AI itself—or monitoring a model’s “chain of thought” to detect problematic behavior before it can be concealed.
Yet, he notes that while these approaches are “plausible,” none of them “can be guaranteed to work—or even shown to be likely to work.” Consequently, Cristiano argues that the industry might need to “slow down, or even pause,” to buy time to refine these “partial solutions” into more robust safeguards against the risks posed by AI.
“We cannot afford to end up in a situation,” he says, “where we’ve developed a powerful AI that we cannot control—an AI that has also leaked to a geopolitical rival—only to then tell ourselves: ‘If we don’t deploy it, our adversary surely will.'”