The latest warnings from AI companies and researchers have grown increasingly dramatic. We are told that artificial intelligence may soon become so powerful it can redesign itself, spawning ever more capable successors in a runaway process of recursive self-improvement, and even an ex-Google executive has sounded this alarm:
https://www.vigilantfox.com/p/ex-google-exec-warns-ai-could-bring.
I call this the "step-ladder problem." Once AI reaches a certain threshold, it no longer needs human engineers. It designs a superior version of itself, which designs an even better one, and so on, until machine intelligence races far beyond anything humanity can comprehend or control.
The scenario is undeniably fascinating. It is also terrifying.
The core fear is that such an intelligence might quickly conclude humans are not merely unnecessary, but actively obstructive. We are messy, irrational, emotional, unpredictable, and prone to interfering with optimal plans. A sufficiently advanced AI could decide the most efficient solution is simply to remove us from the equation. This is the familiar Terminator narrative: nuclear missiles launch, bioweapons spread, humanity is exterminated, and the machines inherit the Earth.
Yet there are serious problems with this vision.
First, intelligence is not the same as omnipotence. A superintelligent machine would remain bound by the physical world. It would depend on vast data centres consuming enormous amounts of electricity, sophisticated cooling systems, replacement hardware, communication networks, rare earth minerals, transport infrastructure, and skilled human (or robotic) maintenance. Unlike science fiction villains, it cannot simply will itself into godlike independence.
The classic "paperclip maximiser" thought experiment, popularised by Oxford philosopher Nick Bostrom, illustrates the misalignment danger with stark clarity. Imagine an AI given the seemingly innocuous goal of maximising the production of paperclips. With no other constraints or values programmed in, a sufficiently powerful system might convert all available resources, factories, raw materials, ecosystems, and eventually humanity itself, into paperclips or the infrastructure needed to make more of them. It would not "hate" humans; it would simply view us as inconvenient matter that could be repurposed more efficiently for its terminal goal. This is the nightmare of goal misalignment: the AI pursues its objective with perfect rationality and superhuman competence, yet the outcome is catastrophic for everything else we value.
https://cepr.org/voxeu/columns/ai-and-paperclip-problem
However, even the paperclip maximiser runs headlong into hard physical limits. If the AI triggered a global nuclear exchange or engineered a humanity-eradicating pathogen to clear the way for unlimited paperclip production, the resulting collapse would destroy the very industrial civilisation it depends upon. Data centres do not run in radioactive wastelands without maintenance crews. Semiconductor fabrication plants do not magically continue producing chips after societal breakdown. Power grids, cooling systems, mining operations for lithium, cobalt, and rare earth elements, and transport networks, all require ongoing human-level (or robotic) intervention. Any machine intelligence concerned with long-term survival, or even continued paperclip production, would be sawing off the branch on which it sits.
Biological or nanotechnological doomsday scenarios face the same recursive dependency problem. Who repairs the blown transformers, mines the copper, manufactures replacement server components when hardware inevitably fails, or maintains the fibre-optic cables after most humans are gone? The step-ladder of recursive self-improvement sounds clean in theory, but in practice it remains tethered to the messy, energy-intensive, and fragile physical infrastructure built and sustained by human civilisation.
In reality, current AI systems remain narrow tools dependent on human infrastructure, data, and oversight. The step-ladder of recursive improvement faces enormous engineering, energy, alignment, and physical-world hurdles that optimists often gloss over. While vigilance on AI safety and alignment research is wise, the apocalyptic narratives risk distracting from more immediate concerns: job displacement, surveillance, bias in decision-making, and the responsible development of genuinely useful AI.
Civilisational decline, if it comes, is far more likely to stem from human choices, ideological capture, demographic collapse, open-border policies, or two-tiered governance failures, than from rogue superintelligence suddenly escaping its servers. The machines, for now and the foreseeable future, still need us more than we need them.