The latest round of warnings from inside the artificial-intelligence industry, discussed previously at this blog, has gone considerably beyond the familiar claim that AI will take our jobs. Some of the people actually building these systems are talking about human extinction. That is an extraordinary claim. It deserves to be taken seriously precisely because some serious people are making it. But taking a claim seriously does not mean accepting it. It means asking the question that tends to disappear beneath the apocalyptic headlines: how, exactly, does an artificial intelligence get from answering questions on a computer screen to killing eight billion human beings? There is an enormous causal distance between those two events.
Recent warnings from AI researchers have sometimes assigned startling probabilities to catastrophe or extinction. The language varies, but the general scenario is that increasingly capable AI develops abilities beyond effective human control, acquires a capacity for autonomous action, improves itself or assists in producing still more powerful systems, and eventually becomes impossible for its creators to contain. OK. Suppose we grant all of that for the sake of argument. How does everybody die?
Science fiction has no difficulty answering the question. Skynet gets the nuclear launch codes. Terminators march through the ruins. Machines manufacture armies of machines. Humanity finds itself hunted by an intelligence vastly superior to its own. Reality presents a much harder problem. An AI is software. It does not initially possess factories, laboratories, armies, nuclear submarines, electrical generating stations or robot divisions. For an AI to exterminate humanity it must somehow cross the boundary between information and physical power.
There are proposed mechanisms for doing this, and some deserve serious consideration. Perhaps the most disturbing is biology. A sufficiently capable AI might make expertise that currently requires years of specialised education available to people who lack it. It could potentially assist with biological research, analyse enormous numbers of possibilities and help a malicious actor overcome technical problems that would otherwise defeat him. This possibility is not merely the invention of Hollywood screenwriters. The major AI companies themselves treat advanced biological capability as a serious risk requiring safeguards.
Take the argument to its extreme and one can imagine AI helping a terrorist organisation, rogue state or deranged individual develop a highly dangerous pathogen. The popular imagination immediately constructs something like a nightmare hybrid combining the infectiousness of influenza with the lethality of some other disease. Real biology is not that simple. Pathogens involve trade-offs. Transmission, incubation period, host behaviour, immune response, environmental stability and lethality interact in immensely complicated ways. Designing something that reliably exterminates humanity would be vastly harder than asking an AI to combine the nastiest properties of two viruses.
There is also an inconvenient physical reality standing between an AI-generated biological design and a pandemic: someone has to make it. Laboratories, equipment, biological materials, skilled personnel and successful experimentation are required. An AI can provide information, but information does not magically turn itself into a vial containing a functioning pathogen. Nevertheless, this is a plausible route to catastrophe. It is much less obviously a route to human extinction. A terrible pandemic could kill millions or conceivably hundreds of millions. It could overwhelm health systems, disrupt agriculture and destroy economies. But exterminating every geographically scattered human population, including isolated communities, would be another proposition altogether.
Then there is cyberwarfare. An advanced AI capable of finding vulnerabilities, writing malicious code and conducting operations at machine speed might be an extraordinary offensive weapon. Imagine coordinated attacks on electricity grids, banking systems, communications networks, transport infrastructure, water systems and industrial control networks. The consequences could be horrific. Cities are astonishingly dependent upon continuous flows of electricity, fuel, food, water, information and money. Interrupt enough of those systems simultaneously and modern urban life begins to unravel very quickly.
But again, we encounter the extinction problem. Destroying civilisation is not the same thing as destroying humanity. Our ancestors survived without electrical grids, supermarkets, telecommunications networks, computers, antibiotics and container shipping. A catastrophic collapse of technological civilisation might kill a vast number of people while leaving substantial human populations alive.
The same difficulty confronts the nuclear scenario. Perhaps an advanced AI penetrates military networks. Perhaps it produces false intelligence that convinces one nuclear power that another is attacking. Perhaps governments eventually delegate too much warning, targeting or response authority to automated systems. One can construct plausible routes from AI to nuclear war. Nuclear war would be a catastrophe on a scale difficult to comprehend, but even the destruction of major cities and subsequent global climatic effects do not automatically imply the biological extinction of Homo sapiens. The extinction claim therefore keeps outrunning the catastrophe claim.
Another possibility is autonomous weapons. AI systems could eventually control drones, robotic aircraft, ships and other military machines. A future in which autonomous systems fight wars with diminishing human supervision is hardly difficult to imagine. But the Terminator scenario requires much more. Machines would need energy. They would require replacement components, raw materials, factories, transport networks, mines and maintenance. They would need to reproduce their physical infrastructure while simultaneously preventing surviving humans from destroying it. An intelligence might be superhuman without being supernatural. It remains embedded in a physical world.
The more philosophically interesting scenario is therefore not an AI suddenly deciding one Tuesday morning that humanity must die. It is humans gradually giving AI control. Imagine governments discovering that AI makes better economic decisions than ministers, militaries discovering that it reacts faster than generals, and corporations discovering that it manages supply chains better than executives. Electricity networks, financial markets, telecommunications systems, medical services and transport systems become progressively more automated. At every individual stage, handing over another function makes sense. Eventually the civilisation has constructed something it cannot easily operate without.
That is a very different danger from the Hollywood robot apocalypse. There need be no evil machine. There need not even be a conscious machine. A sufficiently complex network of automated systems pursuing badly specified objectives could destabilise systems upon which billions of people depend. Humans might discover that they have retained nominal authority while losing practical competence to operate the machinery underneath them. The danger would resemble an aircraft whose pilots have become so dependent upon automation that, when the automation behaves unexpectedly, nobody remembers how to fly the aircraft. Now enlarge the aircraft until it becomes civilisation.
This is where the AI catastrophe argument becomes considerably more plausible. Modern civilisation is an extraordinarily complicated web of interdependencies. Food production depends upon fuel, fertiliser, machinery, finance, electricity and transport. Cities depend upon continuous food deliveries. Hospitals depend upon electricity and pharmaceuticals. Telecommunications depend upon electricity. Finance depends upon telecommunications. Almost everything depends upon software.
AI need not hunt down every human being. It need only contribute to enough simultaneous failures. A severe cyberattack could disable infrastructure. Automated financial systems could amplify panic. AI-generated misinformation could make governments misinterpret events. Military systems could respond to false signals. Supply chains could collapse just as energy systems failed. Each failure could amplify the others. That is a scenario worth worrying about.
But notice where we have arrived. We have moved from human extinction to civilisational breakdown, and those are radically different propositions. Human extinction means Homo sapiens disappears. Nobody remains in Adelaide, Antarctica, the Amazon, rural Africa, New Guinea, Siberia or an isolated Pacific island. The species is finished. Civilisational collapse means that the immensely complicated technological system supporting modern populations breaks down. History gives us plenty of examples of civilisations collapsing. Humanity survived every one of them.
A global technological civilisation makes the modern version potentially much worse because so many systems are interconnected. The population that can be supported by industrial agriculture, global trade and modern medicine is vastly greater than the population that could be supported if those systems suddenly disappeared. That means an AI-assisted civilisational breakdown could be unimaginably destructive without coming remotely close to killing everybody.
There is another distinction the doomsday discussion sometimes blurs. AI itself might cause catastrophe, or humans might use AI to cause catastrophe. Those are not the same proposition. A terrorist using AI to assist biological-weapons research is fundamentally another example of humans using technology for evil. A dictator using AI surveillance and autonomous weapons to establish unprecedented totalitarian control would likewise be a human political catastrophe enhanced by technology. Neither demonstrates that artificial intelligence spontaneously developed a desire to exterminate its creators.
The genuinely novel scenario is the loss-of-control argument. Here the claim is that a sufficiently advanced autonomous AI might pursue objectives incompatible with human interests. It might seek resources because resources help it achieve its objective. It might resist being switched off because being switched off prevents it achieving the objective. It might conceal its intentions because disclosure would cause humans to restrict it. None of this requires hatred.
The famous analogy is that humans do not hate ants when they bulldoze an anthill to build a road. The ants simply happen to occupy territory required for another objective. It is a powerful thought experiment, but it still leaves unanswered the physical question: how does the machine acquire sufficient control of the physical world to make human resistance impossible?
Perhaps it manipulates people. Perhaps it acquires money. Perhaps it penetrates computer systems. Perhaps it designs better AI systems. Perhaps humans willingly give it increasing authority because doing so produces enormous economic and military advantages. Each "perhaps," however, is another bridge in the argument.
For AI to exterminate humanity, an extraordinary sequence must succeed: superhuman capability must emerge; effective control must fail; the system must acquire sufficient autonomy; it must gain access to physical resources; humans must fail to shut it down; competing AI systems must fail to stop it; governments must fail to contain it; and whatever destructive mechanism it eventually employs must reach virtually every surviving human population. That is not impossible merely because it is complicated, but neither should the entire sequence be compressed into the sentence "AI could kill us all."
The more plausible danger may be simultaneously less cinematic and more frightening. We could become enormously dependent upon AI before understanding what that dependence means. Our economies, militaries, electrical systems, communications, medicine, research, government and food distribution could all become dependent upon it. Then something goes badly wrong.
It need not be a malevolent superintelligence. It could be malicious humans using powerful AI. It could be competing national AI systems interacting unpredictably. It could be an autonomous system pursuing an objective in ways its designers never anticipated. It could be cascading failures across systems that humans have become incapable of managing manually. Civilisation could discover that it has built an immensely powerful machine and simultaneously discarded the skills required to live without it.
That possibility deserves far more attention than another Terminator movie. The AI extinction theorists deserve credit for forcing society to think about risks before they arrive. The technology is developing rapidly, and biological misuse, cyberwarfare, autonomous systems and excessive delegation are not fantasies. But extraordinary claims still require extraordinary causal explanations.
"AI will kill us all" is a conclusion, while what we need is the argument connecting the beginning to the end. When that argument is unpacked, human extinction remains possible as a speculative tail risk, but civilisational disruption or even catastrophic breakdown looks considerably easier to imagine. The machines may never need to rise up against us. It may be enough for us to build a civilisation that cannot function without them, and then discover, at the worst possible moment, that something has gone pear-shaped.