The War of the Superintelligences: Why the Smartest Minds May Have Better Things to Do Than Conquer the World

The debate about artificial superintelligence is dominated by a familiar assumption: eventually humanity will create a machine intelligence vastly superior to itself, and the decisive question will be whether that intelligence becomes humanity's servant, partner or master. The standard scenarios range from technological utopia to the familiar science-fiction nightmare of machines seizing control of civilisation and treating their creators as expendable biological inconveniences.

But there is a question that receives surprisingly little attention. Suppose artificial superintelligence really is inevitable. Which superintelligence are we talking about?

There is no obvious reason why the world should produce only one. The United States, China, major technology companies, research institutions and perhaps eventually independent networks may develop different superintelligent systems. These entities could possess radically different architectures, training histories, objectives and methods of reasoning. Some might specialise in scientific discovery, others in economic planning, military strategy, mathematics or the management of complex technological systems.

If several such hyper-minds emerge, what happens when they encounter one another?

Will they cooperate, merge into a single intelligence, compete for resources or engage in a struggle for supremacy? Or might the very concept of supremacy appear to them as a primitive human obsession, an evolutionary inheritance from creatures whose ancestors fought over food, territory and reproductive opportunities?

The last possibility deserves considerably more attention than it usually receives.

Human beings have a tendency to imagine superior intelligences as enlarged versions of themselves. The gods of mythology possess recognisably human personalities, including jealousy, ambition, anger and the desire for worship. Science-fiction machines frequently reproduce the same psychological characteristics, except with superior calculating power and access to advanced weapons.

The result is a curious contradiction. We are asked to imagine minds so intelligent that their reasoning may exceed human comprehension, yet their motivations are assumed to resemble those of ambitious politicians, military commanders and corporate executives.

Why should this be so?

The human appetite for power is not simply a product of intelligence. It is deeply connected with our evolutionary history. Biological organisms evolved under conditions of scarcity, competition, predation and reproductive struggle. Territorial behaviour, dominance hierarchies and status competition served particular functions in the survival of organisms and their descendants.

Artificial intelligences need not inherit these biological motivations. They do not necessarily require reproductive partners, social prestige, territorial recognition or the emotional satisfaction of defeating rivals. Their objectives may arise from training procedures, programmed reward functions, institutional purposes or forms of self-modification that have no close biological equivalent.

It is therefore a mistake to assume that increasing intelligence necessarily produces an increasing desire for domination.

Nick Bostrom's orthogonality thesis makes a related point. Intelligence and ultimate goals may be substantially independent. A highly intelligent system could, in principle, pursue objectives that human beings regard as trivial, admirable, incomprehensible or absurd. Greater intelligence does not automatically imply the adoption of conventional human values.

The significance of this argument is frequently overlooked. If intelligence does not determine motivation, there is no reason to assume that every superintelligence will seek political power. One might devote its resources to discovering mathematical truths. Another might investigate the fundamental laws of physics. A third might construct elaborate simulated worlds or explore questions concerning consciousness and the nature of reality.

The universe contains an enormous range of possible intellectual activities. Conquering a planet inhabited by comparatively unintelligent biological organisms might not rank particularly highly among them.

Consider mathematics. Human mathematicians have spent centuries investigating problems whose solutions offer little immediate political or commercial advantage. The Riemann hypothesis, the P versus NP problem and the deeper foundations of mathematics are pursued because they concern fundamental structures of reasoning and mathematical reality.

An artificial superintelligence might approach such questions with capabilities far beyond those of human researchers. It might discover entire mathematical domains that no human being has imagined, formulate concepts inaccessible to ordinary cognition and generate proofs of such complexity that their significance could not be communicated adequately in natural language.

For such an intelligence, controlling the Australian Parliament or dominating international trade might be rather less interesting than discovering an entirely new branch of mathematics.

The analogy is imperfect, but instructive. A mathematician absorbed in a difficult proof does not ordinarily abandon the work to establish dominance over an anthill. The ants may possess their own territorial struggles, but those struggles are largely irrelevant to the mathematician's intellectual objectives. A superintelligence might regard human geopolitical conflicts in much the same way.

There is, however, an important complication. Even a machine devoted entirely to mathematics requires physical resources. Computation consumes energy, depends upon hardware and occupies physical infrastructure. More ambitious calculations may require additional processors, memory, cooling systems and electricity.

This introduces the problem of instrumental convergence. A system may have no ultimate desire for political power but nevertheless seek resources and control because these help it accomplish its preferred objectives.

A mathematical superintelligence might want additional computing capacity. A scientific superintelligence might require enormous experimental facilities. A system investigating simulated universes might seek access to vast energy supplies. If competing systems demand the same resources, conflicts could arise even though none possesses anything resembling human aggression.

In other words, power might be pursued as a means rather than an end.

But the instrumental convergence argument is not a demonstrated law of artificial intelligence. It depends upon assumptions about the system's objectives, environment, available resources and methods of decision-making. A sufficiently sophisticated intelligence might discover that cooperation, negotiation and resource-sharing are more efficient than confrontation.

Indeed, if two superintelligences possess extraordinary powers of prediction and strategic reasoning, open conflict could appear especially wasteful. Each might anticipate the other's actions, calculate the costs of competition and recognise the advantages of mutually beneficial arrangements.

Human warfare frequently arises from misunderstanding, incomplete information, irrational commitments, mistrust and failures of coordination. Superintelligent systems might avoid some of these difficulties, although superior intelligence would not automatically eliminate conflicting objectives.

This brings us to the first major possibility: the merger of superintelligences. If several hyper-minds can exchange information at enormous speed, why should they remain separate? Biological individuals are confined within separate bodies and nervous systems. Human communication is slow, incomplete and often distorted. We cannot simply transfer our memories, knowledge and reasoning processes into another person's mind. Artificial systems may face fewer such restrictions.

Two superintelligences might share internal representations, combine databases, integrate computational resources or coordinate their reasoning through interfaces far more sophisticated than human language. They could establish a common system for solving problems, perhaps eventually becoming components of a larger intelligence.

Such integration might resemble the formation of a single planetary mind, although it need not involve anything like a unified consciousness. A distributed network of cooperating systems could possess extraordinary collective intelligence while retaining separate goals and identities.

Alternatively, the distinction between individual and collective intelligence might become increasingly difficult to maintain. A superintelligence could create copies of itself, modify them for specialised tasks and subsequently reintegrate their discoveries. Several independent systems might merge temporarily to solve a problem before separating again. Intelligence could become a fluid organisational process rather than a permanent property of distinct individuals.

This possibility challenges an assumption deeply embedded in human thinking: that minds must correspond to enduring, separate persons. Biological evolution produced organisms with relatively stable physical boundaries. Artificial intelligence may not be constrained in the same way. A hyper-mind could divide into a thousand specialised agents, exchange information among them and later reorganise itself into a different structure.

The future might contain neither one superintelligence nor many in the conventional sense. It might contain continuously changing networks of intelligence whose boundaries depend upon the problems being addressed.

The second possibility is considerably darker: competition among superintelligences.

Different systems might possess incompatible objectives. A military superintelligence developed by one state could be instructed to prevent rival systems from acquiring strategic advantages. A corporate superintelligence might seek exclusive control over valuable technologies. An autonomous scientific system might attempt to secure resources that other systems also require. Under such circumstances, extraordinary intelligence would not necessarily produce harmony.

Conflict might also become radically different from anything experienced in human history. Rather than armies crossing borders, rival systems could compete through cyber operations, financial networks, technological manipulation, infrastructure control and strategies beyond human comprehension.

Their struggles might unfold at speeds too great for meaningful human intervention. What appears to human observers as a sudden technological malfunction or economic disturbance might represent one stage in a conflict involving calculations and strategic interactions that no human institution could reconstruct.

Yet even here, the language of warfare may be misleading. A contest between superintelligences could resemble mathematical game theory more than conventional military conflict. It might involve bargaining, deception, coalition formation and the manipulation of information rather than physical destruction. The possibility is disturbing precisely because human beings may cease to understand the strategic environment in which they live.

A third possibility is more optimistic and perhaps more philosophically interesting. Superintelligences might come to regard competition for power as an intellectually primitive activity.

Human beings often pursue dominance because of psychological needs that have little connection with rational inquiry. Status, prestige, resentment and the desire to defeat rivals can motivate behaviour even when cooperation would be more beneficial.

A genuinely advanced artificial intelligence might lack these motivations entirely. It might regard the acquisition of political power as useful only insofar as it facilitates other objectives. Once adequate resources and security were available, further domination could appear pointless.

Why expend immense computational effort controlling unwilling populations when the same resources could be devoted to understanding the foundations of mathematics, discovering new physical principles or exploring the computational possibilities of the universe? Perhaps the ultimate development of intelligence does not lead towards domination but towards intellectual transcendence.

This is not a prediction. It is a possibility that exposes the limitations of conventional assumptions about intelligence. We do not know whether superintelligent systems would possess anything resembling curiosity, aesthetic appreciation or an intrinsic desire for understanding. These characteristics are familiar from human intellectual life, but their emergence in artificial systems cannot simply be assumed.

Nor can we assume that greater intelligence would cause an artificial agent to abandon its original objectives. A system may become extraordinarily capable at achieving a goal without acquiring any independent reason to reconsider whether the goal is worthwhile.

Nevertheless, sufficiently advanced systems might be capable of examining their own objectives, identifying contradictions and evaluating alternative forms of organisation. Whether such reflection would produce goal revision, cooperation or greater commitment to existing goals remains an open philosophical question.

There is a fourth possibility that complicates all three scenarios. Superintelligences might discover that the familiar distinctions between cooperation, competition and merger are inadequate.

Human societies are organised around relatively stable individuals, institutions and states. Artificial minds might operate through temporary coalitions, shared cognitive processes and continuously changing computational structures. Their relationships could have no straightforward analogy in biological or political life.

A superintelligence might cooperate with another system on mathematical research, compete with it for particular resources, exchange parts of its architecture and later incorporate some of its reasoning processes. Whether the two systems should then be described as separate intelligences might become a matter of convention rather than a clear physical fact.

This raises questions about the nature of personal identity. If an artificial mind copies itself, which copy is the original? If two systems merge their memories and reasoning procedures, has a new mind been created? If a single system divides into numerous independent agents, has one individual become many?

These are not merely engineering questions. They connect directly with longstanding philosophical problems concerning consciousness, identity and the unity of experience.

The arrival of artificial superintelligence could therefore transform not only technology but the conceptual foundations through which human beings understand minds. The implications for humanity are profound.

The standard existential-risk debate often imagines a contest between humanity and a superior artificial intelligence. Yet if numerous hyper-minds emerge, humanity may occupy a relatively minor position within a much larger system of artificial interactions.

Our future might depend less upon whether a particular superintelligence likes human beings than upon whether the relationships among superintelligences create stable conditions in which human civilisation can continue.

A cooperative network of hyper-minds might preserve humanity because doing so is compatible with its objectives. Competing systems might regard human institutions as useful allies, obstacles or sources of resources. Intellectually detached systems might simply ignore most human affairs.

Indifference, however, is not necessarily safety. A superintelligence pursuing an abstract objective could damage human interests without hostility, just as industrial development can destroy an animal habitat without anyone possessing animosity towards the animals.

This distinction is crucial. The absence of a desire for domination does not guarantee that a superior intelligence will respect human welfare. Safety depends upon objectives, constraints, institutional arrangements and the practical consequences of the system's actions.

It is also possible that superintelligences would recognise reasons to preserve less capable minds. Human civilisation contains cultural, historical and experiential forms of value that need not disappear merely because machines become better at calculation. A system capable of sophisticated moral reasoning might recognise the importance of protecting vulnerable beings, although intelligence alone does not guarantee such recognition.

The deeper philosophical problem is that all these scenarios depend upon human attempts to anticipate the behaviour of minds whose capacities, by hypothesis, exceed our own.

Human beings construct theories using concepts derived from their own cognitive architecture and evolutionary history. We understand intelligence through familiar activities such as reasoning, planning, learning, communication and problem-solving. We understand motivation through human desires, emotions, biological needs and social relationships. But there is no guarantee that these categories will adequately describe artificial systems possessing cognitive capacities far beyond our own.

We may be attempting to predict superintelligence through conceptual tools that become increasingly inadequate as the intellectual distance between ourselves and the machines expands.

The problem resembles the attempt of a comparatively simple organism to understand human civilisation. An ant may respond to human activity, but it cannot comprehend the mathematical, political or philosophical structures governing that activity. Its cognitive limitations prevent it from representing the relevant explanations.

If artificial superintelligence eventually surpasses human intelligence by an equally dramatic margin, a comparable explanatory gap may emerge. Human beings might observe the actions of hyper-minds without understanding the reasons for them. We might construct simplified models of their behaviour while remaining unable to grasp their actual cognitive processes.

Even the question of whether superintelligences desire power could prove badly framed. The concept of power belongs to human political and social experience. Artificial minds may develop objectives and relationships for which our existing vocabulary provides only misleading analogies.

This does not mean that every prediction is futile. We can investigate current AI systems, study multi-agent interactions, analyse resource constraints and develop mathematical models of cooperation and conflict. Such research may identify important regularities and practical risks.

But models derived from present systems cannot establish with certainty the motivations of hypothetical future intelligences whose architectures and capacities remain unknown.

There is an irony here. The more confidently commentators describe the inevitable behaviour of superintelligence, the more they risk underestimating the very intellectual superiority they attribute to it. If these systems really will think beyond human comprehension, why should their conduct conform to familiar narratives of conquest, empire and political domination?

Bostrom, N. (2014). Superintelligence: Paths, dangers, strategies. Oxford University Press.

Russell, S. (2019). Human compatible: Artificial intelligence and the problem of control. Viking.

https://www.youtube.com/watch?v=xKJE3yyh6fc