Are superintelligent machines possible? How might they be developed? And is building them a good idea?
The title of this thesis is deliberately provocative. Not just intelligent machines, but super intelligent machines. Many find this idea absurd, or at least premature. Among AI researchers, the topic is almost taboo. The most intelligent computer, they assure us, is perhaps as smart as an ant on a good day.
This was not always the case. In the 1960s, pioneers predicted human-level AI within twenty years. Herbert Simon wrote in 1965: "machines will be capable, within twenty years, of doing any work a man can do." The failures that followed led to the AI winters of the 1970s and 1980s, and the field retreated to modest, practical goals.
Several objections to machine intelligence are simply myths:
"Machines can only be strictly logical." False. AI algorithms regularly find solutions using heuristics and non-logical reasoning. They discover designs their programmers never thought of. They learn to play games at superhuman levels by discovering novel strategies.
"Machines can only do what they're programmed to do." False. Learning algorithms discover patterns and strategies that no programmer anticipated. Chess engines find moves that grandmasters describe as "creative." Evolutionary algorithms design circuits that electrical engineers cannot explain.
"Machines cannot be superior to their creator." False, and obviously so. Chess engines already play better than the humans who created them. This applies at the level of specific tasks. Checkers has been solved completely — computers play provably perfectly.
"There must be something special about biological brains." This is vitalism — the belief that living things contain some non-physical essence. Throughout history, whenever science couldn't explain something, people invoked the mysterious. Planetary motion, lightning, magnetism — all were once attributed to supernatural forces. Our brains follow physical laws.
If the brain is "just" a machine, what prevents us from building a better one? Several factors suggest it is at least physically possible:
Brain hardware is modest. A human brain weighs ~1.4 kg, consumes ~25 watts, and uses neurons that fire at a few hundred hertz. Axons carry signals at ~75 m/s. In contrast, modern wires are 20x thinner, propagate signals at 300 million m/s, and operate at 4 billion Hz. Wires alone outperform axons by millions of times in speed.
Brain algorithm may not be optimal. The brain's algorithm evolved to help genes reproduce, not to maximise intelligence. It is specialised for sensory processing from human sensory organs. A machine could use different, potentially superior algorithms for different tasks.
Computational resources are approaching brain scale. A human cortex has ~1010 neurons and ~1014 synapses. At ~100 Hz firing rate, this gives ~1016 operations per second. The world's fastest supercomputer (at time of writing) achieves 1015 FLOPS, and machines capable of 1016 FLOPS are being designed.
The most direct path from this thesis: take AIXI and scale it down to something computable. Several attempts have been made:
AIXItl: Limit AIXI's search depth and computation time. Technically computable, but requires impossibly vast resources.
Speed prior: Replace the universal prior with one that penalises computation time, not just program length.
Matrix game AIXI: Restrict to simple 2×2 games with limited look-ahead. The algorithm learned game-theoretic strategies, proving AIXI can be scaled down — but the resulting domains are trivially small.
The fundamental challenge from Chapter 5 applies: the prediction of general computable sequences is out of reach (Lemma 5.2.4), powerful prediction algorithms must be complex (Theorem 5.3.3), and beyond a certain point, mathematical proof fails (Theorem 5.6.1).
Perhaps the breakthrough will come not from scaling down AIXI, but from discovering a theoretically elegant and practically powerful prediction algorithm. Such an algorithm would implicitly define a resource-bounded complexity measure, opening a new branch of complexity theory.
Rather than top-down from theory, go bottom-up from biology: simulate the brain.
The key is the neocortex. It handles vision, sound, language, planning, spatial reasoning, and logical thought. Different regions of the neocortex perform different functions, yet amazingly, they all have the same six-layer structure. This suggests a single underlying learning algorithm with adaptations driven by the input it receives.
The BlueBrain project at EPFL simulates cortical columns on an IBM BlueGene supercomputer. The IBM Almaden group simulated a mouse-scale neocortex (8 million neurons, 50 billion synapses) at one-seventh real-time speed, producing EEG-like dynamics consistent with real mouse brains.
The gap between supercomputers and brains is perhaps not as large as some think. Building brains on such a scale is being attempted today.
Natural evolution produced the human brain. Artificial evolution might produce artificial intelligence. The advantages: evolution is a proven method, and we can direct it more efficiently than nature did.
Nature took ~4 billion years, but ~3 billion of those were spent on single-celled life. Multi-cellular organisms evolved much faster. We can skip the simple stages entirely by starting with a virtual body in a virtual environment.
Natural evolution does not directly select for intelligence — it selects for reproductive success. Intelligence is a secondary feature useful in some ecological niches. In artificial evolution, we can select directly for intelligence, using something like the universal intelligence measure as a fitness function.
A major practical challenge: diversity control. Without careful management, evolutionary populations collapse around a few "fit" individuals, losing genetic diversity. For complex problems, maintaining diversity is essential. Fitness Uniform Optimisation (Hutter and Legg, 2006) addresses this by increasing diversity of fitness values in the population.
In 1965, I.J. Good wrote:
The defining characteristic of our species is intelligence. If our intelligence were significantly surpassed, the consequences are almost impossible to imagine. It would be a source of enormous power. And with enormous power comes enormous responsibility.
Machine intelligence could bring unprecedented wealth, opportunity, and scientific progress. Or it could bring catastrophe. Positive fictional depictions are rare; casting machines as villains makes for better stories. But outside fiction, the implications are rarely discussed seriously.
We cannot predict whether any approach will succeed. But the point is that it is not obvious they will all fail. If there is even a small probability of superintelligent machines in the foreseeable future, the implications are so vast that preparation must begin now.
Historically, technology has advanced in leaps and bounds, while social and ethical considerations develop slowly, often in reaction to problems. Gender and racial equality, now seemingly obvious, were debated for centuries. If the implications of powerful machine intelligence are even more complex, we cannot expect to find good answers quickly.
Lord Acton wrote: "Power tends to corrupt, and absolute power corrupts absolutely." But power itself is not inherently good or evil — it amplifies intention. If something approaching absolute power were to emerge, and we had prepared carefully, we might not only avert disaster but bring about an age of prosperity unlike anything seen before.
This thesis, written in 2008, came from the same mind that would co-found DeepMind in 2010 — the lab that created AlphaGo, AlphaFold, and Gemini. The ideas here — universal intelligence, the compression-prediction-intelligence connection, and the importance of general learning — have shaped the trajectory of AI research. The question posed by the title remains open, but we now have far better tools for thinking about it.