Will AI lead to superintelligence or just super-automation?

Executive Summary

Engineers from Palo Alto to Beijing are racing to create superintelligence — an artificial intelligence (AI) that can think and reason, with an intellect far superior to our own. But existing state-of-the-art models may be on a path to offer “super-automation,” rather than superintelligence.¹ These existing AIs are typically based on a transformer-based large language model (LLM) capable of natural language processing and understanding, and with the ability to generate content and carry out tasks. As companies pour billions of dollars into semiconductors to unlock ever-greater AI capabilities, it’s important to understand what superintelligence is, what might be needed to achieve it, and whether the latest generation of AIs will be an adequate foundation for these ambitions.

AI to AGI to (safe) SI 

• What is superintelligence? AI refers to a system that passes the Turing Test, i.e., a system that’s able to convince a human interlocutor that it’s human. An artificial general intelligence (AGI) is an AI that displays excellence across all human fields of knowledge.² A superintelligence would be the next step — an AGI that exceeds human capabilities across all, or most, human cognitive tasks, and thus by definition would be an enhanced version of an AGI. The development of superintelligence would, therefore, require at least two steps (or leaps) forward from the current state of the art. Safe superintelligence refers to an AI that has goals and priorities that are aligned with human values, while still demonstrating superhuman ability in most — or all — cognitive tasks.
  
  
• What would superintelligence solve? The types of tasks that a superintelligence deserving of the name would be able to carry out, in my opinion, include: solving mathematical conjectures that have eluded humans (e.g. the Millennium Prize Problems), predicting the weather accurately, say, five years in advance versus five days, and constructing molecules to cure any viral or genetic disease it encounters.
  
  
• Would a superintelligence really “care” about things like developer super-productivity? An enhanced human intelligence (an AGI, say) would likely “care” about enhancing, empowering, and enabling human tasks (e.g. coding) – since it is, after all, a human intelligence. But would a superintelligence? In other words: Would an intellect that is able to perform reasoning tasks at more than a thousand times human speed really only care about performing them a million times faster?³ Human beings are limited by our own imaginations as to what tasks we can envision that a superintelligence could perform, as we are not able to jump out of our own cognitive shadows. Thus, in practical tasks, we humans can only ever imagine scaled or accelerated version of ourselves (e.g. solve faster or better) — something that can be called “super-automation.” This uncertainty as to what tasks really are possible for (or palatable to) a superintelligence raises the question of safety and alignment: How do humans ensure that a superintelligence would still only “care” about human needs (like coding productivity)?

What could be missing to get to (safe) superintelligence?

In my view, there are at least three fundamental research challenges to solve in order to build a safe superintelligence:

• Self-learning (autonomous re-training): Any form of intelligence that handles stationary tasks (e.g. detecting cats in pictures versus predicting the markets) is arguably best called "automation," which means the current generation of AI is on a trajectory to "super-automation," rather than superintelligence. By contrast, intelligence comes into play during regime shifts (e.g. fundamental changes in the market), because these are the situations in which humans must revisit and revise assumptions and reconfigure their reasoning apparatus. Put another way, these are the situations where people must "fine-tune," or (in AI terms) "re-train" themselves. Thus, superintelligence would likely require the ability to re-train itself much better than we as humans are able to re-train ourselves.
  
  
• Structural thinking (reasoning with analogies versus predicting through correlations): Superintelligence will likely require AI architectures that achieve or encode thinking by analogy versus merely capturing statistical correlations (which is still what cutting-edge transformer-based AI does). This would be a paradigm, for example, in which an AI doesn’t just learn probabilities for next-token predictions, but in doing so also sharpens its ability to create metaphors and draw unexpected analogies.⁴
  
  
• Asymmetric ethics (versus ethics of shared rationality): Achieving superintelligence would require solving what may be called the "Solaris problem," i.e. finding practical ways to communicate with forms of intelligence radically different or superior to humans. Think of it this way: How can humankind create a system in which ants can communicate to us what they think is the right way to treat them (and vice versa)?⁵ Fanciful though this may sound, the project of “teaching” a superintelligence deserving of the name what is right and wrong might be similar — where of course, in this case, we humans would be the ants.

Will a superintelligence ever actually be useful or commercial?

Even on the assumption that a safe superintelligence is created, it’s not clear it would prove useful or intelligible (or commercial).

• Solve but verify: Humans’ arithmetic abilities are arguably superintelligent compared to those of an ant. But can humans ever explain to an ant why three times three is nine? Analogously, if we as humans have a (safe) superintelligence, we would likely only feel comfortable using it if we can understand why its answers are correct. In itself, that may be a very time-consuming task. Imagine, for example, that instead of spending 41 years solving a single case of the so-called “busy beaver” problem — a notorious mathematical puzzle that, ironically, essentially asks us to find the most inefficient way possible to calculate something — a superintelligence does it in 41 seconds. But it might still take 41 years for us to verify that the superintelligence’s computation is correct.
  
  
• The operating efficiency frontier: The chart below, which is for illustrative purposes, makes this point. In the long term, it will be interesting to see what the shape of this curve looks like as well as where the equilibrium point (at which time saved by superintelligence equals the extra time it takes for a human to understand why it works) will be. Understanding the shape of this curve as well as developing useful and precise scales for the axes could be a promising direction of research.

• Are explanations for AI outputs necessary in all tasks? On the other hand, the current generation of LLMs is arguably already superintelligent in, say, tasks like producing iambic pentameters using only words that start with the letter F. And is it really necessary to understand how they’re able to do that before people enjoy the poem? Understanding or verifying would not be essential everywhere, but it would likely be essential in commercial, risk-sensitive, or security-conscious applications.

Conclusion

It’s not clear whether transformer-based LLMs can be the foundational architecture that realizes a superintelligent system. More realistically, with the current architectures, the technology sector is on the path to super-automation, and new insights will be needed to take us to the path to superintelligence. Nevertheless, achieving superintelligence in a safe way is an important technical and research problem likely to be pursued by leading scientists and entrepreneurs, even if commercial or real-world impacts are likely very far off.

 

¹Whether these ambitions can be realized is typically a debate that centers around the scaling laws of LLMs (or other models), namely to what extent can we expect their performance to improve the larger they become in terms of parameter count, data, and other factors. A useful technical reference for these investigations is here: Maor Ivgi, Yair Carmon, and Jonathan Berant, Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments, (2022).

²A detailed classification of AGI and a definition of superintelligence is given here: Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg, Position: Levels of AGI for Operationalizing Progress on the Path to AGI, (2024).

³Nick Bostrom, How Long Before Superintelligence?, (International Journal of Future Studies, 1998).

⁴Melanie Mitchell’s research, for example, has grappled with this problem: John Pavlus, The Computer Scientist Training AI to Think with Analogies, (Quanta, 2021).

⁵The link between ants and superintelligence has been explored in various places. For instance, there have been extensive studies on leafcutter ant colonies and how they collectively form an intellect seemingly superior to the individuals comprising it. A classic reference is: Bert Hölldobler and Edward O. Wilson, The Leafcutter Ants: Civilization by Instinct, (W. W. Norton & Co. Ltd., 2010).

 

This article is being provided for educational purposes only. The information contained in this article does not constitute a recommendation from any Goldman Sachs entity to the recipient, and Goldman Sachs is not providing any financial, economic, legal, investment, accounting, or tax advice through this article or to its recipient. Neither Goldman Sachs nor any of its affiliates makes any representation or warranty, express or implied, as to the accuracy or completeness of the statements or any information contained in this article and any liability therefore (including in respect of direct, indirect, or consequential loss or damage) is expressly disclaimed.