Depending on who you ask, the AI industry is either on track to render humans obsolete within the next few years or about to go bust, or maybe somewhere in between.
This matter is among the most important questions right now, not just for global catastrophic risk but also for the world as a whole. If AI will soon render humans obsolete, hardly anything else matters. There’s not much point in caring about climate change or education or any other even slightly long-term issue, nor is there much point in making any sort of normal life plans. We would all be at the mercy of what happens with AI.
While I am skeptical about the extreme short-term AI scenarios, I don’t see how they can be ruled out, and so they should be taken seriously, even if they may be unlikely.
Duck, Rabbit
The famous duck/rabbit illusion shows that people can look at the same image and see different things. It’s a matter of interpretation, of how our minds process the available evidence. So too with the evidence about AI.
On 20 December 2024, AI policy researcher Miles Brundage wrote, “AI that exceeds human performance in nearly every cognitive domain is almost certain to be built and deployed in the next few years”. Brundage is referring to artificial general intelligence, or AGI, meaning AI systems with broad capabilities across domains. Earlier AI systems were narrow, with capabilities across one or just a few domains, such as a system designed to play chess or analyze images. Narrow AI systems might be able to outperform humans at specific tasks, but they couldn’t do anything else, so humans were still needed. In contrast, AGI could conceivably outperform humans at every task, rendering humans obsolete.
At a minimum, a world with advanced AGI could result in mass unemployment as most or all of the economy becomes automated. The best-case scenario may be if humans are supported financially with widespread and generous social welfare programs. This might enable everyone worldwide to live comfortable lives of leisure, though we would lose the dignity of work and we would be subject to the political whims of whoever or whatever controls the welfare programs. The worst-case scenario may be for the AI to take over the world and kill everyone. The recently published scenario AI 2027 describes this in detail, ending with AI killing all humans in mid-2030.
Suffice to say, it is jarring to think that AGI capable of these outcomes may be arriving in the next few years, but this is indeed what some people in the AI space currently project. However, there is much disagreement on this point. Outspoken AI critic Gary Marcus asserts that current generative AI technology—the basis of most of today’s leading-edge AI—is more of a dud. Vanguard Global Chief Economist Joe Davis expects AI’s impact on the job market to be “disruptive, not dystopian”. These positions are echoed in a recent survey by AAAI, a major professional society for AI, in which 76% of respondents stated that current AI approaches are unlikely to yield AGI.
My View
Until recently, projections of imminent AGI often leaned heavily on so-called “scaling laws”, in which advances in AI system performance could be made by making larger AI systems. The expectation was that industry would keep making larger systems until they hit AGI. However, in 2024, the scaling of large language models started to see diminishing returns. More recently, projections have emphasized reasoning models, which use additional techniques to extend standard LLMs. The date 20 December 2024—the day Brundage published his prediction copied above—is when OpenAI announced o3, a more advanced reasoning model that exhibited some striking capabilities. The DeepSeek R1 model, released to much fanfare on 10 January 2025, is also a reasoning model.
The scaling of reasoning models has not yet hit diminishing returns, but my guess is that it will. Ditto for any other techniques that extend current LLMs. That’s because current LLMs seem to be too shaky of a foundation for advanced AGI.
The inner workings of current LLMs are shockingly poorly understood. The best research I’ve found points to them as being, at their core, bags of heuristics, meaning they model the world through large numbers of depictions of specific details of the world instead of a simpler and more general model of how the world works. For example, in an LLM trained to learn the game Othello, one heuristic found was [If the move A4 was just played AND B4 is occupied AND C4 is occupied ⇒ update B4+C4+D4 to “theirs”]. The LLM contained many such heuristics, but apparently not a general model of the rules of the game, even though Othello is a simple game whose rules can easily be built into computer programs.
The “bag of heuristics” theory would explain major characteristics of LLMs. LLMs have shown remarkable ability to answer certain questions while also making egregious mistakes such as hallucinations, in which AI systems produce false outputs but present them as being true, or illegal moves in games like Othello and chess. Perhaps the errors occur when the LLM lacks the right heuristics. The observed “scaling laws” could be due to LLMs building up a more robust collection of heuristics, and then the diminishing returns come when the “bags” started to saturate. Reasoning models may be an improved mechanism for AI systems to access the LLM bag of heuristics, but that too may eventually face diminishing returns due to inherent limitations in the LLM’s “bag of heuristics” model of the world. Indeed, a recent study of mathematical reasoning finds that even the latest reasoning models continue to make hallucinations and other basic errors.
Intuitively, it should be possible to build useful systems out of bags of heuristics: build a large enough bag full of heuristics on relevant topics, and something useful should come out of it. However, without a more robust model of the world, such systems may be inherently limited. This is consistent with the Vanguard view of AI as “disruptive, not dystopian”. LLMs may have substantial impact on our world, but they wouldn’t fully replace humans.
That leaves two types of extreme scenarios. First, this “bag of heuristics theory” may be wrong. Maybe it’s possible to create extreme AGI with bags of heuristics, or maybe LLMs are more than just bags of heuristics. The available evidence is not nearly enough to rule this out. Second, there are groups pursuing AGI via different approaches, such as DeepMind and OpenCog using neurosymbolic AI, which is a more pluralistic approach to AI design. Projections of imminent AGI have focused less on these lines of work, but perhaps they could feature in extreme scenarios.
If extreme AGI is built, pluralistic designs seem safer by giving us more ways of building in safety mechanisms. It seems especially dangerous to pursue AGI via current LLMs, which have proven to be error prone and ill-suited for safety-critical applications.
All things considered, it is alarming that the AI industry is attempting to plow ahead on AGI. Even if its primary approach has only a small chance of creating extreme AGI, the stakes are so high that it still constitutes a massive risk. And this is hardly the only problem the industry is causing—other problems include copyright abuse, environmental impacts, interfering with student learning, and internet slop. Yes, there are also benefits, but it is fair to question whether the benefits outweigh the harms.
What To Do
It may never be good to build the sort of extreme AGI that could render humans obsolete, but it certainly isn’t right now, when AI systems remain error-prone, when AI corporations are rife with irresponsible behavior, and when society as a whole has done so little to consider the implications of AGI. Therefore, the current aims should be to steer the industry away from AGI and toward more responsible applications, and to expand the public conversation about what role we want for AI in society.
It may seem hopeless to try steering the multi-billion-dollar AI industry, but it is actually currently in a vulnerable position. It has a largely negative public reputation, especially on extreme AGI, while safety regulation limiting the industry is popular. The industry has also been largely a financial loser, with costs often exceeding revenues. The splashy new reasoning models are especially expensive. Investment in AI development has recently become less aggressive. There is both a public relations case and an ordinary business case for the industry to adopt a products not gods orientation, in which it aims to build useful tools instead of extreme AGI. It should be pushed in that direction. To that end, prior GCRI research on AI corporate governance shows how different groups of people can constructively influence AI companies, from corporate executives to members of the public.
For the public conversation, we should raise a fundamental question: What is the point of an economy? Thus far, automation has shifted profits from workers to business owners and has increased income inequality between those whose work can be replaced by machines and those whose work complements and extends the machines. In some communities, the job loss from automation has contributed to a perceived loss of self-reliance, causing widespread feelings of shame and humiliation. If AGI could automate virtually everything, is that something we should want?
Finally, there should be continual monitoring for warning signs of AGI catastrophe and contingency plans in case extreme AGI starts appearing more imminent. That should include plans to shut down AGI projects unless there is high confidence that they are safe and in the public interest. Importantly, plans must include provisions for international cooperation, even among geopolitical rivals, especially the US and China; improved international cooperation would also have other benefits and is worth pursuing regardless of AGI. Even if extreme AGI does not currently seem imminent, it is good to have contingency plans just in case. The stakes are too high to get this wrong.
Image credit: Unknown artist
