Artificial Intelligence

The risk of AI catastrophe is a contentious point of debate within the overall space of AI issues. There is a need for analysis that clarifies the risk and solutions that address the risk alongside other AI issues.

In 2017, GCRI published a paper Reconciliation between factions focused on near-term and long-term artificial intelligence, which outlined our perspective on solutions and strategy for AI. It addressed the divide that existed between people concerned about the issues that current AI technology was already posing and people concerned about the issues that future AI technology might eventually pose, including future catastrophic risks. The main idea of the paper is to focus on win-win solutions that make progress on both near-term and long-term issues. Since then, much of GCRI’s AI work has helped to develop such win-win solutions, on matters such as collective action, corporate governance, and AI ethics.

In 2024—the time of this writing—we believe this win-win perspective has aged well. We are heartened to see AI governance initiatives covering a multitude of near-term and long-term AI issues. At the same time, we are saddened to see the perpetuation of the same old near-term vs. long-term debates. We continue to believe that it would be more constructive to focus efforts on win-win solutions, or even on single-win solutions, instead of endless debate over which AI issues are more important. People involved in addressing AI issues don’t have to agree on everything in order to partner together for positive change.

Efforts to address the catastrophic risks from AI benefit from being grounded in the actual nature of the risks. Unfortunately, this is easier said than done. All global catastrophic risk analysis faces a fundamental shortage of data—no civilization-ending global catastrophe has ever happened before. Uncertainty about long-term AI risk is compounded by the fact that it involves technology that doesn’t exist yet. To study the risk, some speculation is inevitable and should not be frowned upon. Nonetheless, AI risk and decision analysis can help by pulling together what is known, identifying important uncertainties, and evaluating the implications for decision-making.

GCRI’s work on AI risk has proceeded along two fronts. First, we have modeled the risk using established risk analysis techniques. Our ASI-PATH model provided a framework for synthesizing and understanding certain concepts of AI risk. The model does not eliminate the uncertainty about AI risk, but it can help to make sense of it, including expert disagreements on it. Second, we have characterized the current state of affairs in AI R&D, with a specific focus on artificial general intelligence (AGI), a potential future type of AI that is closely associated with catastrophic risk. We published reports in 2017 and 2020 that surveyed the landscape of AGI research and development around the world, such as in the map below. These surveys provide a more empirical grounding to work on AI risk.

GCRI’s AI work has also addressed issues in AI ethics. Our work has focused on machine ethics: the ethical concepts to build into AI systems. These concepts define what goals the AI systems will pursue and likewise can be a major factor in the impacts that AI systems have on the world. This holds for all types of AI system, especially for the sorts of high-stakes future systems that may pose a catastrophic risk. As humans build more and more capable AI systems, it becomes important to ask what exactly we want these systems to do.

GCRI’s AI ethics work has critiqued standard AI ethics paradigms such as value alignment and human compatibility. These paradigms often treat machine ethics in monolithic terms, such as “the” value alignment problem, as if there is only one. Our work on social choice ethics shows that in fact, there are many values that AI systems can be aligned to or made compatible with. One reason for this is because human values are diverse. Indeed, the concept of social choice exists to handle these disagreements, such as through voting in elections. Another reason is that there are many beings whose values may merit inclusion in machine ethics. Whereas discussions of machine ethics often focus narrowly on the values of current humans, an ethical argument can be made for also including future humans and various nonhumans. In high-stakes AI systems, whether these beings are included could be the difference between very good and catastrophically bad outcomes for them.

Featured GCRI Publications on Artificial Intelligence

2020 survey of artificial general intelligence projects for ethics, risk, and policy

An AI system with intelligence across a wide range of domains could be extremely impactful—for better or for worse. Traditionally, AI systems have been narrow, focusing on a limited set of domains. This paper surveys the projects seeking to build artificial general intelligence (AGI). It finds 72 projects across 37 countries, mainly in corporations and academia. The projects vary in terms of their stated goals, commitment to safety, and other parameters of relevance to AI governance.

Moral consideration of nonhumans in the ethics of artificial intelligence

Work on AI ethics often seeks to align AI to human values. Arguably, the values of nonhumans should also be included: to exclude nonhumans would be an inappropriate privileging of humans over other entities. If they are excluded, it could result in potentially catastrophic harms to nonhuman populations and other unethical outcomes. This paper documents the limited amount of attention to nonhumans within the field of AI ethics and presents an argument for giving nonhumans adequate attention.

Collective action on artificial Intelligence: A primer and review

To achieve positive outcomes for AI and avoid harms, including catastrophic harms, many people will need to take a variety of actions. In other words, collective action is needed. Fortunately, human society has considerable experience with collective action in other domains, and this has been studied extensively in prior research. This paper provides a two-way primer, helping people with collective action expertise learn more about AI and helping people with AI expertise learn more about collective action.

Full List of GCRI Publications on Artificial Intelligence

Image credits: landscape: Buckyball Design; map: Seth Baum; voting line: April Sikorski