Artificial Intelligence
Artificial intelligence has gone from a fringe issue to a major factor in human society. There is an urgent need to responsibly govern the technology, including for the possibility of it posing catastrophic risks.
An Introduction to Artificial Intelligence
When GCRI was founded in 2011, we were among the only groups taking AI risk seriously. Our 2017 and 2020 surveys of artificial general intelligence (AGI) projects was published in part to show that while AGI may seem like speculative science fiction, it was an active pursuit of many research and development groups. Now—early 2025—AGI doesn’t seem so speculative anymore. Current state-of-the-art systems already have some generality, being able to field inquiries across a wide range of domains, even while they still have significant limitations. AI has also become a major economic force and a significant political issue.
Where things go from here is not known. It is clear that the technology and its place in society will continue to change. However, the directions things could go in are extremely divergent. AI could take over the world, rendering humans obsolete, or it could be yet another economic bubble soon to burst, or it could end up with significant but comparatively modest impacts on the economy and society. These events could unfold within just a few months or over the course of many decades. The outcomes could be good or bad, or utopian or dystopian, or even catastrophic. The history of AI is full of false predictions and cycles of boom and bust, but it has never had such a high profile.
The only way to guarantee we avoid AI catastrophe is to stop building more advanced AI. That would also avoid other potential harms from the technology, such as mass unemployment, wealth inequality, environmental degradation, weapons proliferation, misinformation, harms to marginalized communities, and a loss of human connections. However, it would also miss out on any potential benefits, such as gains in productivity and economic growth, progress in science and technology, and, in the extreme case, the realization of more utopian scenarios, including scenarios involving large-scale expansion into outer space.
A central challenge, then, is to pursue courses of action that would, as much as can be done, obtain the benefits of AI while avoiding the harms, including catastrophic harms. This requires an understanding of both the technology itself and the social, political, and economic contexts in which it is governed. For example, to reduce the risk of catastrophe, it is important to identify the particular AI capabilities that could result in catastrophic outcomes and to craft governance approaches that address these capabilities across all AI groups, including those in rival corporations and countries. Whereas rival groups may typically prefer to compete, they may be more willing to cooperate to avoid catastrophic outcomes that no one wants, especially if there is outside pressure for them to do so.
GCRI’s work on AI centers on three broad themes: risk analysis, governance, and ethics.
Our AI risk analysis work develops models for assessing the potential for AI technology to cause catastrophic harm, especially the risk of catastrophic AI takeover. Our early risk analysis developed the ASI-PATH model as a theoretical framework for synthesizing and understanding certain concepts of AI risk. More recently, our study of large language model (LLM) risk develops a more practical framework for studying takeover catastrophe risk from LLMs and applies it to actual LLMs. The study found that the LLMs available at the time fall well short of the capabilities that may be needed for takeover, though the technology continues to change and continued monitoring is warranted.
Our AI risk governance work seeks to develop practical means for addressing catastrophic AI risk alongside other AI issues. We recognize that AI raises a wide range of issues and that some people active in AI issues are not as focused on catastrophic risk. Therefore, as a matter of strategy, our research seeks to develop win-win solutions that reduce catastrophic risks while also making progress on other issues. This includes advancing responsible corporate governance, achieving collective action between AI groups, and bridging divides between those concerned about catastrophic risks and other issues. We have also worked on environmental implications of AI, drawing on our background in climate and environment. We believe better outcomes will occur when those with synergistic goals can work together instead of being at odds with each other.
Our AI ethics work explores ideas for what ethical values should be built into AI systems, also known as machine ethics. This is another domain of wide relevance, including for both the AI systems that could pose catastrophic risks and for other, less capable systems. One specific focus is on “social choice” ethics that seek to align the values of AI systems with the values of humanity as a whole. This is analogous to how democracy can align the values of governments to the values of the populations being governed. It raises difficult questions about how to define the values of humanity as a whole, given that people have many disagreements about values, the potential for aggregate values to be manipulated, and the possibility of including the values of nonhumans. This work extends our broader work on ethics.
Image credits: map: Seth Baum; landscape: Buckyball Design; voting line: April Sikorski
Featured GCRI Publications on Artificial Intelligence
For over 50 years, experts have worried about the risk of AI taking over the world and killing everyone. The concern had always been about hypothetical future AI systems—until recent LLMs emerged. This paper, published in the journal Risk Analysis, assesses how close LLMs are to having the capabilities needed to cause takeover catastrophe.
Private corporations are at the forefront of AI. This paper, published in the journal Information, surveys how to orient AI corporate governance toward the public interest, covering nine types of actors: management, workers, investors, corporate partners, industry consortia, nonprofit organizations, the public, the media, and governments.
In AI ethics, a major theme is the challenge of aligning AI to human values, but this raises the question of the role of nonhumans. This paper, published in the journal AI & Ethics, documents widespread neglect of nonhumans in AI ethics and argues for giving nonhumans adequate attention, including to avoid potentially catastrophic outcomes.
Full List of GCRI Publications on Artificial Intelligence
Baum, Seth D. Assessing the risk of takeover catastrophe from large language models. Risk Analysis, forthcoming, DOI:10.1111/risa.14353.
Baum, Seth D. and Andrea Owe. On the intrinsic value of diversity. Inquiry, forthcoming, DOI 10.1080/0020174X.2024.2367247.
Baum, Seth D. Manipulating aggregate societal values to bias AI social choice ethics. AI and Ethics, forthcoming, DOI 10.1007/s43681-024-00495-6.
Baum, Seth D. and Andrea Owe, 2023. From AI for people to AI for the world and the universe. AI & Society, vol. 38, no. 2 (April), pages 679-680, DOI 10.1007/s00146-022-01402-5.
Owe, Andrea and Seth D. Baum, 2021. The ethics of sustainability for artificial intelligence. In Philipp Wicke, Marta Ziosi, João Miguel Cunha, and Angelo Trotta (Editors), Proceedings of the 1st International Conference on AI for People: Towards Sustainable AI (CAIP 2021), Bologna, pages 1-17, DOI 10.4108/eai.20-11-2021.2314105.
Baum, Seth D. and Andrea Owe, 2023. Artificial intelligence needs environmental ethics. Ethics, Policy, & Environment, vol. 26, no. 1, pages 139-143, DOI 10.1080/21550085.2022.2076538.
Baum, Seth D. and Jonas Schuett, 2021. The case for long-term corporate governance of AI. Effective Altruism Forum, 3 November.
Galaz, Victor, Miguel A. Centeno, Peter W. Callahan, Amar Causevic, Thayer Patterson, Irina Brass, Seth Baum, Darryl Farber, Joern Fischer, David Garcia, Timon McPhearson, Daniel Jimenez, Brian King, Paul Larcey, and Karen Levy, 2021. Artificial intelligence, systemic risks, and sustainability. Technology in Society, vol. 67 (November), article 101741, DOI 10.1016/j.techsoc.2021.101741.
Cihon, Peter, Jonas Schuett, and Seth D. Baum, 2021. Corporate governance of artificial intelligence in the public interest. Information, vol. 12, article 275, DOI 10.3390/info12070275.
de Neufville, Robert and Seth D. Baum, 2021. Collective action on artificial intelligence: A primer and review. Technology in Society, vol. 66 (August), article 101649, DOI 10.1016/j.techsoc.2021.101649.
Owe, Andrea and Seth D. Baum, 2021. Moral consideration of nonhumans in the ethics of artificial intelligence. AI & Ethics, vol. 1, no. 4 (November), pages 517-528, DOI 10.1007/s43681-021-00065-0.
Cihon, Peter, Moritz J. Kleinaltenkamp, Jonas Schuett, and Seth D. Baum, 2021. AI certification: Advancing ethical practice by reducing information asymmetries. IEEE Transactions on Technology and Society, vol. 2, issue 4 (December), pages 200-209, DOI 10.1109/TTS.2021.3077595.
Fitzgerald, McKenna, Aaron Boddy, and Seth D. Baum, 2020. 2020 survey of artificial general intelligence projects for ethics, risk, and policy. Global Catastrophic Risk Institute Technical Report 20-1.
Baum, Seth D., 2021. Artificial interdisciplinarity: Artificial intelligence for research on complex societal problems. Philosophy & Technology, vol. 34, no. S1 (November), pages 45-63, DOI 10.1007/s13347-020-00416-5.
Baum, Seth D., 2020. Deep learning and the sociology of human-level artificial intelligence – Book review: Artifictional Intelligence: Against Humanity’s Surrender to Computers. Metascience, vol. 29, no. 2 (July), pages 313-317, DOI 10.1007/s11016-020-00510-6.
Baum, Seth D., 2020. Medium-term artificial intelligence and society. Information, vol. 11, no. 6, article 290, DOI 10.3390/info11060290.
Baum, Seth D., Robert de Neufville, Anthony M. Barrett, and Gary Ackerman, 2022. Lessons for artificial intelligence from other global risks. In Maurizio Tinnirello (editor), The Global Politics of Artificial Intelligence. Boca Raton: CRC Press, pages 103-131.
Baum, Seth D., 2018. Countering superintelligence misinformation. Information, vol. 9, no. 10 (September), article 244, DOI 10.3390/info9100244.
Baum, Seth D., 2018. Superintelligence skepticism as a political tool. Information, vol. 9, no. 9 (August), article 209, DOI 10.3390/info9090209.
Baum, Seth D., 2018. Preventing an AI apocalypse. Project Syndicate, 16 May.
Baum, Seth D., Anthony M. Barrett, and Roman V. Yampolskiy, 2017. Modeling and interpreting expert disagreement about artificial superintelligence. Informatica, vol. 41, no. 4 (December), pages 419-427.
Baum, Seth D., 2017. A survey of artificial general intelligence projects for ethics, risk, and policy. Global Catastrophic Risk Institute Working Paper 17-1.
Baum, Seth D., 2017. On the promotion of safe and socially beneficial artificial intelligence. AI & Society, vol. 32, no. 4 (November), pages 543-551, DOI 10.1007/s00146-016-0677-0.
White, Trevor N. and Seth D. Baum, 2017. Liability law for present and future robotics technology. In Patrick Lin, Keith Abney, and Ryan Jenkins (editors), Robot Ethics 2.0, Oxford: Oxford University Press, pages 66-79.
Baum, Seth D., 2020. Social choice ethics in artificial intelligence. AI & Society, vol. 35, no. 1 (March), pages 165-176, DOI 10.1007/s00146-017-0760-1.
Baum, Seth D., 2017. The social science of computerized brains – Book review: The Age of Em: Work, Love, and Life When Robots Rule the Earth. Futures, vol. 90 (June), pages 61-63, DOI 10.1016/j.futures.2017.03.005.
Baum, Seth D., 2018. Reconciliation between factions focused on near-term and long-term artificial intelligence. AI & Society, vol. 33, no. 4 (November), pages 565-572, DOI 10.1007/s00146-017-0734-3.
Baum, Seth, 2016. Should we let uploaded brains take over the world? Scientific American Blogs, 18 October.
Baum, Seth, 2016. Tackling near and far AI threats at once. Bulletin of the Atomic Scientists, 6 October.
Barrett, Anthony M. and Seth D. Baum, 2017. A model of pathways to artificial superintelligence catastrophe for risk and decision analysis. Journal of Experimental & Theoretical Artificial Intelligence, vol. 29, no. 2, pages 397-414, DOI 10.1080/0952813X.2016.1186228.
Barrett, Anthony M. and Seth D. Baum, 2017. Risk analysis and risk management for the artificial superintelligence research and development process. In Victor Callaghan, James Miller, Roman Yampolskiy, and Stuart Armstrong (editors), The Technological Singularity: Managing the Journey. Berlin: Springer, pages 127-140.
Baum, Seth and Trevor White, 2015. When robots kill. The Guardian Political Science blog, 23 June.
Baum, Seth, 2015. Stopping killer robots and other future threats. Bulletin of the Atomic Scientists, 22 February.
Baum, Seth D., 2014. Film review: Transcendence. Journal of Evolution and Technology, vol. 24, no. 2 (September), pages 79-84.
Baum, Seth, 2013. Our Final Invention: Is AI the defining issue for humanity? Scientific American Blogs, 11 October.
Baum, Seth and Grant Wilson, 2013. How to create an international treaty for emerging technologies. Institute for Ethics and Emerging Technologies, 21 February.
Wilson, Grant S., 2013. Minimizing global catastrophic and existential risks from emerging technologies through international law. Virginia Environmental Law Journal, vol. 31, no. 2, pages 307-364.
An Introduction to Artificial Intelligence
When GCRI was founded in 2011, we were among the only groups taking AI risk seriously. Our 2017 and 2020 surveys of artificial general intelligence (AGI) projects was published in part to show that while AGI may seem like speculative science fiction, it was an active pursuit of many research and development groups. Now—early 2025—AGI doesn’t seem so speculative anymore. Current state-of-the-art systems already have some generality, being able to field inquiries across a wide range of domains, even while they still have significant limitations. AI has also become a major economic force and a significant political issue.
Where things go from here is not known. It is clear that the technology and its place in society will continue to change. However, the directions things could go in are extremely divergent. AI could take over the world, rendering humans obsolete, or it could be yet another economic bubble soon to burst, or it could end up with significant but comparatively modest impacts on the economy and society. These events could unfold within just a few months or over the course of many decades. The outcomes could be good or bad, or utopian or dystopian, or even catastrophic. The history of AI is full of false predictions and cycles of boom and bust, but it has never had such a high profile.
The only way to guarantee we avoid AI catastrophe is to stop building more advanced AI. That would also avoid other potential harms from the technology, such as mass unemployment, wealth inequality, environmental degradation, weapons proliferation, misinformation, harms to marginalized communities, and a loss of human connections. However, it would also miss out on any potential benefits, such as gains in productivity and economic growth, progress in science and technology, and, in the extreme case, the realization of more utopian scenarios, including scenarios involving large-scale expansion into outer space.
A central challenge, then, is to pursue courses of action that would, as much as can be done, obtain the benefits of AI while avoiding the harms, including catastrophic harms. This requires an understanding of both the technology itself and the social, political, and economic contexts in which it is governed. For example, to reduce the risk of catastrophe, it is important to identify the particular AI capabilities that could result in catastrophic outcomes and to craft governance approaches that address these capabilities across all AI groups, including those in rival corporations and countries. Whereas rival groups may typically prefer to compete, they may be more willing to cooperate to avoid catastrophic outcomes that no one wants, especially if there is outside pressure for them to do so.
GCRI’s work on AI centers on three broad themes: risk analysis, governance, and ethics.
Our AI risk analysis work develops models for assessing the potential for AI technology to cause catastrophic harm, especially the risk of catastrophic AI takeover. Our early risk analysis developed the ASI-PATH model as a theoretical framework for synthesizing and understanding certain concepts of AI risk. More recently, our study of large language model (LLM) risk develops a more practical framework for studying takeover catastrophe risk from LLMs and applies it to actual LLMs. The study found that the LLMs available at the time fall well short of the capabilities that may be needed for takeover, though the technology continues to change and continued monitoring is warranted.
Our AI risk governance work seeks to develop practical means for addressing catastrophic AI risk alongside other AI issues. We recognize that AI raises a wide range of issues and that some people active in AI issues are not as focused on catastrophic risk. Therefore, as a matter of strategy, our research seeks to develop win-win solutions that reduce catastrophic risks while also making progress on other issues. This includes advancing responsible corporate governance, achieving collective action between AI groups, and bridging divides between those concerned about catastrophic risks and other issues. We have also worked on environmental implications of AI, drawing on our background in climate and environment. We believe better outcomes will occur when those with synergistic goals can work together instead of being at odds with each other.
Our AI ethics work explores ideas for what ethical values should be built into AI systems, also known as machine ethics. This is another domain of wide relevance, including for both the AI systems that could pose catastrophic risks and for other, less capable systems. One specific focus is on “social choice” ethics that seek to align the values of AI systems with the values of humanity as a whole. This is analogous to how democracy can align the values of governments to the values of the populations being governed. It raises difficult questions about how to define the values of humanity as a whole, given that people have many disagreements about values, the potential for aggregate values to be manipulated, and the possibility of including the values of nonhumans. This work extends our broader work on ethics.
Image credits: map: Seth Baum; landscape: Buckyball Design; voting line: April Sikorski
Featured GCRI Publications on Artificial Intelligence
For over 50 years, experts have worried about the risk of AI taking over the world and killing everyone. The concern had always been about hypothetical future AI systems—until recent LLMs emerged. This paper, published in the journal Risk Analysis, assesses how close LLMs are to having the capabilities needed to cause takeover catastrophe.
Private corporations are at the forefront of AI. This paper, published in the journal Information, surveys how to orient AI corporate governance toward the public interest, covering nine types of actors: management, workers, investors, corporate partners, industry consortia, nonprofit organizations, the public, the media, and governments.
In AI ethics, a major theme is the challenge of aligning AI to human values, but this raises the question of the role of nonhumans. This paper, published in the journal AI & Ethics, documents widespread neglect of nonhumans in AI ethics and argues for giving nonhumans adequate attention, including to avoid potentially catastrophic outcomes.
Full List of GCRI Publications on Artificial Intelligence
Baum, Seth D. Assessing the risk of takeover catastrophe from large language models. Risk Analysis, forthcoming, DOI:10.1111/risa.14353.
Baum, Seth D. and Andrea Owe. On the intrinsic value of diversity. Inquiry, forthcoming, DOI 10.1080/0020174X.2024.2367247.
Baum, Seth D. Manipulating aggregate societal values to bias AI social choice ethics. AI and Ethics, forthcoming, DOI 10.1007/s43681-024-00495-6.
Baum, Seth D. and Andrea Owe, 2023. From AI for people to AI for the world and the universe. AI & Society, vol. 38, no. 2 (April), pages 679-680, DOI 10.1007/s00146-022-01402-5.
Owe, Andrea and Seth D. Baum, 2021. The ethics of sustainability for artificial intelligence. In Philipp Wicke, Marta Ziosi, João Miguel Cunha, and Angelo Trotta (Editors), Proceedings of the 1st International Conference on AI for People: Towards Sustainable AI (CAIP 2021), Bologna, pages 1-17, DOI 10.4108/eai.20-11-2021.2314105.
Baum, Seth D. and Andrea Owe, 2023. Artificial intelligence needs environmental ethics. Ethics, Policy, & Environment, vol. 26, no. 1, pages 139-143, DOI 10.1080/21550085.2022.2076538.
Baum, Seth D. and Jonas Schuett, 2021. The case for long-term corporate governance of AI. Effective Altruism Forum, 3 November.
Galaz, Victor, Miguel A. Centeno, Peter W. Callahan, Amar Causevic, Thayer Patterson, Irina Brass, Seth Baum, Darryl Farber, Joern Fischer, David Garcia, Timon McPhearson, Daniel Jimenez, Brian King, Paul Larcey, and Karen Levy, 2021. Artificial intelligence, systemic risks, and sustainability. Technology in Society, vol. 67 (November), article 101741, DOI 10.1016/j.techsoc.2021.101741.
Cihon, Peter, Jonas Schuett, and Seth D. Baum, 2021. Corporate governance of artificial intelligence in the public interest. Information, vol. 12, article 275, DOI 10.3390/info12070275.
de Neufville, Robert and Seth D. Baum, 2021. Collective action on artificial intelligence: A primer and review. Technology in Society, vol. 66 (August), article 101649, DOI 10.1016/j.techsoc.2021.101649.
Owe, Andrea and Seth D. Baum, 2021. Moral consideration of nonhumans in the ethics of artificial intelligence. AI & Ethics, vol. 1, no. 4 (November), pages 517-528, DOI 10.1007/s43681-021-00065-0.
Cihon, Peter, Moritz J. Kleinaltenkamp, Jonas Schuett, and Seth D. Baum, 2021. AI certification: Advancing ethical practice by reducing information asymmetries. IEEE Transactions on Technology and Society, vol. 2, issue 4 (December), pages 200-209, DOI 10.1109/TTS.2021.3077595.
Fitzgerald, McKenna, Aaron Boddy, and Seth D. Baum, 2020. 2020 survey of artificial general intelligence projects for ethics, risk, and policy. Global Catastrophic Risk Institute Technical Report 20-1.
Baum, Seth D., 2021. Artificial interdisciplinarity: Artificial intelligence for research on complex societal problems. Philosophy & Technology, vol. 34, no. S1 (November), pages 45-63, DOI 10.1007/s13347-020-00416-5.
Baum, Seth D., 2020. Deep learning and the sociology of human-level artificial intelligence – Book review: Artifictional Intelligence: Against Humanity’s Surrender to Computers. Metascience, vol. 29, no. 2 (July), pages 313-317, DOI 10.1007/s11016-020-00510-6.
Baum, Seth D., 2020. Medium-term artificial intelligence and society. Information, vol. 11, no. 6, article 290, DOI 10.3390/info11060290.
Baum, Seth D., Robert de Neufville, Anthony M. Barrett, and Gary Ackerman, 2022. Lessons for artificial intelligence from other global risks. In Maurizio Tinnirello (editor), The Global Politics of Artificial Intelligence. Boca Raton: CRC Press, pages 103-131.
Baum, Seth D., 2018. Countering superintelligence misinformation. Information, vol. 9, no. 10 (September), article 244, DOI 10.3390/info9100244.
Baum, Seth D., 2018. Superintelligence skepticism as a political tool. Information, vol. 9, no. 9 (August), article 209, DOI 10.3390/info9090209.
Baum, Seth D., 2018. Preventing an AI apocalypse. Project Syndicate, 16 May.
Baum, Seth D., Anthony M. Barrett, and Roman V. Yampolskiy, 2017. Modeling and interpreting expert disagreement about artificial superintelligence. Informatica, vol. 41, no. 4 (December), pages 419-427.
Baum, Seth D., 2017. A survey of artificial general intelligence projects for ethics, risk, and policy. Global Catastrophic Risk Institute Working Paper 17-1.
Baum, Seth D., 2017. On the promotion of safe and socially beneficial artificial intelligence. AI & Society, vol. 32, no. 4 (November), pages 543-551, DOI 10.1007/s00146-016-0677-0.
White, Trevor N. and Seth D. Baum, 2017. Liability law for present and future robotics technology. In Patrick Lin, Keith Abney, and Ryan Jenkins (editors), Robot Ethics 2.0, Oxford: Oxford University Press, pages 66-79.
Baum, Seth D., 2020. Social choice ethics in artificial intelligence. AI & Society, vol. 35, no. 1 (March), pages 165-176, DOI 10.1007/s00146-017-0760-1.
Baum, Seth D., 2017. The social science of computerized brains – Book review: The Age of Em: Work, Love, and Life When Robots Rule the Earth. Futures, vol. 90 (June), pages 61-63, DOI 10.1016/j.futures.2017.03.005.
Baum, Seth D., 2018. Reconciliation between factions focused on near-term and long-term artificial intelligence. AI & Society, vol. 33, no. 4 (November), pages 565-572, DOI 10.1007/s00146-017-0734-3.
Baum, Seth, 2016. Should we let uploaded brains take over the world? Scientific American Blogs, 18 October.
Baum, Seth, 2016. Tackling near and far AI threats at once. Bulletin of the Atomic Scientists, 6 October.
Barrett, Anthony M. and Seth D. Baum, 2017. A model of pathways to artificial superintelligence catastrophe for risk and decision analysis. Journal of Experimental & Theoretical Artificial Intelligence, vol. 29, no. 2, pages 397-414, DOI 10.1080/0952813X.2016.1186228.
Barrett, Anthony M. and Seth D. Baum, 2017. Risk analysis and risk management for the artificial superintelligence research and development process. In Victor Callaghan, James Miller, Roman Yampolskiy, and Stuart Armstrong (editors), The Technological Singularity: Managing the Journey. Berlin: Springer, pages 127-140.
Baum, Seth and Trevor White, 2015. When robots kill. The Guardian Political Science blog, 23 June.
Baum, Seth, 2015. Stopping killer robots and other future threats. Bulletin of the Atomic Scientists, 22 February.
Baum, Seth D., 2014. Film review: Transcendence. Journal of Evolution and Technology, vol. 24, no. 2 (September), pages 79-84.
Baum, Seth, 2013. Our Final Invention: Is AI the defining issue for humanity? Scientific American Blogs, 11 October.
Baum, Seth and Grant Wilson, 2013. How to create an international treaty for emerging technologies. Institute for Ethics and Emerging Technologies, 21 February.
Wilson, Grant S., 2013. Minimizing global catastrophic and existential risks from emerging technologies through international law. Virginia Environmental Law Journal, vol. 31, no. 2, pages 307-364.