Assessing the Risks Posed by the Convergence of Artificial Intelligence and Biotechnology

Introduction
 Technological Progress S ince the explosion of the information age, the collection of data has been increasing. About 2.5 quintillion (10 18 ) bytes of data are generated each day, and 90% of all data have been generated in the past few years alone. 1 Although valuable information is often hidden in these vast datasets, due to their size and complexity, a human analyst could take weeks, months, or even years to discover them, if they are able to at all. If one can analyze a large enough set of data, it is possible to infer corollary relationships or patterns that may be of salience. In the past, it was not conceivable for extremely large datasets, or big data, to have been meaningfully analyzed due to the daunting task of doing so manually. Partially as a result of the increasing prevalence of big data requiring new tools for analysis and increased computing power enabling their development, the use of artificial intelligence (AI) in a variety of fields, including the biological sciences, has gained popularity in the past few years. AI can be defined as a system's ability to correctly interpret external data, to learn from such data, and to use those insights to achieve specific goals and tasks through flexible adaptation. 2 Machine learning, a technique that has enabled rapid advancements in the field of AI, is a subset of AI that is heavily based on pattern recognition and describes methods that help computers learn without explicitly being programmed to do so. 2 Deep learning, in turn, is a machine learning technique that arose in the early 2010s and uses computational models that exhibit similar characteristics to the hierarchal information processing in the human brain. 3 This technique is partially responsible for the increased power potential of machine learning algorithms and has been a large contributor to the recent surge of AI research. 4 Alongside AI, developments in biotechnology and the life sciences have progressed rapidly in the past several years. In fact, some biotechnological advancements may even be outpacing Moore's Law for computers. 5 Critical advances in biology such as next-generation DNA sequencing, de novo DNA synthesis, gene editing, genomics, and bioinformatics may result in the life sciences becoming a principal component of the next technological revolution. Given recent progress and projected trajectories, AI and biotechnology have both been described as key pillars of the fourth industrial revolution, with a growing number of technologies and applications arising at their point of convergence every year. [6] [7] [8] [9]
 Technological Convergence In tandem, AI and biotechnology increase the potential capability of the life sciences and lower the tacit knowledge required to perform previously tedious laboratory tasks. Widening the pool of actors will almost certainly accelerate technological developments, but this could also pose security risks. With increased access to these technologies, so comes the potential for a resulting high-consequence biological event, some of which may have such large consequences that they constitute a global catastrophic biological risk. 10 The risks posed by the convergence of AI with the biological sciences are plentiful and varied, ranging from AI-assisted identification of virulence factors to in silico design of novel pathogens, which could increase the probability of a global catastrophic biological risk event if accidently, carelessly, or deliberately released. For the purposes of this paper, convergence is defined as the technological commingling between the life sciences and AI such that the power of their interaction is greater than the sum of their individual disciplines. That is, the convergence of these 2 fields has a multiplicative effect. This interaction mostly manifests as AI-developments influencing advances, as well as new risks, in the life sciences and biotechnology rather than the other way around. A more comprehensive understanding of such emerging threats is highly important for the field of health security but demands rigorous examination and a systematic approach. In this paper, risk assessment for converging technologies will be discussed and evaluation of 3 existing biosecurity risk assessment frameworks undertaken. After analyzing their strengths and limitations, suggestions will be made regarding how such frameworks could be adjusted to be applicable to the risks arising at the convergence of AI and biotechnology. Some illustrative examples from the converging risk landscape will be discussed, followed by a proposed hybridized framework and key recommendations to begin addressing gaps in current health security risk assessment for converging technologies.
 Risk Assessments For Converging Technologies
 Challenges Assessing Emerging and Converging Risks Risk assessment for emerging technologies is an inherently difficult task and general frameworks are needed for evaluating the risks associated with technological convergence, including AI and biotechnology. Many frameworks are overly specific to a particular scenario or technology (such as narrowly focusing on synthetic biology), contain many assumptions and guesswork, and are not easily applied to other converging or emerging technologies. This is largely because different frameworks serve different purposes, so expectations that they can be implemented more generally are limited. However, the primary strength of a more generic framework for biosecurity risk assessments would be the ability to compare across different technologies or scenarios. Decision makers could then use such frameworks to be more informed as to which risks may be of higher priority. Compared to assessments of singular domains, analyzing risks of converging technologies presents an even greater challenge due to the added layers of complexity and the presence of multidisciplinary components. Converging risks such as those described herein are uniquely challenging to assess because their uncertainties multiply such that one must consider all possible outcomes of AI development in tandem with all possible outcomes of developments in the life sciences. For example, the evaluation criteria in a scenario-based risk assessment for virtual laboratories or 'cloud labs' that are too focused on the biological components may miss important bottlenecks, risks, or gaps that exist within the underlying AI components of the technology. Full assessment may not always be possible for some unpredictable technological advancements. Challenges also arise due to limited access to data and the inability to quantitatively assess emerging risks. Many frameworks include the potential adversary as a component to calculate risk. 11, 12 However, the adversary has a large range of potential attributes that are difficult to capture in their entirety. The adversary could be a nation, a nonstate actor, or an individual. When evaluating the likelihood that one of these adversaries could carry out ASSESSING ARTIFICIAL INTELLIGENCE AND BIOTECHNOLOGY RISKS an attack, assumptions must be made and variables considered on a spectrum. For example, a nation-state could be a low-resourced or well-resourced country. An individual could be anyone from an undergraduate biology student to a senior research professor. Such approaches fail to capture the range of possibilities and thus suffer from decreased granularity. Along these lines, without access to classified information, it is impossible to appreciate a full picture of adversary capability and intent. Furthermore, frameworks are sometimes focused on a technology's ability to aid in a specific element in the development process of a biological agent, which then makes it difficult to assess the risks of a full scenario in which it was implemented successfully. Conversely, assessments that solely focus on a full scenario fail to accurately capture the broad range of potential misapplications of a technology and consequently suffer from excessive specificity. After searching published literature, we identified 3 previously employed frameworks that inform the development of an assessment for converging technologies. These frameworks are examined in this paper in detail, with their strengths and limitations explored.
 Overview of Existing Frameworks The AAAS-FBI-UNICRI Framework The first framework was published in the 2014 report, National and Transnational Security Implications of Big Data in the Life Sciences, and was developed by a joint team consisting of members from the American Association for the Advancement of Science (AAAS) Center for Science, Technology, and Security Policy, the Biological Countermeasures Unit of the Federal Bureau of Investigation (FBI) Weapons of Mass Destruction Directorate, and the United Nations Interregional Crime and Justice Research Institute (UNICRI). 11 The main elements of the AAAS-FBI-UNICRI framework are highlighted in Table 1 . A strong component of this framework is the inclusion of security specialists, in addition to experts from multiple disciplines to evaluate the risks. While being one of the only attempts at risk assessment of a converging technology, the framework is limited in generality because it uses a scenario process to assess risks and its evaluation criteria are heavily skewed to big data and do not neatly map on to other converging technologies such as AI, despite many commonalities between them. The lack of generality is best illustrated by the framework table itself in which the ''Needed scientific expertise and skills'' block is divided into 2 sections to cover all 3 scenarios evaluated. This may be due to the fact that the working group developed the risk scenarios prior to developing the framework itself, resulting in a framework with a narrow scope. When attempting to implement a scenario where a malicious actor uses AI-enabled cloud labs to synthesize a pathogen in silico, the narrowness of this framework becomes more evident. The parameter, ''vulnerabilities in data repositories, software, and/or underlying cyber infrastructure'' 11 does not map onto this scenario since none of those 3 vulnerabilities are necessarily present in the case of cloud labs accessed through legitimate channels, albeit for nefarious purposes.
 The NASEM Framework The second framework emerged from the National Academies of Science, Engineering, and Medicine (NASEM) 2018 report, Biodefense in the Age of Synthetic Biology. The NASEM framework (Table 2 ) is strong in that it actively considers the synergy with other technologies as a factor to calculate level of concern. The report also considers machine learning and automation as potential technologies that could facilitate the development of a biological agent. This framework, however, excludes actors who may be beneficial, considering the broad range of potential adversaries. The limitation in using this framework for converging AI and biotechnology risks is that the scope is too specific and restricted to risks from synthetic biology and technologies relevant to the design-build-test cycle. The authors of the report are transparent about these limitations and recognize that every factor may not map on to a specific technology. If one wanted to apply this framework to evaluate the level of concern posed by risks from the convergence of AI and biotechnology, they could face difficulties since the questions used to evaluate the factors are heavily skewed toward synthetic biology. However, since the framework was developed to assess capabilities of a technology to enable harm more 12 This framework (Table 3 ) benefits from including a governability assessment that is used if the average value of the factors from the risk assessment portion are ''medium'' or ''high.'' While assessing governability for approaches to governance has unquestionable value, it can also be an indicator for probability. If a technology is less governable, it is likely to have a higher chance for misuse and resulting risk. A potential weakness in Tucker's framework is the use of an ordinal scale with just 3 values (high, medium, or low). A spectrum spanning a range of only 3 could result in lost information and decreased specificity, which may have negative downstream effects in determining which technologies pose the greatest or lowest risk. Another potential shortcoming is that the framework considers the imminence of the technology to be of equal weight to the other factors in calculating risk. As a result, it could potentially fail to accurately categorize a long-term, high-consequence risk as one worth governing. In a 2017 working paper, Koblentz et al expanded on Tucker's framework, proposing the Tucker-Koblentz framework, which includes the range of applications for misuse as an indicator of risk. 14 This is particularly useful in the context of the converging technologies since they tend to have synergistic effects that increase the range of appli-cations. The authors also noted how risk assessments are implicitly impacted by the anticipated benefits of a technology since such technologies are more likely to be intensely pursued by researchers and private companies, which results in accelerated innovation and diffusion of knowledge. This perspective is useful in the context of the convergence of AI and biotechnology since the range of beneficial applications is rather large.
 The Converging Risk Landscape The overlap between AI and biotechnology is already considerable, with increasing integration of AI into various facets of the biological sciences likely to continue for the foreseeable future. 9 This poses a challenge for the health security risk assessment frameworks previously described and requires further understanding of the risks that need to be prioritized for examination. This section provides an overview of some of the intersections of AI and biotechnology that could be considered for future risk assessment exercises. These sections are presented as illustrative examples and are not meant to propose a taxonomy of risks.
 Synthetic Biology Increasingly, engineering principles are being applied to biology for research and manufacturing purposes. Synthesis and design of existing and novel microorganisms have many legitimate and beneficial applications; however, these same technologies could be used to engineer pathogens for malicious use. AI and machine learning have the potential
 ASSESSING ARTIFICIAL INTELLIGENCE AND BIOTECHNOLOGY RISKS to greatly aid in the design-build-test stages of the synthetic biology process. 15 Current DNA synthesis methods and gene editing technologies, such as the CRISPR-Cas9 system, still pose a high technical barrier for viable pathogen creation. Nonetheless, several recent controversial experiments have showcased the ability to de novo synthesize live viruses, including horsepox and the 1918 flu. 16 Research on emerging techniques, such as enzymatic synthesis of DNA, aims to revolutionize the field of synthetic biology in the next decade, with the end goal being benchtop printers able to synthesize gene-length sequences of DNA to completion. 17 Advances in this space are not only increasing the potential power and capability of synthetic biology but are also lowering the technical barriers and tacit knowledge required for its widespread use. 18 Within the realm of synthetic biology, AI could potentially lower some of the barriers for a malicious actor to design dangerous pathogens with custom features. It is worth noting, however, that AI will have a differential effect on technological hurdles, with some bottlenecks remaining difficult to traverse. For a more detailed examination of how the difficulty of design steps can affect the probability of a successful biological attack, we recommend the paper by Sandberg and Nelson also published in this issue. 19 AI has enabled a substantial reduction in the time required to design and test functionality in fields such as protein design. 20 It may also lower barriers in the design stages of the design-build-test cycle and could be used to identify optimal mutations of a pathogen that enhance virulence factors, expand host range, alter routes of transmission, encode resistance to medical countermeasures, and intensify mortality. 21 Genetic changes could be identified and made to a virus that is traditionally benign in nature, allowing it to bypass screening procedures currently used by mail-order DNA synthesis services. Similarly, AI could be leveraged to circumvent field detection technologies by identifying code sequences that could be altered without changing function. 11 Deep learning could potentially identify genetic functions that code for vulnerabilities and interconnections between the immune system and microbiome. It may be possible that malicious actors could leverage this information to engineer ''precision maladies,'' or pathogens that target important mechanisms the immune system or the microbiome of specific subpopulations. 22 It is important to note, however, that simply identifying certain vulnerabilities does not mean actors can then freely make changes, significant bottlenecks would still have to be traversed in the overall development process and the feasibility of targeting specific subpopulations would remain an unknown. Moreover, in the past few years, several studies have been published on predicting the pathogenic potential of natural and synthetic DNA using AI. Although improving biosecurity is one of the motives for this research, the results of this work may have the reverse effect and pose information hazards if the methods and algorithms are published. PaPrBaG and DeePaC are examples of machine learning algorithms predicting pathogenic potential that have made all source code available online. 23, 24 These algorithms could potentially aid a bad actor with the intent of designing a pathogen that can cause maximal harm. AI can not only assist in the design step of the process, but it can also, to a lesser degree, support the building and testing phases. The rise of AI-enabled cloud labs to automate procedures that used to require hands-on, tacit knowledge to perform tasks could further widen the pool of actors who could perform certain biology experiments. Cloud labs could allow a malicious actor to perform synthesis and carry out subsequent testing undetected and with greater ease. 25 The next evolution of cloud labs may further enable AI scientists capable of carrying out experiments in entirety by developing and testing hypotheses, interpreting results, and amending hypotheses for retesting, thus lowering the technical barrier even further. 26 Furthermore, directed evolution of pathogens, which spans all 3 phases of the design-build-test cycle, is another route a malicious actor could take in the development of a customized pathogen. 13 AI techniques have already been leveraged to accelerate this process and guide directed evolution in protein engineering. 27 Overall, in the future, it may be possible for a bad actor to perform complete hands-off, in silico design, building, and testing of a novel or recreated pathogen by leveraging the technologies and advancements outlined above. 25, 28 The potential democratization and synergy of these technologies pose potentially high-consequence risks that could elevate to a globally catastrophic level if not recognized and mitigated.
 Vulnerabilities and AI Systems As technologies in the biological sciences continue to be developed, AI-enabled offensive cyber-attacks may become an increasing biosecurity threat. For example, cybersecurity vulnerabilities in machines such as DNA synthesizers, could allow the introduction of malware to corrupt the design or audio-record printed DNA sequences, disrupt laboratory biosecurity, or allow intruder access to sensitive information. 29, 30 In addition, AI system vulnerabilities could also have downstream effects on human health. If healthcare becomes increasingly automated with the use of AI, so does increased risk for cyber-intrusion and subsequent manipulation of medical devices, which could result in harm for the end-user. TrapX Security Labs notes in their latest MED-JACK report 31 that many medical devices and equipment in use today, such as infusion pumps, medical lasers, LASIK surgical machines, and life support equipment are vulnerable to hijacking by malicious actors. Wireless connectivity is becoming increasingly common in implantable medical devices, opening up the risk of malicious actors mounting attacks from anywhere in the world. 32
 O'BRIEN AND NELSON Beyond the risks associated with medical device exploitation, it is possible that in the future computer systems will be integrated with human physiology and, therefore, pose novel vulnerabilities. Brain-computer interfaces (BCIs), traditionally used in medicine for motor-neurological disorders, are AI systems that allow for direct communication between the brain and an external computer. 33 BCIs allow for a bidirectional flow of information, meaning the brain can receive signals from an external source and vice versa. The neurotechnology company, Neuralink, has recently claimed that a monkey was able to control a computer using one of their implants. 34 This concept may seem farfetched, but in 2004 a paralyzed man with an implanted BCI was able to play computer games and check email using only his mind. 35 Other studies have shown a ''brainbrain'' interface between mammals is possible. 36, 37 In 2013, one researcher at the University of Washington was able to send a brain signal captured by electroencephalography over the internet to control the hand movements of another by way of transcranial magnetic stimulation. 38 Advances are occurring at a rapid pace and many previous technical bottlenecks that have prevented BCIs from widespread implementation are beginning to be overcome. [39] [40] [41] Research and development of BCIs have accelerated quickly in the past decade. 42 Future directions seek to achieve a symbiosis of AI and the human brain for cognitive enhancement and rapid transfer of information between individuals or computer systems. 34 Rather than having to spend time looking up a subject, performing a calculation, or even speaking to another individual, the transfer of information could be nearly instantaneous. There have already been numerous studies conducted researching the use of BCIs for cognitive enhancement in domains such as learning and memory, perception, attention, and risk aversion (one being able to incite riskier behavior). 43 Additionally, studies have explored the military applications of BCIs, and the field receives a bulk of its funding from US Department of Defense sources such as the Defense Advanced Research Projects Agency. 44 While the commercial implementation of BCIs may not occur until well into the future, it is still valuable to consider the risks that could arise in order to highlight the need for security-by-design thinking and avoid path dependency, which could result in vulnerabilities-like those seen with current medical devices-persisting in future implementations. Cyber vulnerabilities in current BCIs have already been identified, including those that could cause physical harm to the user and influence behavior. [45] [46] [47] [48] In a future where BCIs are commonplace alongside advanced understandings of neuroscience, it may be possible for a bad actor to achieve limited influence over the behavior of a population or cause potential harm to users. This issue highlights the need to have robust risk assessment prior to widespread technological adoption, allowing for regulation, governance, and security measures to take identified concerns into account.
 Combined Illustrative Framework Given the rapid developments in AI and biotechnology and the convergence of these 2 areas, existing risk assessment frameworks could be adapted to properly examine emerging threats. Table 3 is an illustrative framework that adapts the original AAAS-FBI-UNICRI framework by removing some elements while adding some from other frameworks to improve its capability to assess converging risks, such as those posed by AI and biotechnology. Convergent risk scenarios from this paper are included as examples of how this framework might be implemented; the completed boxes presented in the table are illustrative only and not meant to be conclusive, as there was no large-scale expert review and consensus. In Table 4 , the adversary spans the same range for each element, which may suggest it is a variable that might not be worth including or at least reconsidered. However, a comprehensive assessment by multidisciplinary experts with multiple scenarios would need to be conducted to determine this. In this framework, rather than using an ordinal scale of only 3 (low, moderate, high) to assess each parameter, an ordinal scale of 5 (very low, low, moderate, high, very high) was used to increase specificity and better delineate between scenarios. Each parameter is meant to be evaluated based on the projected timeline. For instance, in the case of BCIs, it is assumed the parameter will be highly accessible and democratized in the long-term. Governability was included as a component contributing to probability since a technology convergence that is less governable may be more likely to be maliciously used. Democratization, called ''accessibility'' in other frameworks, was included because a technology that is more accessible to a wide range of individuals is assumed to have higher potential for misuse. The remaining evaluation criteria (vulnerabilities, needed skills and expertise, magnitude of potential consequences, and existing countermeasures) were selected because they are common to many of the frameworks reviewed herein and we believe are integral to any risk assessment framework. ''Vulnerabilities'' is meant to describe and illuminate any gaps that could be exploited to cause harm. ''Needed skills and expertise'' is meant to describe the scope of potential adversaries and help identify potential bottlenecks. ''Magnitude of potential consequences'' and ''existing countermeasures'' are meant to contextualize the scope of potential harm and balance that with any mitigating factors in place. As one might expect, this combined framework has many limitations. For example, depending on the type of pathogen designed or synthesized, the high variance changes the amount of expertise needed and potential consequences over time, as does the nature and sophistication of the type of BCI exploitation. The framework also includes uncertainties related to the timeline and assessing governability, which highlights the high degree of assumptions required to complete and use it. Moreover, the framework lacks a standardized mechanism for evaluating factors in a way that calculates risk. The hope is that explicitly identifying ASSESSING ARTIFICIAL INTELLIGENCE AND BIOTECHNOLOGY RISKS these limitations can help illustrate the gaps and challenges in developing a comprehensive framework for converging technologies that may pose high-consequence risks.
 Recommendations
 Risk Assessment Framework Approaches A risk assessment framework is needed that is applicable to converging technologies in the life sciences that balances generality to capture a broad scope but also maintains enough specificity to capture nuances. It is paramount that such frameworks be developed and implemented by a multidisciplinary team of experts that includes both security officials and scientists. Such a strategy allows for a more comprehensive and, subsequently, more accurate depiction of the potential risks. To further increase robustness, existing frameworks should be iteratively built upon and improved, such as the combined framework presented in this paper. A general framework intended to apply to a diverse range of converging or emerging technologies should be developed prior to determining the scenario to be evaluated. It can then be revised as scenarios or technologies are assessed. This would benefit the applicability of such a framework to other scenarios or technologies rather than narrowly limiting it to a specific issue. The role that time plays in calculating risk for an emerging or converging technology should be weighed. While a technology that is assumed to not be relevant until the far future may not pose an immediate risk, it should still be considered, as it could pose a significant threat once the technology materializes. Additionally, projected timelines for emerging technologies are inherently uncertain and technologies may develop more rapidly than expected. Therefore, it is worth considering projected timelines as a framing element rather than a factor to calculate risk.
 Revisiting the AAAS-FBI-UNICRI Framework As 5 years have passed since publication of the initial AAAS-FBI-UNICRI report, we recommend this working group reconvene to reassess the risks outlined in this paper and update their framework to be more inclusive of other converging threats. As noted in their last report, ''if scientists and security experts can assess the risks and benefits together and on a routine basis, the potential risk that biological [big data] would be used by adversaries decreases.'' We believe that this team is best equipped to develop an iterative set of frameworks related to this topic because the workgroup includes knowledgeable experts from a wide range of disciplines and the issues related to big data and AI are similar in nature. With perspectives from experts in law enforcement, the life sciences, and AI, the team is uniquely positioned to take on such a task. While their initial project focused narrowly on big data, it would be of great value to extend that scope to AI in the life sciences and provide relevant stakeholders in academia, government, and the private sector with an informational risk assessment that will help them make informed research, policy, and security decisions on these topics. This working group can leverage frameworks-such as the NASEM and Tucker-Koblentz frameworks-that have been developed since their initial report to improve upon their own. The Koblentz et al 14 paper could provide them with an insightful review of the strengths and weaknesses of many previous frameworks and would be a useful resource as the working group develops the next iteration. Components from the NASEM framework could be leveraged to increase the generality and broadness of their framework. The illustrative framework presented in this paper (Table 4 ) may serve as a useful starting point for the joint team.
 Conclusion AI and biotechnology are both rapidly growing fields. The biosecurity risks arising at their points of convergence are vast, highlighting the need for comprehensive mapping and thorough, robust risk assessment. Current frameworks used for risk assessment in health security settings may fail to capture the nuanced areas of these converging fields and could benefit from considering generality and timeline for technology adoption. We recommend that a multidisciplinary working group comprised of AAAS, the FBI, UNICRI, and other specialists gather to formulate a framework to comprehensively assess risks the results from converging technologies such as AI and biotechnology as well as others that may emerge in the future. The combined framework presented in this paper could serve as a starting point for the development of more robust biosecurity risk assessments for converging technologies like AI and biotechnology. While this new framework does not solve all of the identified challenges, it explicitly elucidates the gaps in our ability to accurately assess risks associated with emerging and converging technologies and will hopefully facilitate adaption of previous frameworks in order to develop a more inclusive health security risk assessment that properly balances generality and specificity and is regularly and iteratively revisited. Table 1 . 1 Main AAAS-FBI-UNICRI Framework Elements to Evaluate the Risks of Big Data 11 Risk Assessment Probability Consequences Adversary Vulnerabilities in Needed scientific expertise and skills Severe consequences Sufficient existing data repositories, software, and/or underlying cyber infrastructure To exploit vulnerabilities in the system To use big data analytics to design harmful biological agents to economics, political system, society, health, environment, and/or agriculture countermeasures
 Table 2 . 2 Main NASEM Framework Elements to Evaluate Level of Concern of Technologies in Synthetic Biology 14 Level of Concern About the Capability Usability of the Technology Usability as a Weapon Requirements of Actors Potential for Mitigation Ease of use Production and delivery Access to expertise Deterrence and prevention capabilities Rate of development Scope of casualty Access to resources Capability to recognize an attack Barriers to use Predictability of results Organizational footprint Attribution capabilities Synergy with other requirements Consequence management capabilities technologies
 Table 3 . 3 Main Tucker Decision Framework Elements to Assess Misuse and Governability 13 Risk of Misuse Governability Accessibility Embodiment Ease of misuse Maturity Magnitude of potential harm Convergence Imminence of potential misuse Rate of advance International diffusion Note: The elements were determined by averaging the values (high, medium, or low) yielded by the parameters. Governability was determined if risk of misuse is medium or high.
 Table 4 . 4 Illustrative Framework Adapted from AAAS-FBI-UNICI, NASEM, and Tucker Frameworks [12] [13] [14] [15] Convergent Risk Scenarios In Silico Design In Silico Synthesis Brain-Computer of a Pathogen of a Pathogen Interface Exploitation
 Volume 18, Number 3, 2020


A model of pathways to artificial superintelligence catastrophe for risk and decision analysis

Introduction An important issue in the study of future artificial intelligence (AI) technologies is the possibility of AI becoming an artificial superintelligence (ASI), meaning an AI significantly more intelligent than humans in all respects. Computers have long outperformed humans in specific domains, such as multiplication. As AI advances, humans are losing in more and more domains, as seen in the recent Jeopardy! victory of IBM's Watson computer. These AI can greatly help society, and also pose some risks, but they do not fundamentally change the status quo of humans being in ultimate control of the planet. However, some scholars propose that AIs could be designed that significantly exceed human capabilities, changing the status quo, with results that could be either fantastically good or catastrophically bad for the whole of humanity, depending on how the initial 'seed' AI is programmed (Bostrom, 2014; Good, 1965; Yudkowsky, 2008) . This proposition is controversial, 1 but if true, the implications are enormous. Given the stakes, the proposition should be taken seriously even if it only has a small chance of being true (Ćirković, 2012) . In this paper, we present a graphical model that depicts major pathways to catastrophe caused by ASI seizing control from humanity. We call the model the Artificial Superintelligence Pathways Model or ASI-PATH. The model's pathways involve AIs that recursively self-improve, i.e. AIs that create successively more intelligent AIs via improvements to their hardware or software. 2 The model includes both the human process of building the AI and some aspects of the AI itself. The model uses established modelling paradigms from risk and decision analysis, namely fault trees and influence diagrams, in order to depict combinations of events and conditions that could lead to ASI catastrophe, as well as interventions that could avoid ASI catastrophe. Model structures are derived from published literature on ASI catastrophe risk. no attempt is made to quantify model parameters beyond the use of Boolean variables because many of the parameters are highly uncertain; quantifying them rigorously would require significant research beyond the scope of this paper. More generally, given that the broader study of ASI is young and in flux, we expect the ASI-PATH model to evolve in future work. The model presented here can thus be treated as ASI-PATH 1.0. even without quantified parameters, the models can serve as communication tools and guides for actions to reduce ASI catastrophe risk. Furthermore, if parameter values are quantified, then the models can be used for quantitative risk analysis, technology forecasting and decision analysis. By characterising the space of possible AI futures, the models help show, amongst other things: what could happen; where key points of uncertainty are; which short-term and medium-term precursor events to monitor; and what types of interventions could help avoid harmful outcomes and promote beneficial outcomes. The model presented here thus offers a foundation for rigorous quantitative evaluation and decision-making on the risk of ASI catastrophe. Focusing on ASI catastrophe scenarios, we do not mean to imply that scenarios involving positive ASI outcomes are less important. A global-scale ASI catastrophe would clearly be important, but a global-scale positive ASI event could be similarly important, including because it could help address other global catastrophic risks. That said, much of the modelling presented in this paper can readily be extended to positive ASI scenarios. The following section provides detailed background on the ASI scenarios, ASI risk-reduction interventions, and the modelling paradigms used in the paper. Section 3 presents the ASI-PATH model by showing a series of interconnected model sections. Section 4 concludes. An Appendix compiles definitions of select technical terms.
 Conceptual background
 AI catastrophe scenarios We focus on scenarios in which AI becomes significantly more intelligent and more capable than humans, resulting in an ASI causing a major global catastrophe. Specifically, we focus on scenarios involving the following steps: (1) Humans create a seed AI that is able to undergo recursive self-improvement in either hardware or software. (2) The seed AI undergoes recursive self-improvement, resulting in a takeoff of successively more intelligent AIs. (3) The takeoff results in one or more ASIs. (4) The ASI(s) gain decisive strategic advantage over humanity, meaning 'a level of technological and other advantages sufficient to enable it [the AI] to achieve complete world domination' (Bostrom, 2014, p. 78 ). ( 5 ) The ASI(s) uses their decisive strategic advantage to cause a major global catastrophe, such as (but not necessarily limited to) human extinction. 3 Whilst other AI scenarios may also be able to bring great benefits or harms to humanity, this type of scenario has been the subject of considerable scholarly and popular attention, enough to merit its own dedicated treatment. We remain intentionally mute -with one important exception -on what qualifies as a major global catastrophe. The nature of major global catastrophe is a thorny moral issue, touching on issues such as the value of distant future generations. 4 The ASI scenarios raise additional moral issues, such as whether the ASI itself could be inherently morally valuable. How these moral issues are resolved affects how specific seed AIs are to be evaluated, but it does not affect the modelling presented in this paper. The one exception -the one position this paper takes -concerns the mere existence of superintelligence. We assume, for purposes of this paper and without committing our own views, that the mere existence of an ASI with decisive strategic advantage over humanity does not itself qualify as a major global catastrophe. In other words, this paper takes the position that it is not inherently catastrophic for humanity to lose control of its own destiny. We recognise that this is a contestable position, and in using this position in this paper we do not mean to advocate for it. Instead, we use this position in this paper because much discussion of superintelligence effectively takes this position and because it makes for a reasonable starting point for analysis. Similarly, we have no comment on the relative merits of various concepts for safe (or otherwise desirable) seed AI that have been proposed. These concepts have been discussed using terms such as 'Friendliness' (Yudkowsky, 2001 (Yudkowsky, , 2012 , 'safe AI scaffolding ' (omohundro, 2012, 2014) , 'AI safety engineering' (Yampolskiy & Fox, 2013) and 'motivation selection' (Bostrom, 2014) . Distinguishing between these is also an important task but again beyond the scope of this paper. Instead, we use the generic term 'safe AI' throughout the paper without reference to what qualifies as a safe AI and without intending preference for any particular 'safe AI' concept. The modelling presented here is of sufficient generality that it could be applied to a wide range of conceptions of desirable outcomes and desirable seed AI.
 Risk reduction interventions Models like ASI-PATH can help people make better decisions towards ASI safety -that is, towards reducing the risk of ASI catastrophe. each model component suggests opportunities to reduce this risk. The following bulleted list describes the risk reduction intervention options used in ASI-PATH. These are some (but not all) potential risk reduction intervention options, drawing heavily on Sotala and Yampolskiy (2015) . The interventions can be made by a range of actors, including governments, companies, universities, funding bodies, civil society and AI researchers (including both ASI researchers and their colleagues in other branches of AI). • ASI research risk review boards. review boards would evaluate the riskiness of specific ASI research projects in order to steer research in less risky directions. The review boards could steer research, for example, by disallowing certain projects, directing funds away from riskier projects and towards safer ones and establishing guidelines for publication. Successful review boards would need to correctly evaluate the riskiness of AI research, which could be difficult (Sotala & Yampolskiy, 2015) . review boards might also be limited in the scope of research they review. A board set up by one country may not be able to review research conducted in other countries, though this could be addressed via international treaty (Wilson, 2013) . review boards could also overlook ASI projects that operate secretly (Sotala & Yampolskiy, 2015) . that limit the AI's ability to affect the rest of the world so that it does not gain decisive strategic advantage. 5 Confinement can be applied to the seed AI, the recursively self-improving AIs or the ensuing ASI(s). For example, AI software could be designed such that the AI can only answer questions; this is known as 'oracle AI' (Armstrong, Sandberg, & Bostrom, 2012) . Another type of confinement would keep AI completely disconnected from the world, for example to test the AI's safety (Yampolskiy & Fox, 2013 ). In such scenarios, the first AI(s) must be designed to conduct enforcement.
 Fault tree and influence diagram modelling ASI-PATH uses conventions from two established modelling paradigms: fault trees and influence diagrams. Fault trees offer a simple but powerful method for modelling sets of interrelated possible scenarios. Fault trees have been used in modelling risks in a wide variety of contexts, including nuclear power plant safety (rasmussen, 1981) , homeland security risk (Cheesebrough, 2010) , nuclear war (Barrett, Baum, & Hostetler, 2013; Davis, 1989) and other kinds of systems (Bedford & Cooke, 2001; Kumamoto & Henley, 1996) . Fault trees represent the ways that events and conditions could combine to be at fault for causing a particular outcome, in this case an ASI catastrophe. each node in the tree represents a particular event, such as an attempted creation of a seed AI, or a condition, such as the presence of some technology keeping the AI safe. The 'top event' node in the tree represents the scenario outcome. Below the top node, the tree branches out with additional nodes corresponding to different ways that the top node can be reached. each layer in the tree represents the combination of events and conditions that could lead to the outcome in the layer directly above it. nodes are connected by Boolean logic gates, such as or, AnD and noT gates. The tree thus shows a set of possible scenarios that could result in the top event. Influence diagrams represent the ways that certain events can influence subsequent events. The events can be random variables, determined by factors external to the model, or decision variables, determined by decisions informed by the model. Influence diagrams thus can be thought of as Bayesian networks with the addition of decision nodes that represent variables whose values can be chosen instead of having a random value (Pearl, 1988) . Influence diagrams have been used for decades in decision analysis to inform business, government and other policy decisions under conditions of uncertainty (Clemen & reilly, 2001; edwards, Miles, & von Winterfeldt, 2007) . ASI-PATH uses influence diagrams to model the risk reduction interventions described in Section 2.2. The figures in this paper use standard visual conventions for fault trees and influence diagrams. Connector lines are used to connect nodes representing events and conditions. Vertical connectors with no arrows are used for fault trees; the lower node is at fault for causing the upper node. Horizontal connectors with arrows are used for influence diagrams; the left node influences the right node. Intervention options appear as unrounded rectangles with white text on black background; in influence diagram terminology, these are called decision nodes. other (non-intervention) events appear as rounded rectangles with black text on white background; these are called random-variable nodes. Some figures use both fault tree and influence diagram conventions, as well as both decision nodes and random-variable nodes. 7 The third layer in Figure 1 contains more model detail: six conditions that must all be in place in order for ASI catastrophe to occur. If any of these conditions are missing, then there will be no AI takeoff, or the resulting ASI will not cause catastrophe. Starting on the left side: in order for a seed AI to be created and have the capability to gain decisive advantage, three conditions must all be met:
 The artificial superintelligence pathways model
 Model overview
 Superintelligence from AI takeoff is possible There is some debate about whether it could ever be physically possible for ASI to be created via recursive self-improvement. There is also debate on whether humans are capable of creating such ASI. For example, Bringsjord, Bringsjord, and Bello (2012, pp. 403-406) argue that it is far from certain that humans would ever be able to create AI with human-level intelligence, or that such an AI would be able to recursively create more intelligent AI to arrive at superintelligence.
 Seed AI is created For ASI catastrophe to occur, there must be a project in which humans build a seed AI capable of becoming superintelligent via recursive self-improvement. Such an AI is presumably not going to spontaneously materialise out of thin air. There are a variety of ways in which ASI projects could build a seed AI (Section 3.2).
 Containment fails If containment fails, then the ASI gains decisive strategic advantage. Containment measures could include takeoff limits, such that the seed AI does not begin recursive self-improvement, or such that successive AIs do not continue recursive self-improvement so as to become superintelligent. other containment measures could prevent a post-takeoff ASI from gaining decisive strategic advantage, for example via confinement, human enhancement or AI enforcement (Section 2.2). Moving to the right side of Figure 1 , third layer: in order for an ASI to have unsafe goals -in order for it to not use its decisive strategic advantage to cause catastrophe -three conditions must all be met:
 Human attempts to make AI goals safe fail Goal safety can potentially be built into an ASI by the AI designers. Indeed, some people are currently calling for ASI developers to make their AIs' goals safe (Bostrom, 2014; Yudkowsky, 2012) , and there is ongoing research into various ways to make ASI goals safe (Goertzel & Pitt, 2012; Yampolskiy & Fox, 2013) . There is some debate about how to design safe ASI goals -more on this below.
 AI does not make its own goals safe Several people have suggested that AIs could make their own goals safe even if the goals were not designed to be safe (Hall, 2011; Waser, 2008) . one possible reason could be that 'minds which are intelligent enough will, due to game-theoretical and other considerations, become altruistic and cooperative' (Sotala & Yampolskiy, 2015, p. 21) . others have disputed this point, arguing that without significant dedicated effort, catastrophe is likely to result (e.g. Bostrom, 2014; omohundro, 2008 Yampolskiy & Fox, 2013; Yudkowsky, 2001 Yudkowsky, , 2012 . For example, omohundro (2008) argues that almost any simple goal, such as playing chess well, could be unsafe. The AI would cause catastrophe in its relentless pursuit of that goal, such as by hoarding the world's resources to play a better game of chess.
 ASI is not deterred from acting unsafely To deter is to persuade another agent to refrain from taking some action by convincing the agent that the action is not in their own interest. even if an ASI has decisive strategic advantage and unsafe goals, it potentially could still be deterred from using its advantage to cause catastrophe. However, Bostrom (2014) argues that an ASI with decisive strategic advantage would be undeterrable. We now fill in some additional model details with additional fault tree layers, first for the possibility of limits to AI takeoff, then for human efforts to make the AI safe. The subsequent figures show some major relationships between decision nodes (rounded rectangles, black text on white background) and random-variable nodes (unrounded rectangles, white text on black background). The selection of relationships presented is not intended to be comprehensive. The decision nodes can influence other random-variable nodes besides those shown in the figures. There can also be other decision nodes representing other interventions not modelled in this paper. What is shown is intended only to be illustrative of how decision nodes can be used to incorporate risk reduction interventions in the ASI-PATH model.
 Seed AI creation A critical question is how and when an AI research and development project could build a seed AI. When the seed AI is created matters for several reasons. earlier creation could make takeoff slower and easier to contain (Section 3.3), but earlier creation also gives less opportunity to design a safe seed AI. Figure 2 presents additional detail on seed AI creation, extending the 'Seed AI is created' node from Figure 1 . Figure 2 shows two major types of seed AI: AI of novel design and whole brain emulation. Figure 2 also has a catch-all 'other seed AI' branch for any types of seed AI that do not fit into the two major types. This section of the model draws heavily on Bostrom (2014, ch. 2) . AI of novel design refers to seed AI whose design is at least novel relative to the human brain. In other words, the seed AI design is at least in part not based on the human brain. Four different pathways to seed AI of novel design are modelled: (1) Artificial general intelligence (AGI) better AIs, an AI at least as smart as humans in all respects would also be able to programme better AIs, thereby beginning a recursive self-improvement process (Bostrom, 2014; Sotala, 2012) . The AGI model branch contains a lower layer to model one pathway to AGI that has been identified, which comes from networks or organisations of AIs, for example if 'the Internet might one day "wake up"' (Bostrom, 2014, p. 49) . The lower layer also includes a catch-all 'other AGI' node. (2) Self-improvement specialist is seed AI that is 'sub-human in most respects, but with a domain-specific talent for coding and AI research' (Bostrom, 2014, p. 29) . In other words, the AI is a specialist in recursive self-improvement. For example, Sotala (2012) and Bostrom (2014) discuss ways in which superintelligence could be achieved by AI with initially subhuman general intelligence but greater than human ability to take control of other networked systems, make copies of itself and to coordinate between copies of itself. (3) Neuromorphic AI is seed AI that is based in part, but not entirely, on the human brain: a partial brain emulation. For example, neuromorphic AI could be based on human patterns of cognition whilst lacking human motivations; the difference in motivations could make it harder to achieve goal safety. ( 4 ) Other novel design, a catch-all node for any pathways to novel seed AI designs that do not fit into the three identified pathways. Moving on to the second major branch of Figure 2 -the second major type of seed AI: Whole brain emulation is seed AI that emulates the entirety of the human brain (Alstott, 2014; Sandberg & Bostrom, 2008) . Because the human brain is capable of designing AIs, a whole brain emulation AI presumably could also design AIs. A whole brain emulation AI could also self-improve using more and faster hardware, something that biological human brains cannot readily do. By imitating the human brain, whole brain emulation may require less intellectual breakthrough than seed AI of novel design -it 'relies less on theoretical insight and more on technological capability' (Bostrom, 2014, p. 33) . Whole brain emulation requires each of the three steps to occur: (1) Scanning, in sufficiently detail, of one or more whole human brains. The brain scan creates an image of the whole brain to be emulated. (2) Translation, using data analysis and image processing, to construct digital representation(s) of the brains neuronal network and/or any other relevant information. The translation thus converts the scanned image into a format that can be emulated. (3) Simulation, which creates a whole brain emulation seed AI based on the whole brain translation. Figure 2 shows the decision node 'review Boards' (i.e. ASI research risk review boards) influencing each of the model section bottom nodes. review boards could evaluate the safety of each type of seed AI development project. For projects found to be unsafe, review boards could recommend that the projects stop their work or that funders stop funding the projects. Such a board would be analogous to, for example, the review of risky biomedical research ('dual use research of concern' or DurC) by the united States Department of Health and Human Services (HHS; see e.g. oBA, 2012; Patterson, Tabak, Fauci, Collins, & Howard, 2013) . The DurC review helps determine whether HHS (which includes the national Institutes of Health) funds the research and what safety procedures the research must follow. Figure 2 shows separate arrows pointing to each of the bottom nodes instead of one arrow pointing to the top node ('Seed AI is created') because the influence of review Boards can be different for each bottom node. That is, review Boards could have different influence on different types of seed AI project. For example, perhaps review Boards are more effective for neuromorphic AI, which could require large dedicated projects, and less effective for networked AI, which could emerge from entrenched infrastructure (e.g. the Internet). The use of separate arrows illustrates how this model section would be coded, with different influence relations for different bottom nodes.
 AI containment Containment is defined in this paper as any measure that restricts an AI such that it does not result in an ASI gaining decisive strategic advantage. The restriction can be on seed AI, self-improving AI or ASI. Figure 3 models some key details about containment, extending the 'Containment fails' node in Figure 1 . Working down from the top of Figure 3 , containment fails if takeoff limits fail or if superintelligence containment fails. In other words, an ASI gains decisive strategic advantage if a seed AI takes off, becoming superintelligent, and if the resulting ASI is not contained. Starting on the left branch of Figure 3 , AI takeoff limits fail if either hard takeoff limits fail or soft takeoff limits fail, simply because these are the two types of takeoff: (1) Hard takeoff limits fail. A hard takeoff proceeds rapidly, with no opportunity for human intervention. Figure 3 2 ) Soft takeoff limits fail. A soft takeoff proceeds slowly, permitting significant human intervention in the takeoff process. Figure 3 models three soft takeoff scenarios (i.e. failures of soft takeoff limits): (1) hardware quantity, in which the AI takes off due to gradually increasing its amount of available hardware; 10 (2) Hardware quality, in which AI takes off due to gradually develops faster hardware; and (3) software quality, in which the AI gradually develops smarter AI software. Takeoff speed is important because it strongly affects how much opportunity there will be to manage the recursive self-improvement process. If a hard takeoff is more likely, then it is especially important to design the seed AI so as to achieve desirable outcomes because there will be no opportunity to adjust the AI once takeoff begins. Shulman and Sandberg (2010) and Goertzel and Pitt (2012) suggest that a soft takeoff would be more likely if the seed AI is created earlier because the hardware, programming languages, etc. available for the AI would tend to result in slower seed AI. Thus, to the extent that the seed AI creation model section (Section 3.2) can indicate when seed AI may be created, this can inform the success of containment. Figure 3 shows the decision node 'enhance Humans' (i.e. enhance human capabilities) influencing the three random-variable nodes below 'Soft takeoff limits fail' . In principle, enhancing human capabilities could help with a wide range of AI safety measures, but it could be especially useful for limiting soft takeoff. This is because soft takeoff involves AIs gradually gaining capabilities; enhanced humans may be able to keep up with the AI during much of the takeoff, ensuring that no unsafe ASI gains decisive strategic advantage. In comparison, a hard takeoff would by definition occur too quickly for even enhanced humans to keep up. As with Figures 2 and 3 show separate arrows pointing to each of the bottom nodes instead of one arrow pointing to the top node ('Soft takeoff limits fail') because the influence of enhance Humans can be different for each bottom node -for each type of soft takeoff. For example, perhaps enhance Humans is more effective for containing hardware quality soft takeoff than for containing software quality soft takeoff because enhanced humans could control the AI's access to the physical equipment necessary for hardware production. As with Figure 2 , the use of separate arrows illustrates how this model section would be coded, with different influence relations for different bottom nodes. Moving on to the right branch of Figure 3 , superintelligence containment fails if there is no superintelligence containment or if the AI escapes whatever superintelligence containment is used: (1) ASI is created without superintelligence containment. Some ASI development projects might not use any superintelligence containment. This could happen for example if the developers have no ability to use superintelligence containment, or if they believe the superintelligence containment would not work, or if they want the ASI to gain decisive strategic advantage. Whatever the reason, ASI cannot be confined if there is no superintelligence containment. Figure 3 shows the decision node 'review Boards' (i.e. ASI research risk review boards) influencing the node 'ASI is created without superintelligence containment' . review boards could monitor ASI r&D projects and require them to have some sort of containment. ASI escapes superintelligence containment. Superintelligence containment might not be designed and implemented well, and even world-class superintelligence containment may fail to contain some ASIs. Bostrom (2014, pp. 129-131, 145-148, 283) discusses many potential ways an ASI could escape containment, including persuading or tricking any gatekeepers who can remove containment, using methods of interacting with the outside world that containment designers failed to anticipate, or pretending to malfunction so that the containment is at least temporarily disabled. Figure 3 shows the decision nodes 'AI confinement' and 'AI enforcement' influencing the random-variable node 'AI escapes superintelligence containment' . AI confinement and AI enforcement are two types of superintelligence containment, so this influence relation follows from basic logic.
 Human attempts at making ASI goals safe Modelling ASI goal safety is a subtle and challenging task because there is no clear consensus on what it takes to make ASI goals safe. The study of ASI goal safety is, we think it is fair to say, in early stages, with a lot of uncertainty and disagreement. Because of this, one cannot model ASI goal safety with much specificity, or at least not with much confidence about the specifics. Therefore, the models in this section are intended as exploratory, in order to illustrate some of what is known and to promote open discussion with an eye towards future work. We expect to change these models pending such discussion and pending results of ongoing research on ASI goal safety.
 General failure types Figures 4-7 model general types of failure of human attempts at making ASI goals safe. Figure 4 models two major lines of current thinking about how humans could make ASI goals safe. The two lines of thinking are the left and centre nodes of layer 2, with the right node being a catch-all category for any other lines of thinking: (1) Failure of attempts involving mathematical proof. Some people involved in ASI goal safety research have called for goal safety measures to be built into the seed AI, and furthermore for the correctness of these measures to be mathematically proven before the AI is launched. For example, ASI developers could build in goal stability, such that the AIs goals would not change as it undergoes recursive self-improvement. It may be possible to mathematically prove goal stability prior to launch. 11 Mathematical proofs may be able to give AI designers and other observers a high confidence that the ASI's goals would be safe, though it would not guarantee safety with 100% probability (Brundage, 2014; Muehlhauser, 2013) . requiring mathematical proofs before launch could also delay launch, by creating an additional hurdle that must be cleared before the seed AI is launched. Given that the failure of ASI goal safety could result in major global catastrophe, requiring mathematical proofs before launching is thus in the spirit of the precautionary principle, a major policy paradigm for risky emerging technologies. 12 Finally, note that this line of thinking does not exclude goal safety measures that occur after seed AI launch. (2) Failure of attempts involving AI biasing during takeoff but not involving mathematical proofs. other people in ASI goal safety research have argued against mathematical proofs prior to seed AI launch, and instead argue for biasing the AI's goals towards safety as the AI recursively self-improves. This line of thinking argues that the AI's goals could be biased through human interaction with the AI as it grows and learns, using reward and punishment from a trainer/ biaser in a fashion somewhat analogous to what human children experience from their parents (Goertzel & Pitt, 2012) . Such AI biasing requires a soft takeoff; otherwise human interaction during takeoff would be impossible. Because mathematical proofs before launch are not sought, this line of thinking could enable earlier launch of the seed AI. There could be reasons for favouring earlier launch, for example in the context of a race between competing AI projects. However, supporters of requiring mathematical proofs have argued that the belief that proofs are unnecessary might give AI designers a dangerous misperception that an AI's goals would remain safe outside its training environment (Sotala & Yampolskiy, 2015, pp. 21-23) . ( 3 ) Failure of other types of attempts. This node is a catch-all for any attempt at ASI goal safety that does not involve either mathematical proofs or biasing during takeoff. Whilst other types of attempts are, to our knowledge, not prominent in ASI goal safety literature, there may nonetheless be other types of attempts that could achieve ASI goal safety.
 Y each of the three nodes in the second layer of Figure 4 has a lower box labelled x. The x box is a model module, essentially an abbreviation for a repeating model section. The abbreviation x is used to conserve space in the diagram, with no implications for the underlying model logic. We refer to this as a model module by analogy to modular computer programming. If the model is written in computer code, then modular programming could be used for x. The details of x are shown in Figure 5 . Figure 5 shows that the x module has five nodes connected by an or gate, each with a new module Y beneath it. The five nodes are: No attempt, meaning no attempt is made at ASI goal safety using the types of attempt shown in Figure 4 . Theory failure, meaning a failure to achieve ASI goal safety due to some flaw in the theory of the type of goal safety attempted. Theory failure could be due, for example, to a philosophical mistake, such as selecting a goal that turns out to be unsafe, as in an ASI that causes catastrophe in the relentless pursuit of better chess (omohundro, 2008) . Implementation failure, meaning a failure to correctly implement the ASI goal safety technique in the actual AI. Implementation failure could be due to such things as a coding error or a hardware glitch. Adversary failure, meaning the ASI goals become unsafe due to interference by some sort of adversary. The adversary could be a rogue member of the AI r&D team, an external party such as a competing ASI team or a misguided governing body, or another AI, as in AI enforcement. Other failure, a catch-all for failure modes not covered in the other four nodes. each of the five Figure 5 nodes has a module Y beneath it. Module Y is shown in Figure 6 . Module Y contains two nodes connected by an or gate: failure before and during takeoff. Some ASI goal safety measures must be taken before takeoff, such as those involved in designing and coding the seed AI. other safety measures must be taken after takeoff, such as those involving interacting with the AI so that its self-improvement proceeds in a safe direction. Figure 6 shows the decision node 'encourage Safety' (i.e. encourage research into AI safety) influencing the bottom random-variable nodes in Figure 6 . These influence relations indicate that encourage Safety could influence all types of human attempts at making ASI goals safe. ASI goal safety is not the only type of AI safety that one could encourage, but it is a notable type. Active efforts exist to promote ASI goal safety research, accompanied by arguments that this is an especially important task for improving ASI outcomes (e.g. Muehlhauser & Bostrom, 2014) . The use of separate arrows pointing to each of the bottom nodes in Figure 6 is analogous to the use of separate arrows in Figures 2 and 3 : to indicate that the influence can be different for different bottom nodes. Because of the x and Y modules, the bottom nodes in Figure 6 are each repeated 15 times, meaning that there are 30 different encourage Safety influence relations. Code of this model section could have different relations for each of the 30 bottom nodes. For example, perhaps it is relatively effective to encourage safety for 'Failure of attempts involving mathematical proof' -'no attempt' -'Failure before takeoff' because before takeoff, it is relatively easy to encourage ASI developers to make at least some attempt at mathematical proofs, and perhaps it is relatively ineffective to encourage safety for 'Failure of attempts involving AI biasing during takeoff but not involving mathematical proofs' -'Theory failure' -'Failure during takeoff' because after takeoff, there might not be enough time to develop adequate theory.
 Specific safety measures Figure 7 presents a different format for modelling human attempts to make ASI goals safe. Instead of organising the attempts by lines of thinking and general failure types, as in Figures 4-7 organises the attempts by the specific safety measures the attempts could involve. Figure 7 thus aims to model the details of what goal safety requires. The goal safety measures are grouped into two categories: measures built into the seed AI before its launch and measures taken during takeoff. This categorisation assumes that no human attempts could succeed after takeoff, since then the AI would be superintelligent and presumably impervious to human interventions. Figure 7 includes four types of safety measures that could be built into the seed AI before takeoff: (1) goal safety, meaning safety measures that result in the post-takeoff superintelligence having safe goals; 13 (2) Goal stability, meaning the AI keeps the same goals throughout recursive self-improvement; (3) openness to training/bias during takeoff, such as to permit 'attempts involving AI biasing during takeoff but not involving mathematical proofs' (Figure 3 and above); 14 and ( 4 ) other seed AI measures, a catch-all category for any other safety measures that could be built into the seed AI. Figure 7 uses two 'AnD/or?' logic gates. The reason for this unusual logic structure is because it is not clear whether the logic gates should be AnDs or ors, and indeed they potentially could be either. The 'AnD/or?' gates are thus a representation of model uncertainty, analogous to the use of probability distributions for parameter uncertainty. likewise, the gates can be coded by considering each combination of AnD and or, perhaps with a probability assigned to each combination. For the top logic gate: • An AnD gate would be used if, in order for the top node failure to occur, it is necessary for both layer 2 node failures to occur. That would mean that ASI goal safety could be achieved either via building measures into the seed AI or by taking measures during takeoff. For example, if the seed AI can be made safe before launch, then additional measures during takeoff might not be necessary. Indeed, if the takeoff is hard, then there may not be opportunity for additional measures during takeoff, and goal safety would depend on safety measures built into the seed AI. • An or gate would be used if, in order for the top node failure to occur, it is sufficient for either of the layer 2 node failures to occur. That would mean that AI goal safety would require measures built into the seed AI and measures taken during takeoff. For example, for an AI to be biased towards safe goals during takeoff, it might need certain safety measures built into the seed AI, such as openness to training/biasing during takeoff. For the bottom logic gate: • An AnD gate would be used if, in order for the top node failure to occur, it is necessary for all four bottom node failures to occur. That would mean that pre-takeoff ASI goal safety could be achieved via any one of the four safety measures. For example, if the seed AI is open to bias/training during takeoff, then it may not need to have safe or stable goals before takeoff, as its goals can be adjusted during takeoff. • An or gate would be used if, in order for the top node failure to occur, it is sufficient for any of the four bottom node failures to occur. That would mean that AI goal safety would require all four safety measures to be built into the seed AI before takeoff. For example, goal safety may be useless if the goals are not also stable because the goals would become unsafe during takeoff.
 Conclusion It is far from certain that ASI will ever be built, or that, if it is built, it will result in catastrophe. But given the stakes -given the extreme severity of the catastrophe that could result -ASI catastrophe risk merits careful analysis and management. The ASI-PATH model presented in this paper is one step towards ASI risk analysis and risk management decision-making. ASI-PATH uses fault trees and influence diagrams, which are established risk and decision analysis methodologies with a long history of success at informing risk management decisions. This history, and the experience it has generated, is leveraged in ASI-PATH for the analysis of ASI risk. one might think that ASI is such a novel risk that established methodologies would not be suitable. ASI risk does indeed have some significant novel characteristics, in particular the possibility of humans losing control to the ASI. However, core aspects of ASI risk are broadly similar to other risks. ASI risk involves sets of pathways, events and conditions involving interactions between humans and the technologies they develop. Analysis of ASI catastrophe pathways is thus not fundamentally different from analysis of pathways to other technology catastrophes. This is good news because it makes analysis of ASI catastrophe risk an easier endeavor. That said, ASI catastrophe risk remains poorly understood. These are early days for the study of ASI risk. The ASI-PATH model presented here and the understanding of ASI risk it represents are both likely to evolve as the field matures. This is to say that the ASI-PATH model is wrong -but then again, all models are wrong. Models like ASI-PATH can still be useful by helping people better understand the underlying risk, including the major points of uncertainty. For example, the modelling in this paper suggests that one key uncertainty is the combination(s) of measures needed to make ASI goals safe, hence the model uncertainty ('AnD/or?' gates) depicted in Figure 7 . To the extent that future research could reduce the uncertainty, it would be of value for understanding ASI risk and informing ASI risk management decisions. An important task for future work is the quantification of ASI-PATH parameters using real-number values instead of Boolean logic. Doing so would enable the evaluation of ASI risk and specific risk reduction interventions, towards the goal of taking the most effective actions to reduce ASI risk. Quantifying ASI-PATH parameters is a delicate task due to the significant uncertainty that exists about parameter values, but this only means that quantification should proceed carefully, not that it should not proceed. Indeed, given the stakes associated with ASI catastrophe and the corresponding opportunities for reducing ASI catastrophe risk, the entire study of ASI catastrophe risk should be a significant research priority.
 Notes 7. The use of white text on black background is not a standard influence diagram visual convention. We use it to further distinguish decision nodes from random-variable nodes in our figures. 8. The model can be adapted for analyzing safe ASI by switching the layer 2 right node to 'ASI actions are safe'; much of the lower layer modelling (discussed below) will be similar. 9. The term AGI can refer to more than just seed AI. Indeed, an ASI built from a seed AGI would likely also be an AGI, as the AI is unlikely to lose general intelligence as it undergoes recursive self-improvement. 10. one might argue that a 'hardware quantity' soft takeoff is unlikely on grounds that if an AI is able to access more hardware than its designers intended, it would likely be able to access much or all of the entire world's hardware all at once, causing a hard takeoff. Whilst quantifying scenario probabilities is beyond the scope of this paper, such probabilities can be plugged directly into the model. 11. For discussion of this and related ideas, see for example Yampolskiy and Fox (2013 ), omohundro (2012 , Soares and Fallenstein (2014), and russell, Dewey, and Tegmark (2015) . Details of failure modes for these ASI goal safety measures could be modelled in extensions of Figure 3 . 12. on the precautionary principle in the context of catastrophic risk, see for example Posner (2004) ; Sunstein (2009) . 13. Goal safety does not require the seed AI or subsequent pre-superintelligence AIs to have safe goals -thus included here are ideas like 'coherent extrapolated volition' (Yudkowsky, 2004) in which AI acquires safe goals during takeoff. 14. See also Soares (2015) for a characterisation of openness to training: 'corrigibility' , which an AI system would possess 'if it cooperates with what its creators regard as a corrective intervention, despite default incentives for rational agents to resist attempts to shut them down or modify their preferences' . Figure 1 1 Figure1presents the first three layers of ASI-PATH. The first two layers of Figure1should be fairly intuitive: an ASI catastrophe occurs if a seed AI takes off, resulting in a superintelligence with decisive strategic advantage over humanity, AnD if the ASI's actions are unsafe, such that it uses the decisive strategic advantage to cause catastrophe.8 The third layer in Figure1contains more model detail: six conditions that must all be in place in order for ASI catastrophe to occur. If any of these conditions are missing, then there will be no AI takeoff, or the resulting ASI will not cause catastrophe. Starting on the left side: in order for a seed AI to be created and have the capability to gain decisive advantage, three conditions must all be met:
 Figure 2 . 2 Figure 2. Detail on seed ai creation.
 Figure 3 . 3 Figure 3. Detail on containment failure.
 Figure 4 . 4 Figure 4. two lines of thinking on human attempts at making ai goals safe.
 Figure 6 .Figure 5 . 65 Figure 6. Explanation of the Y module used in figure 4.
 Human enhancement in this context refers to anything that could help humans keep up with AI, such that the AI does not gain decisive strategic advantage even during or after takeoff. Human capabilities could be enhanced, for example, via brain-computer interfaces or mind uploading. Human enhancement could reduce some ASI risks, but it could increase other ASI risks (e.g. if enhanced humans create unsafe ASI) or cause other problems (e.g. moral problems associated with modifying human nature). Additionally, human enhancement could require advanced technology that may not be developed before the seed AI, in which case human enhancement would not help with ASI safety. • AI confinement. AI confinement measures are restrictions built into the AI's hardware or software • Encourage research into ASI safety. AI safety research can be on any aspect of ASI software or hard- ware, or on the humans involved in ASI development. ASI safety research could be encouraged, for example, by increasing funding for ASI safety research, creating open source platforms for safe ASI development or cultivating norms of responsibility amongst ASI researchers. encouraging research can be an easier intervention to perform than something like review boards because encour- aging research can be done informally and outside institutional procedures. However, informal encouragement can sometimes be less effective when it lacks institutional clout. As with research risk review boards, encouraging research requires knowing whether a particular activity would increase or decrease ASI risks. • Enhance human capabilities.
 6 • AI enforcement. AI enforcement occurs when one or more AIs prevent other AI(s) from gaining decisive strategic advantage. As with confinement, AI enforcement can be applied to seed AI, self-improving AIs or ASIs; the enforcement can also be performed by any of these AIs, though more intelligent AIs may tend to be more effective enforcers. For AI enforcement to work, the AIs with strategic advantage must be motivated and able to stop other AIs. For scenarios with multiple AIs, there could be a first-mover advantage, such that the first AI(s) gain and keep strategic advantage.