Aligning AI Regulation to Sociotechnical Change

Introduction How do we regulate a changing technology, with changing uses, in a changing world? As artificial intelligence ('AI') is anticipated to drive extensive change, the question of how we can and should reconfigure our regulatory ecosystems for AI change matters today. In just the past decade, advances in both AI research and in the broader data infrastructure have begun to spur extensive take-up of this technology across society (D. Zhang et al. 2021) . As a 'general-purpose technology' (Trajtenberg 2018 ), AI's impact on the world may be both unusually broad and deep. It may even prove as 'transformative' as the industrial revolution (Gruetzemacher and Whittlestone 2020) . This may provide grounds for anticipation-and also for caution. Most of all, it is grounds for reflection on the choices that societies want to make and instil in the trajectory of this technology. While still at an early stage of development, uses of AI technology are already creating diverse policy challenges. Internationally, AI is the subject to intense contestation. Further AI progress, along with the technology's global dissemination, are set to further raise the stakes. It is clearly urgent to reflect on the purposes and suitability of the regulatory ecosystem for AI governance. The urgent question is; when we craft AI regulation, how should we do so? Many AI governance approaches at both the national and international level, remain hampered by siloed policy responses to individual AI applications (that is, they are technologycentric), or for AI applications' effects on individual legal fields or doctrines (that is, they are law-centric). In contrast, this chapter argues that to craft adequate AI policies, a better regulatory perspective takes a step back, and first asks a better question: when we craft AI regulation, what are we seeking to regulate? And what are the ingredients for a more systematic regulatory template for crafting AI governance? This chapter argues that to foster an effective AI regulatory ecosystem, policy institutions and actors must be equipped to craft AI policies in alignment with systematic assessments of how, when, and why AI applications enable broader forms of sociotechnical change (Maas 2020) . It argues that this approach complements existing technologycentric and law-centric examinations of AI policies; and that, supported by adequate institutional processes, it can provide an informed and actionable perspectives on when and why AI applications actually create a rationale for regulation-and how they are consequently best approached as targets for regulatory intervention. This enables more tailored policy formulation for AI issues, facilitates oversight and review of these policies, and helps address structural accountability, alignment, and (lack of) information problems in the emerging AI governance regulatory ecosystem. The chapter is structured as follows. It first (1) sketches the general value of a 'change-centric' approach to AI governance. The chapter then (2) proposes and articulates a framework focused on 'sociotechnical change', and explores how this model allows an improved consideration of (3) when an AI application creates a regulatory rationale, and (4) how it is subsequently best approached as a regulatory target, considering 6 distinct 'problem logics' that appear in AI issues across domains. Finally (5), the chapter reflects on some of the limits of this approach, before discussing concrete institutional and regulatory actions that can draw on this approach in order to improve the regulatory triage, tailoring, timing & responsiveness, and regulatory design of AI policy.
 Towards change-centric approaches in an AI regulatory ecosystem In response to AI's emerging challenges, scholars and policymakers have appealed to a wide spectrum of regulatory tools to govern AI. It should be no surprise that recent years have seen increasing public demands for AI regulation (B. Zhang and Dafoe 2020) , and diverse new national regulatory initiatives (Cussins 2020; Law Library of Congress 2019) . Much work to date has focused on the regulation of AI within particular domestic regulatory contexts. For instance, this has explored the relative institutional competencies of legislatures, regulatory agencies, or courts at regulating AI (Guihot, Matthew, and Suzor 2017; Scherer 2016) . Others have emphasised the governance roles of various actors in the AI landscape (Leung 2019) , exploring for instance how tech companies' ethics advisory committees (Newman 2020, 12-29) , AI company employee activists and 'epistemic communities' (Belfield 2020; Maas 2019a ), AI research community instruments (such as scientific conference research impact assessment mechanisms) (Prunkl et al. 2021) , or private regulatory markets architectures (Clark and Hadfield 2019) , could all help shape AI regulation. There is also a growing recognition of the importance of global coordination or cooperation for AI governance (Feijóo et al. 2020; Kemp et al. 2019; Turner 2018) . At the global level, much focus to date has been on the burgeoning constellation of AI ethics principles that has sprung up in the last half-decade (Fjeld et al. 2019; Jobin, Ienca, and Vayena 2019; Schiff et al. 2020; Stahl et al. 2021; Stix 2021) . However, this is being increasingly complemented by regulatory proposals. These have ranged from relying on existing norms, treaty regimes, or institutions in public international law (Kunz and Ó hÉigeartaigh 2021; Burri 2017; Smith 2020) . Others have proposed entirely new international organizations in order to coordinate national regulatory approaches (Erdelyi and Goldsmith 2018; Kemp et al. 2019; Turner 2018, chap. 6) , or have compared such centralized institutions for AI to more decentralized or fragmented alternatives (Cihon, Maas, and Kemp 2020b, 2020a) . Others have focused more on the role of soft law instruments Gutierrez, Marchant, and Tournas 2020) , international standard-setting bodies (Cihon 2019; Lorenz 2020) , or certification schemes (Cihon et al. 2021) . Others have proposed the adaptation of existing informal governance institutions such as the G20 (Jelinek, Wallach, and Kerimi 2020) . This is clearly a diverse constellation of efforts. Yet there are underlying classes and patterns in these approaches to AI regulations. For instance, one group of 'technology-centric' approaches focuses on 'AI' as an overarching class that should be considered in whole (Turner 2018) . While this is more reflective of the cross-domain application and impact of many AI techniques, however, this approach is not without problems. For one, it is undercut by intractable debates over how to define 'AI' (Russell and Norvig 2016)-and by the fact that AI is not a single thing (Schuett 2019; Stone et al. 2016) . This shortfall is more addressed in the second 'technology-centric' approach, which is 'application-centric'-or, as some call it, 'use-case centric' (Schuett 2019, 5) . This approach seeks to unpack the umbrella term 'AI', and split out the specific AI applications that regulation should focus on. In the past decade, many policy responses to AI have been sparked by one or another use case of AI technology-involving concrete issues that have been thrown up, or visceral problems that are anticipated-such as autonomous cars, drones, facial recognition, or social robots (Turner 2018, 218-19) . As Petit puts it, this technology-centric approach to AI involves charting "legal issues from the bottom-up standpoint of each class of technological application" (Petit 2017) . Application-centric approaches remain the default response to AI governance. However, like the AI-centric approach, this orientation also has shortfalls. For one, it emphasizes visceral edge cases, and is therefore easily lured into regulating edge-case challenges or misuses of the technology (e.g. the use of DeepFakes for political propaganda) at a cost of addressing far more common but less visceral use cases (e.g. the use of DeepFakes for gendered harassment) (Liu et al. 2020) . Moreover, the resulting AI policies and laws are frequently formulated in a piecemeal and ad-hoc fashion, which means that this perspective can promote siloed regulatory responses (Turner 2018, 218-19) . A focus on individual applications also may inadvertently foregrounds technologyspecific regulations even where these are not the most effective (Bennett Moses 2013). It moreover induces a 'problem-solving' orientation (Liu and Maas 2021 ), aimed at narrowly addressing local problems caused (or envisioned) by the specific use case that prompted the regulatory process (Crootof and Ard 2021) . A distinct set of regulatory responses to AI are instead law-centric. They invoke what Nicolas Petit has called a 'legalistic' approach, which 'consists in starting from the legal system, and proceed by drawing lists of legal fields or issues affected by AIs and robots." (Petit 2017, 2) . This approach segments regulatory responses by departing from AI's impacts on-and within specific conventional legal codes or subjects (e.g. privacy law, contract law, the law of armed conflict), or by exploring the ways in which these create questions about the scope, intersection, assumptions, or adequacies of existing law (Crootof and Ard 2021) . To be clear, technology-, application-, and law-centric approaches to AI regulation have important insights, and must play a role in any AI governance ecosystem. Nonetheless, they have their drawbacks. Importantly, AI governance proposals could be grounded in a better understanding of how AI applications translate into cross-domain changes, and how future capabilities or developments might further shift this problem portfolio (Maas 2019b ). An alternative is therefore to shift (or complement) these approaches with a framework that is not (solely) anchored in 'new technology' (whether on the umbrella term of 'AI', or on isolated AI applications), nor on isolated legal domains, but which rather examines types of change. What impacts are we concerned about? AI regulatory ecosystem require a protocol for considering where, and why AI change warrants regulatory intervention, and how and when this regulatory intervention should take place. It should be able to adequately identify when AI applications create regulatory rationales, as well as the best levers to approach AI as a regulatory target. Can we reformulate an impact-focused approach for AI regulation, that provides superior levers for regulation? To achieve this, this chapter instead draws on existing theories from the emerging paradigms of law, regulation and technology, and 'TechLaw'-"the study of how law and technology foster, restrict, and otherwise shape each other's evolution." (Crootof and Ard 2021, n. 1; Ard and Crootof 2020) . In particular, it proposes to approach AI governance through the lens of 'sociotechnical change' (Bennett Moses 2007a . As such, this chapter will now turn to how this approach can bridge the gap between technology/application-centric and law-centric approaches, by guiding reflection on when and why new AI applications require new regulation-and how the resulting regulatory interventions are best tailored.
 Reframing regulation: AI and sociotechnical change It should be no surprise that changes in technology have given rise to extensive scholarship on the relation between law and these new technologies. In some cases, such work has focused in on identifying the assumed 'exceptional' nature or features of a given technology (Calo 2015) . However, as noted, other scholars have influentially argued that it is less the 'newness' of a technology that brings about regulatory problems, but rather the ways it enables particular changes in societal practices, behaviour, or relations (Balkin 2015; Bennett Moses 2007b; Friedman 2001) . That is, in what ways do changes in a given technologies translate to new ways of carrying out old conduct, or create entirely new forms of conduct, entities, or new ways of being or connecting to others? When does this create problems for societies or their legal systems? Rather than focus on 'regulating technology' (either in general, or in specific applications), this scholarship accordingly puts a much greater emphasis on 'adjusting law and regulation for sociotechnical change' (Bennett Moses 2017, 574) . In particular, Brownsword, Scotford, and Yeung have highlighted three dimensions of technological 'disruption': "legal disruption, regulatory disruption, and the challenge of constructing regulatory environments that are fit for purpose in light of technological disruption" (Brownsword, Scotford, and Yeung 2017, 7) . As noted, various scholars of law, regulation and technology have emphasized the importance of 'sociotechnical change'. In her 'theory of law and technological change' (Bennett Moses 2007b), Lyria Bennett Moses has argued that questions of 'law and technology' are rarely if ever directly about technological progress itself (whether incremental or far-reaching). Instead, she argues that lawyers and legal scholars who examine the regulation of technology are focused on questions of "how the law ought to relate to activities, entities, and relationships made possible by a new technology" (Bennett Moses 2007b, 591) . Indeed, Lyria Bennett-Moses has argued that from a regulatory perspective, "[t]echnology is rarely the only 'thing' that is regulated and the presence of technology or even new technology alone does not justify a call for new regulation"(Bennett Moses 2017, 575) . In doing so, she calls for a shift in approach from 'regulating technology' to 'adjusting law and regulation for sociotechnical change.' (Bennett Moses 2017, 574) This shifts the focus on patterns of 'socio-technical change' (Bennett Moses 2007b Moses , 591-92, 2007a )instances where changes in certain technologies actually expand human capabilities in ways that give rise to new activities or forms of conduct, or new ways of being or of connecting to others (Bennett Moses 2007b, 591-92) . As such, the question of governing new technologies is articulated not with reference to a list of (sufficiently 'new') technologies (Bennett Moses 2017, 576), but is relatively 'technology-neutral'. It is this functional understanding of 'socio-technological change' that informs more fruitful analysis of when and why we require regulation for new technological or scientific progress. It can also underlie a more systematic examination of which developments in AI technology are relevant for a regulatory system to focus on.
 AI as regulatory rationale Electronic copy available at: https://ssrn.com/abstract=3871635 What types of sociotechnical changes (e.g. new possible behaviours or states of being) actually give rise to regulatory rationales? When a technology might create an opportunity for certain problematic behaviour, does that opportunity need to be acted upon, or can the mere possibility of that behaviour constitute a regulatory rationale? Can sociotechnical changes be anticipated? This entails a more granular understanding of the dynamics of sociotechnical change-and when or how such changes can constitute a rationale for regulation.
 Varieties of sociotechnical change When and why do AI capabilities rise to a problem that warrants legal or regulatory solutions? It is important to recognize that not all new scientific breakthroughs, new technological capabilities, or even new use cases will necessarily produce the sort of 'sociotechnical change' that requires regulatory responses. In practical terms, this relates to the observations that new social (and therefore governance) opportunities or challenges are not created by the mere fact of a technology being conceived, or even prototyped, but rather by them being translated into new 'affordances' for some actors-relationships "between the properties of an object and the capabilities of the agent that determine just how the object could possibly be used" (Norman 2013, 11) . AI affordances can be new types of behaviour, entities, or relationships that were not previously possible (or easy), and which are now available to various actors (Liu et al. 2020) . How does technological change translate into sociotechnical change? When would this be disruptive to law? There are various types of sociotechnical changes that new AI applications can create or enable (Maas 2019c, 33 ; see also Crootof and Ard 2021) . (1) Allowing older types of behaviours to be carried out with new items or entities, including artefacts which are potentially not captured under existing (technology-specific) regulatory codes, or which blur the boundaries between existing domains or regimes, potentially causing problematic gaps, overlaps, or contradictions in how these behaviours are covered by regulation; (2) Absolute or categorical capability changes, where AI progress expands the action space and 'unlocks' new capabilities or behaviour which were previously simply out of reach for anyone, and which could be of regulatory concern for one of various reasons; (3) Relative capability changes, where AI increases the prominence of a previously rare behaviour, for instance because progress lowers thresholds or use preconditions for a certain capability (e.g. advanced video editing; online disinformation campaigns; cryptographic tools), which was previously reserved to a narrow set of actors; or because progress allows the scaling up of certain existing behaviours (e.g. phishing emails). (4) Positional changes amongst actors, where AI applications that drive shifts in which particular state actors are dominant, while leaving the general 'rules' of that international system more or less unaltered. (5) Changing structural dynamics in a (international) society, for instance, by; i. Shifting prevalent influence between types of actors (e.g. away from states and towards non-state actors or private companies); ii. Shifting the means by which certain actors seek to exercise 'influence' (e.g. from 'hard' military force to computational propaganda, or from multilateralism to 'lawfare', as a result of new communications technologies increasing the scope, velocity and effectiveness of such 'lawfare' efforts) (Dunlap 2008, 146-48) ; iii. Altering the norms or identities of actors, and thereby changing the terms by which they conceive of their goals and orient their behaviour. As mentioned, there may be certain AI innovations or breakthroughs that do not create very large sociotechnical changes of these forms, even if from a pure scientific or engineering standpoint they involve considerable alterations to the state of the art. Conversely, technological change or improvements also need not be qualitatively novel, dramatic, sudden, or cutting-edge for them to drive intense and meaningful change in balances of power, or in societal structures (Cummings et al. 2018, iv) . The question is therefore not only how large these sociotechnical changes are, but how, or whether, they touch on the general rationales for new regulatory interventions.
 Mapping regulatory rationales There are various accounts for when and why regulatory intervention is warranted by the introduction of new technologies. For instance, Deryck Beyleveld and Roger Brownsword argue that emerging technologies generally give rise to two kinds of concerns: "one is that the application of a particular technology might present risks to human health and safety, or to the environment […] and the other is that the technology might be applied in ways that are harmful to moral interests" (Beyleveld and Brownsword 2012, 35) . However, while these may be the most prominent rationales, the full scope of reasons for regulation may extend further. In a non-technology context, Tony Prosser has argued that regulation, in general, has four grounds: "(1) regulation for economic efficiency and market choice, (2) regulation to protect rights, (3) regulation for social solidarity, and (4) regulation as deliberation" (Prosser 2010, 18) . How do these regulatory rationales relate to technological change? As Bennett Moses (2017, 578) notes, all four of these rationales can certainly become engaged by new technologies. That is, new technologies (or new applications) can: (1) create sites for new market failures, warranting regulatory interventions such as technical standards or certification, to ensure economic efficiency and market choice, and remedy information inadequacies for consumers; (2) generate many new risks or harms-either to human health or the environment, or to moral interests-which create a need for regulation to protect the rights of these parties (e.g. restrictions of new weapons; the ban on human cloning). (3) create concern about social solidarity, as seen in concerns over the 'digital divide' at both a national and international level, creating a need for regulation to ensure adequate inclusion. (4) create sites or pressures for the exertion of proper democratic deliberation over the design or development pathways of technologies. (Bennett Moses 2017, 579-83) To be sure, as a technology-centric approach would note, these cases all involve new technologies which require regulation. However, Bennett Moses argues that in each of these cases, it is not the involvement of 'new technology' per se that provides a special rationale for regulation, above and beyond the resulting social changes (e.g. potential market failures; risks to rights; threats to solidarity; or democratic deficits) that are at stake (Bennett Moses 2017, 583). We are not worried about technology; we are worried about its effects. As such, the primary regulatory concern is over the emergence of the 'sociotechnical' effects that occur. This conceptual shift can help address one limit that regulatory or governance strategies encounter if they focus too much or too narrowly on technology. As she argues: "… treating technology as the object of regulation can lead to undesirable technology specificity in the formulation of rules or regulatory regimes. If regulators ask how to regulate a specific technology, the result will be a regulatory regime targeting that particular technology. This can be inefficient because of the focus on a subset of a broader problem and the tendency towards obsolescence" (Bennett Moses 2017, 584). As such, taking a sociotechnical (rather than a technology-centric) approach, this lens helps keep into explicit focus specific rationales for governance in each use case of AI: on what grounds and when regulation is needed and justified? These four accounts of rationales are valuable as a starting point for AI regulation. However, we can refine this account. For one, it is analytically valuable to draw a more granular distinction between (physical) harms to human health or the environment, and (moral) harms to moral interests (Beyleveld and Brownsword 2012) . Moreover, these categories all concern rationales for governance to step in, in response to sociotechnical changes that are affecting society (i.e. the regulatees) directly. However, there may also be cases where AI-enabled sociotechnical change creates an indirect regulatory rationale, because it presents some risk directly to the existing legal order charged with mitigating the prior risks. In such cases of 'legal disruption' (Liu et al. 2020; Maas 2019c) , sociotechnical change can produce a threat to the regulatory ecosystem itself. This can be because these tools allow regulatees to more effectively challenge or bypass existing laws, resulting in potential 'legal destruction' (Maas 2019c ). Alternatively, it can result because certain AI tools can drive 'legal displacement' (Maas 2019c) , by offering substitutes or complements to existing legal instruments, in shaping or managing the behaviour of citizens (Brownsword 2019) . Drawing together the above accounts, one might then speak of a regulatory rationale for an AI system or application, whenever it drives sociotechnical changes (new ways of carrying out old behaviour, or new behaviours, relations or entities) which result in one or more of the following situations: (1) new possible market failures; (2) new risks to human health or safety, or the environment; (3) new risks to moral interests, rights, or values; (4) new threats to social solidarity; (5) new threats to democratic process; (6) new threats directly to the coherence, efficacy or integrity of the existing regulatory ecosystem charged with mitigating the prior risks (1-5). All this is not to say that these rationales apply in the same way in all specific contexts. Indeed, they will be weighted differently across distinct legal systems and jurisdictions-and between domestic and international law. Nonetheless, they provide a rubric for understanding when or why we (should) want to regulate a new AI applicationand a reminder that it is the sociotechnical changes, not the appearance of new technology in itself, that we are concerned about.
 AI as regulatory target Along with providing a greater grounding for understanding whether, when and why to regulate new AI applications, a consideration of sociotechnical change can also shed light on the regulatory 'texture' of the underlying AI capabilities-that is, its constitution as a 'regulatory target' (Buiten 2019, 46-48) . That is, once regulators have confirmed a regulatory rationale (i.e. they have asked 'do we need regulation? For what sociotechnical change? What regulatory rationale?'), they then face the question of how to craft regulatory actions. In considering AI applications as a target for regulation, a sociotechnical change-centric perspective must on the one hand take stock of the material aspects of a technology (as an artefact). Material features certainly matter from the perspective of understanding key parameters for regulation, such as: (1) Its trajectory and distribution: i.e. the state of leading AI capabilities (across its different sub-fields), possible and plausible rates and directions of progress given material constraints on the design space and process (Verbruggen 2020) affect stakeholder perceptions of the imminence of various applications of the technology, and of the need for urgent regulation (Crootof 2019; Watts 2015) (4) Potential sites or vectors for regulatory leverage: for instance, the degree to which proliferation of certain systems could be meaningfully halted through export control policies (Brundage et al. 2020; Fischer et al. 2021) . In these ways, such material features certainly matter, especially when considering AI regulation at the global level. For instance, scholars have argued that the modern global digital economy, far from consisting solely of ethereal digital products ungraspable by law, is instead populated by distinct 'regulatory objects', which vary in their degree of apparent 'materiality' (from high-capital submarine cables and satellite launch facilities, to ethereal cloud services), and their degree of centralization (from diverse suppliers of various 'smart' appliances, to dominant social networks or computationally intensive search engine algorithms) (Beaumier et al. 2020) . Critically, some of these may not easily be subjected to global regulation, but many which can certainly be captured by various regulatory approaches. However, for regulatory purposes, a material analysis is not sufficient. A sociotechnical change-centric perspective on AI regulation rather can and should go beyond the technology itself, and consider a broader set of 'problem logics' in play. For instance, we can fruitfully distinguish between: 'ethical challenges', 'security threats', 'safety risks', 'structural shifts', 'common benefits', and 'governance disruption'. Distinguishing amongst these ideal-types is valuable, as these clusters can introduce distinct problem logics, and foreground distinct regulatory logics or levers (see Table 1 ). 1 It is important to note that this taxonomy is not meant to be mutually exclusive, nor exhaustive. It aims to capture certain regularities which help ask productive regulatory questions. For each category, we can ask-how does the AI capability produce sociotechnical change? Why does this create a governance rationale? How should this be approached as governance target? What are the barriers, and what regulatory tools are foregrounded? There is insufficient space to go into exhaustive detail on each of these categories within the taxonomy. However, at glance, we can pick out a number of ways in which clustering AI's sociotechnical impacts along these various problem logics, can facilitate structural regulation-relevant insights for AI regulators. These include consideration of aspects. For one, this model enables examination of the underlying origins of the sociotechnical challenge of concern, in terms of: (1) the key actors (e.g. principals, operators, malicious users) whose newly AI-enabled or -related behaviour or decisions create the governance concerns, and (2) those actors' traits, interests, or motives which drive the AI-sociotechnical-problem related behaviour or decisions (e.g. actor apathy, malice, negligence, or the way the new capability sculpts choice architectures in ways shift structural incentives or strategic pressures).
 Safety risks Can we rely onand control this? • New risks to human health or safety
 Structural shifts How does this shape our decisions? • (all, indirectly)
 Governance Disruption How does this change how we regulate? • New risks directly to existing regulatory order This in turn can be linked to the contributing factors which sustain or exacerbate this sociotechnical impact, such as: (1) The range and diversity of AI-related failure modes and issue groups (Hernandez-Orallo et al. 2020, 7) , including emergent interactions with other actors (human or algorithmic) in their environment (Rahwan et al. 2019) , peculiar behavioural failure modes (Amodei et al. 2016; Krakovna et al. 2020; Kumar et al. 2019; Leike et al. 2017) . Moreover, this model enables a study of the barriers to regulation; that is, the factors that drive the difficulty of formulating or implementing policy solutions, and which will themselves have to be overcome to achieve effective regulatory responses for AI, because of: (5) the live societal or cross-cultural value pluralism (Gabriel 2020) or disagreement over the values, interests or rights affected, and how these should be weighted in the context of a specific contested AI application (ethical challenges); (6) the disproportionately high costs of 'patching' vulnerabilities of human social systems (e.g. our faith in the fidelity of human voices) against AI-enabled social engineering attacks, relative to past costs of patching 'conventional' cybersecurity vulnerabilities by the dissemination of software fixes (Shevlane and Dafoe 2020b, 177) . (security threats) (7) the difficulty of foreseeing indirect effects of AI on the structure of different actors' choice architectures (van der Loeff et al. 2019; Zwetsloot and Dafoe 2019)-and the difficulty of resolving those situations through any one actor's unilateral action (Dafoe 2020) , or to coordinate behaviour in response (structural shifts) Finally, on the basis of the above, it allows a consideration of the types of regulatory approaches and levers that are highlighted and foregrounded for each of these challenges. It highlights the role of 'mend-it-or-end-it' debates around algorithmic accountability (Pasquale 2019), auditing frameworks (Raji et al. 2020) , and underlying cross-cultural cooperation (ÓhÉigeartaigh et al. 2020) to diverse ethical challenges. Of perpetratorfocused and target-focused (e.g. 'security mindset' (Severance 2016)) interventions to shield against AI security threats. The development of support programs to guarantee public goods such as the use of AI in 'AI for Good' interventions (Floridi et al. 2018; ITU 2019) , humanitarian uses (Roff 2018, 25 ; but see Sapignoli 2021), or redistributive Electronic copy available at: https://ssrn.com/abstract=3871635 guarantees such as a 'Windfall clause' that sees tech companies pledge extreme future profits above a certain threshold towards redistribution (O' Keefe et al. 2020) . That is not to say that this framework provides conclusive recipes or roadmaps for regulation. Rather, it provides a beginning structuring framework for thinking through common challenges across diverse regulatory domains charged with resolving questions around seemingly separate applications of AI (Crootof and Ard 2021) . Such an approach can at least avoid duplication of effort, and at best can support the formulation and spread of better, more resilient policies. In sum, a sociotechnical-change-centric approach is not without its pitfalls or limits. Still, it can have various benefits in organizing and orienting an AI regulatory ecosystem. It prompts regulators to ask themselves: (1) when, why and how a given AI application produces particular types of sociotechnical changes; (2) When and why these changes rise to create a rationale for governance; (3) How to approach the target of regulation. As such, this can be an important regulatory complement to the insights provided by-and the interventions grounded in-technology-centric or law-centric perspectives.
 Implementation: AI regulation through a sociotechnical lens This lens of sociotechnical change does not provide single substantive answers for how to resolve each and every AI policy problem. However, it can help answer common recurring questions in AI policy around institutional choice and regulatory timing and design (Bennett Moses 2017, 585-91) . In particular, regulatory actors can improve governance for AI challenges in terms of regulatory triage, tailoring, timing and responsiveness, and design.
 Regulatory triage In the first place, the sociotechnical-change-centric perspective on AI can help in carrying out regulatory triage. This is not just of value to AI regulation: indeed, it fits in with a broader initiative in recent legal scholarship towards exploring questions of 'legal prioritization' (Winter et al. 2021) . However, within the AI regulation ecosystem, this lens helps focus attention on the most societally disruptive impacts of the technology, and as such helps re-focus scarce regulatory attention. This reduces the risk that regulatory attention is over-allocated on visceral applications of AI which may not ultimately prove scalable, or on 'legally interesting puzzles', at a cost of more opaque but prevalent indirect impacts. Better triage can be a valuable corrective to approaches that select, organize or prioritize AI policy issues based on high-profile but non-representative incidents, popularcultural resonance, or 'fit' to pre-existing legal domains. Moreover, if regulatory bodies focus less on the 'newness' of AI technology, or on the steady stream of each new AI application, but rather on which downstream sociotechnical impacts in fact create particular regulatory rationales, they can step back from a reactive firefighting mode, and help defuse or dissolve the so-called 'pacing problem' (Marchant 2011) . Regulatory triage is also aided by the ways in which this framework can expand regulatory actors' scope of analysis of which sociotechnical impacts are relevant for regulatory consideration. While technology-centric approaches can highlight the direct challenges of AI (in the areas of ethics, security, and safety), the sociotechnical-change-centric perspective also allows regulators to consider interventions for various indirect sociotechnical changes, including the ways AI systems can shift incentive structures, how to realize beneficial opportunities and public goods around AI technology, or how AI applications can disrupt the regulatory tools or systems which these regulators would rely on. What does that entail in practice? Improving triage around AI regulation could involve (1) improving information infrastructures or 'technical observatories' (Clark [forthcoming] (in this volume)), to not only equip regulators with relevant and up-to-date technical information around AI techniques and applications, but also think through how these relate to downstream sociotechnical impacts. This may help ensure regulators are less easily dazzled by the 'newness' of new AI applications themselves (Mandel 2017) , and enable them to become more aware of how different analogies can highlight different regulatory narratives in potentially counterproductive ways (Crootof and Ard 2021) . This can also involve (2) setting up a cross-ecosystem agency to focus on 'legal foresighting' (Laurie, Harmon, and Arzuaga 2012) , and forecasting methodologies (Avin 2019; Ballard and Calo 2019) aimed at eliciting AI's technologies' disparate sociotechnical impacts, link these to potential and actual regulatory rationales, and study the shifting material textures and problem logics around that application. In particular, this can support more democratic and inclusive stakeholder debate over the choices affected parties would seek to make around the deployment of potentially disruptive AI breakthrough capabilities (Cremer and Whittlestone 2021).
 Regulatory tailoring and scope Secondly, and relatedly, the sociotechnical change-centric lens helps in tailoring regulatory solutions to effective clusters of AI techniques, applications, -users, and societal effects. Rather than consign regulators to confront self-similar AI challenges (e.g. around meaningful human control; susceptibility to adversarial attack; unaccountable opacity in algorithmic decision-making) many times across individual legal domains, (Crootof and Ard 2021, 1) , this approach highlights common themes, underlying material value chains, or usage problem logics of AI, as they are expressed in these various domains. Practically, improved regulatory tailoring can require (3) the establishment of various institutions and meta-regulatory oversight mechanisms-connoting "activities occurring in a wider regulatory space, under the auspices of a variety of institutions, including the state, the private sector and public interest groups [which] may operate in concert or independently" (Grabosky 2017, 150) . In so doing, such mechanisms could foster improved cross-regime dialogue of AI policy (Cihon, Maas, and Kemp 2020b) . This can support regulatory harmonization or the bundling of regulatory interventions for various AI applications where appropriate. It can also examine how and where different regimes and institutions can exploit the same regulatory levers (e.g. compute hardware production) that intersect on the AI development value chain. (4) Establishing mechanisms and fora for dialogue amongst various actors in the AI space that may have a hand in shaping the overarching 'problem logic'-in terms of the problem's origins, contributing factors, or barriers to regulation. The aim of such discussions would ideally be to reconfigure some of these wider conditions to be more supportive or conducive to AI regulation, or-where they are not very tractable, to explore alternative levers or vectors for regulation. Finally, it can promote the exchange of best practices or lessons learned around how regulators can address some of the problem logics that generate the problem or impede regulation.
 Regulatory timing Thirdly, in terms of regulatory timing and responsiveness, a study of AI's sociotechnical changes highlights the inadequacies of governance strategies that are grounded either in an attempt to predict sociotechnical changes in detail, or reactive responses which prefer to 'wait out' technological change until its societal impact has become clear-which demands a threshold of clarity that is in fact rarely achieved, even decades after a technology's deployment (Horowitz 2020) . Rather, it emphasises the importance of anticipatory and adaptive regulatory approaches (Maas 2019b ). This helps mitigate some of the information problems facing AI regulation, by helping ensure AI regulation can remain adaptive and 'scalable' to ongoing sociotechnical change, given the profound lack of information about future pathways. This could be pursued by ( 5 ) incorporating provisions such as sunset clauses that prompt re-examination (at the domestic level) or designating authoritative interpreters (at the international level).
 Regulatory design Fourthly, in terms of regulatory design, the sociotechnical change lens highlights when and why governance should prefer technology-neutral rules versus technology-specific rules. By considering the specific regulatory or governance rationale in play, we may understand when or whether technological neutrality is to be preferred. Generally, Bennett- Moses (2017, 586) argues that "regulatory regimes should be technology-neutral to the extent that the regulatory rationale is similarly neutral". In this view, the point is not, to find a regulatory strategy that already details long lists of anticipated future applications of AI. The idea is rather to develop institutional mechanisms that are up to the task of managing distinct problem logics-new ethical challenges, security threats, safety risks, structural shifts, opportunities for benefit, or governance disruptions-in a way that can be relatively transferable across-or agnostic to the specific AI techniques used to achieve those affects. Establishing clearer guidelines about formulation of AI-specific regulations, and the circumstances in which these should rely on standards or rules, and when they should be tech-specific or tech-neutral (Crootof and Ard 2021) .
 Conclusion Electronic copy available at: https://ssrn.com/abstract=3871635 This chapter has introduced, articulated, and evaluated a 'sociotechnical change-centric' perspective on aligning AI regulation. It first briefly sketched the general value of a 'change-centric' approach to the problems facing the AI governance ecosystem. The chapter next articulated a framework focused on Lyria Bennett Moses's account of regulation for sociotechnical change. It explored how, when, and why law and regulation for AI ought to tailor themselves to broad sociotechnical change rather than local technological change. It applied this model to AI technology, in order to show how this model allows a better connection of AI applications to five types of sociotechnical change-and how these in turn can be mapped to six types of regulatory rationales. It then turned to the mirror question of how, having established a need for governance, regulators might craft policy interventions to the particular regulatory target of AI. This involved a consideration of both the material textures of AI applications, but especially demands focus on the 'problem logics' involved. It argued that socio-technical changes created by AI applications can be disambiguated into six specific types of challenges-ethical challenges, security threats, safety risks, structural shifts, public goods, and governance disruption-which come with distinct problem features (origins, contributory factors, barriers to regulation), and which may each be susceptible to (or demand) different governance responses. Finally, the chapter concluded by reflecting on the limits and uses of this approach, before sketching some indicative institutional and regulatory actions that might draw on this framework to improve regulatory triage, tailoring, timing & responsiveness, and regulatory design of AI policy. To be clear, an emphasis on sociotechnical change is not a new insight in scholarship on law, regulation and new technology. However, in a fragmented and incipient AI governance landscape, it remains a valuable tool. In sum, 'sociotechnical change' should be considered not a new or substitute paradigm for AI governance, but rather a complementary perspective. Such a lens is subject to its own conditions and limits, but when used cautiously, can offer regulators a more considered understanding of which of AI's challenges are possible, plausible, or already-pervasive-and how these might be best met. Notes 1 An earlier version of this framework is presented and unpacked in further detail in (Maas 2020, 166-86) . Note, this version referred to the 'public goods' as the 'common goods' problem logic, instead. (safety risks) (2) Human overtrust and automation bias, rendering some AI systems susceptible to emergent and cascading 'normal accidents' (Maas 2018) (safety risks) (3) The underlying 'offense-defense balance' of AI scientific research (Shevlane and Dafoe 2020a), and how it evolves along with more sophisticated AI capabilities (Garfinkel and Dafoe 2019) (security threats). (4) The susceptibility of existing legal and regulatory systems themselves to 'disruption' by AI uses, at the level of doctrinal substance, law-making and enforcement processes, or the political foundations (Liu et al. 2020; Maas 2019c) (governance disruption).
 Table 1 . 1 Taxonomy of AI problem logics • AI systems • O. Push towards legal • Provisions to render creating efficiency governance 'innovation- substantive • CF. Legal system proof': technological ambiguity in law exposure and neutrality; authoritative • Legal automation dependence on interpreters, sunset altering processes conceptual orders or clauses; etc. … of law operational assumptions • Oversight for legal • Erodes political automation; distribution foundations Electronic copy available at: https://ssrn.com/abstract=3871635
 Electronic copy available at: https://ssrn.com/abstract=3871635


Modeling and interpreting expert disagreement about artificial superintelligence

Introduction Artificial superintelligence (ASI) is artificial intelligence (AI) with capabilities that are significantly greater than human capabilities across a wide range of domains. If developed, ASI could have impacts that are highly beneficial or catastrophically harmful, depending on its design A hallmark of the ASI issue is disagreement among experts. Experts disagree on if ASI will be built, when it would be built, what designs it would use, and what its likely impacts would be. 1 The extent of expert disagreement speaks to the opacity of the underlying ASI issue and the general difficulty of forecasting future technologies. This stands in contrast with other major global issues, such as climate change, for which there is extensive expert agreement on the basic parameters of the issue (Oreskes 2004 ). Expert consensus does not guarantee that the issue will be addressed-the ongoing struggle to address climate change attests to this-but it does offer direction for decision making. In the absence of expert agreement, those seeking to gain an understanding of the issue must decide what to believe given the existence of the disagreement. In some cases, it may be possible to look at the nature of the disagreement and pick sides; this occurs if other sides clearly have flawed arguments that are not worth giving any credence to. However, in many cases, multiple sides of a disagreement make plausible arguments; in these cases, the thoughtful observer may wish to form a belief that in some way considers the divergent expert opinions. This paper demonstrates and discusses methodological options for modeling and interpreting expert disagreement about the risk of ASI catastrophe. The paper accomplishes this by using a new ASI risk model called ASI-PATH (Barrett and Baum 2017a; 2017b) . Expert disagreement can be modeled as differing estimates of parameters in the risk model. Given a set of differing expert parameter estimates, aggregate risk estimates can be made using weighting functions. Modeling expert disagreement within the context of a risk model is a method that has been used widely across a range of other contexts; to our knowledge this paper marks the first application of this method to ASI. The paper uses a well-documented recent disagreement between Nick Bostrom and Ben Goertzel as an illustrative example-an example that is also worthy of study in its own right. Bostrom and Goertzel are both longstanding thought leaders about ASI, with lengthy research track records and a shared concern with the societal impacts of ASI. However, in recent publications, Goertzel (2015; 2016 ) expresses significant disagreement with core arguments made by Bostrom (2014) . The Bostrom-Goertzel disagreement is notable because both of them are experts whose arguments about ASI can be expected to merit significant credence from the perspective of an outside observer. Therefore, their disagreement offers a simple but important case study for demonstrating the methodology of modeling and interpreting expert disagreement about ASI. The paper begins by summarizing the terms of the Bostrom-Goertzel disagreement. The paper then introduces the ASI-PATH model and shows how the Bostrom-Goertzel disagreement can be expressed in terms of ASI-PATH model parameters. The paper then presents model parameter estimates based on the Bostrom-Goertzel disagreement. The parameter estimates are not rigorously justified and instead are intended mainly for illustration and discussion purposes. Finally, the paper applies the risk modeling to a practical problem, that of AI confinement.
 The Bostrom-Goertzel disagreement Goertzel (2015; 2016) presents several disagreements with Bostrom (2014) . This section focuses on three disagreements of direct relevance to ASI risk.
 Human evaluation of AI values One disagreement is on the potential for humans to evaluate the values that an AI has. Humans would want to diagnose an AI's values to ensure that they are something that humans consider desirable (henceforth "human-desirable"). If humans find an AI to have human-undesirable values, they can reprogram the AI or shut it down. As an AI gains in intelligence and power, it will become more capable of realizing its values, thus making it more important that its values are humandesirable. A core point of disagreement concerns the prospects for evaluating the values of AI that have significant but still subhuman intelligence levels. Bostrom indicates relatively low prospects for success at this evaluation, whereas Goertzel indicates relatively high prospects for success. Bostrom (2014, p.116-119) posits that once an AI reaches a certain point of intelligence, it might adopt an adversarial approach. Bostrom dubs this point the "treacherous turn": The treacherous turn: While weak, an AI behaves cooperatively (increasingly so, as it gets smarter). When the AI gets sufficiently strongwithout warning or provocation-it strikes, forms a singleton [i.e., takes over the world], and begins directly to optimize the world according to the criteria implied by its final values. ( Goertzel provides a contrasting view, focusing on Step 2. He posits that an AI of intermediate intelligence is unlikely to successfully pretend to have humandesirable values because this would be too difficult for such an AI. Noting that "maintaining a web of lies rapidly gets very complicated" (Goertzel 2016, p.55), Goertzel posits that humans, being smarter and in control, would be able to see through a sub-human-level AI's "web of lies". Key to Goertzel's reasoning is the claim that an AI is likely to exhibit human-undesirable behavior before it (A) learns that such behavior is human-undesirable and (B) learns how to fake humandesirable behavior. Thus, Step 2 is unlikely to occurinstead, it is more likely that an AI would either have actual human-desirable values or be recognized by humans as faulty and then be reprogrammed or shut down. Goertzel does not name his view, so we will call it the sordid stumble: The sordid stumble: An AI that lacks humandesirable values will behave in a way that reveals its human-undesirable values to humans before it gains the capability to deceive humans into believing that it has human-desirable values. It should be noted that the distinction between the treacherous turn and the sordid stumble is about the AI itself, which is only one part of the human evaluation of the AI's values. The other part is the human effort at evaluation. An AI that is unskilled at deceiving humans could still succeed if humans are not trying hard to notice the deception, while a skilled AI could fail if humans are trying hard. Thus, this particular Bostrom-Goertzel debate covers only one part of the AI risk. However, it is still the case that, given a certain amount of human effort at evaluating an AI's values, Bostrom's treacherous turn suggests a lower chance of successful evaluation than Goertzel's sordid stumble.
 Human creation of human-desirable AI values A There is nothing paradoxical about an AI whose sole final goal is to count the grains of sand on Borcay, or to calculate the decimal expansion of pi, or to maximize the total number of paperclips that will exist in its future light cone. In fact, it would be easier to create an AI with simple goals like these than to build one that had a human-like set of values and dispositions (Bostrom 2014, p.107 ). The logic of the above passage is that creating an AI with human-desirable values is more difficult and thus less likely to occur. Goertzel (2016) , citing Sotala (2015), refers to this as the difficulty thesis: The difficulty thesis: Getting AIs to care about human values in the right way is really difficult, so even if we take strong precautions and explicitly try to engineer sophisticated beneficial goals, we may still fail (Goertzel 2016, p.60) . Goertzel (2016) discusses a Sotala (2015) argument against the difficulty thesis, which is that while human values are indeed complex and difficult to learn, AIs are increasingly capable of learning complex things.Per this reasoning, giving an AI human-desirable values is still more difficult than, say, programming it to calculate digits of pi, but it may nonetheless be a fairly straightforward task for common AI algorithms. Thus, while it would not be easy for humans to create an AI with human-desirable values, it would not be extraordinarily difficult either. Goertzel (2016) , again citing Sotala (2015), refers to this as the weak difficulty thesis: The weak difficulty thesis. It is harder to correctly learn and internalize human values, than it is to learn most other concepts. This might cause otherwise intelligent AI systems to act in ways that went against our values, if those AI systems had internalized a different set of values than the ones we wanted them to internalize. A more important consideration than the absolute difficulty of giving an AI human-desirable values is its relative difficulty compared to the difficulty of creating an AI that could take over the world. A larger relative ease of creating an AI with human-desirable values implies a higher probability that AI catastrophe will be avoided for any given level of effort put to avoiding it. There is reason to believe that the easier task is giving an AI human-desirable values. For comparison, every (or almost every) human being holds humandesirable values. Granted, some humans have more refined values than others, and some engage in violence or other antisocial conduct, but it is rare for someone to have pathological values like an incessant desire to calculate digits of pi. In contrast, none (or almost none) of us is capable of taking over the world. Characters like Alexander the Great and Genghis Khan are the exception, not the rule, and even they could have been assassinated by a single suicidal bodyguard. By the same reasoning, it may be easier for an AI to gain humandesirable values than it is for an AI to take over the world. This reasoning does not necessarily hold, since AI cognition can differ substantially from human cognition, but it nonetheless suggests that giving an AI humandesirable values may be the easier task.
 AI creation of human-desirable AI values A third point of discussion concerns the potential for an AI to end up with human-desirable values even though its human creators did not give it such values. If AIs tend to end up with human-desirable values, this reduces the pressure on the human creators of AI to get the AI's values right. It also increases the overall prospects for a positive AI outcome. To generalize, Bostrom proposes that AIs will tend to maintain stable values, whereas Goertzel proposes that AIs may tend to evolve values that could be more human-desirable. Bostrom's (2014) thinking on the matter centers on a concept he calls goal-content integrity: Goal-content integrity: If an agent retains its present goals into the future, then its present goals will be more likely to be achieved by its future self. This gives the agent a present instrumental reason to prevent alteration of its final goals (Bostrom 2014, p.109-110) . The idea here is that an AI would seek to keep its values intact as one means of realizing its values. At any given moment, an AI has a certain set of values and seeks to act so as to realize these values. One factor it may consider is the extent to which its future self would also seek to realize these values. Bostrom's argument is that an AI is likely to expect that its future self would realize its present values more if the future self retains the present self's values, regardless of whether those values are human-desirable. Goertzel (2016) proposes an alternative perspective that he calls ultimate value convergence: Ultimate value convergence: Nearly all superintelligent minds will converge to the same universal value system (paraphrased from Goertzel 2016, p.60). Goertzel further proposes that the universal value system will be "centered around a few key values such as Joy, Growth, and Choice" (Goertzel 2016, p.60). However, the precise details of the universal value system are less important than the possibility that the value system could resemble human-desirable values. This creates a mechanism through which an AI that begins with any arbitrary human-undesirable value system could tend towards human-desirable values. Goertzel does not insist that the ultimate values would necessarily be human-desirable. To the contrary, he states that "if there are convergent 'universal' values, they are likely sufficiently abstract to encompass many specific value systems that would be abhorrent to us according to our modern human values" (Goertzel 2016, p.60). Thus, ultimate value convergence does not guarantee that an AI would end up with human-desirable values. Instead, it increases the probability that an AI would end up with human-desirable values if the AI begins with human-undesirable values. Alternatively, if the AI begins with human-desirable values, then the ultimate value convergence theory could cause the AI to drift to human-undesirable values. Indeed, if the AI begins with human-desirable values, then more favorable results (from humanity's perspective) would accrue if the AI has goal-content integrity.
 The ASI-PATH model The ASI-PATH model was developed to model pathways to ASI catastrophe (Barrett and Baum 2016) . ASI-PATH is a fault tree model, which means it is a graphical model with nodes that are connected by Boolean logic and point to some failure mode. For ASI-PATH, a failure mode is any event in which ASI causes global catastrophe. Fault tree models like ASI-PATH are used widely in risk analysis across a broad range of domains. A core virtue of fault trees is that, by breaking catastrophe pathways into their constituent parts, they enable more detailed study of how failures can occur and how likely they are to occur. It is often easier to focus on one model node at a time instead of trying to study all potential failure modes simultaneously. Furthermore, the fault tree's logic structure creates a means of defining and quantifying model parameters and combining them into overall probability estimates. Indeed, the three points of the Bostrom-Goertzel disagreement (human evaluation of AI values, human creation of human-desirable AI values, and AI creation of human-desirable AI values) each map to one of the ASI-PATH parameters shown in Figure 1 . In Figure 1 , the top node is ASI catastrophe. The left branch covers events that lead to the ASI gaining "decisive strategic advantage", defined as "a level of technological and other advantages sufficient to enable it [the AI] to achieve complete world domination" (Bostrom, 2014, p. 78). The left branch models scenarios in which an initial "seed" AI undergoes recursive selfimprovement and "takes off", becoming successively more and more intelligent until it becomes an ASI. P1 is the probability that such an AI is possible in the first place. P2 is the probability that a seed AI is created and undergoes recursive self-improvement. P3 is the probability that the AI is contained from gaining decisive strategic advantage; the containment can occur at any point in the process from seed AI to ASI. Containment is any measure that prevents a seed AI from gaining decisive strategic advantage, either by limiting recursive self-improvement or by preventing ASI from gaining decisive strategic advantage. Containment includes confinement, in which the AI's ability to affect the rest of the world is restricted (Section 5), and enforcement, in which AI(s) prevent other AI(s) from gaining decisive strategic advantage. 2 The left branch of Figure 1 covers events that could lead to the ASI taking actions that are "unsafe", which is defined as actions that would result in a major global 2 Barrett and Baum (2017a, p. 400) define confinement as "restrictions built into the AI's hardware or software that limit the AI's ability to affect the rest of the world so that it does not gain decisive strategic advantage". This is slightly different than the Yampolskiy (2012) definition used in Section 5. This difference does not affect the overall argument of the present paper. catastrophe. P4 is the probability that humans will fail to make ASI goals safe. P5 is the probability that the ASI will not make its own goals safe. Finally, P6 is the probability that the ASI will not be deterred from acting unsafely by some other agent, potentially another AI. Because all the logic gates in Figure 1 are "AND", the probability of ASI catastrophe, P, is simply the product of the six component probabilities:    6 1 n n P P (1) For convenience, we assume {P1, P2, P6} = 1. These parameters are unrelated to the Bostrom-Goertzel disagreement as discussed in this paper. Instead, we focus on {P3, P4, P5}, for which there is significant disagreement. P3 relates to the Bostrom-Goertzel disagreement about human evaluation of AI values (Section 2.1). In general, it should be easier to contain an AI earlier in the recursive self-improvement process because at that point it has less intelligence with which it could resist containment. Therefore, one factor in P3 is the potential for human observers to determine early in the process that this particular AI should be contained. The easier it is for humans to evaluate AI values, the earlier in the process they should be able to notice which AIs should be contained, and therefore the more probable it is that containment will succeed. In other words, easier human evaluation of AI values means lower P3. P4 relates to the Bostrom-Goertzel disagreement about human creation of human-desirable AI values (Section 2.2). Human-desirable values are very likely to be safe in the sense that they would avoid major global catastrophe. While one can imagine the possibility that somehow, deep down inside, humans actually prefer global catastrophe, and thus that an AI with humandesirable values would cause catastrophe, we will omit this possibility. Instead, we assume that an AI with human-desirable values would not cause catastrophe. Therefore, the easier it is for humans to create AIs with human-desirable values, the more probable it is that catastrophe would be avoided. In other words, easier human creation of AI with human-desirable values means lower P4. P5 relates to the Bostrom-Goertzel disagreement about AI creation of human-desirable AI values (Section 2.3). We assume that the more likely it is that an AI would create of human-desirable values for itself, the more probable it is that catastrophe would be avoided. In other words, more likely AI creation of AI with humandesirable values means lower P5. For each of these three variables, we define two "expert belief" variables corresponding to Bostrom's and Goertzel's positions on the corresponding issue:  P3B is the value of P3 that follows from Bostrom's position, the treacherous turn.  P3G is the value of P3 that follows from Goertzel's position, the sordid stumble.  P4B is the value of P4 that follows from Bostrom's position, the difficulty thesis.  P4G is the value of P4 that follows from Goertzel's position, the weak difficulty thesis.  P5B is the value of P5 that follows from Bostrom's position, goal-content integrity.  P5G is the value of P5 that follows from Goertzel's position, ultimate value convergence. Given estimates for each of the above "expert belief" variables, one can calculate P according to the formula:       6 1 n nG nG nB nB P W P W P (2) In Equation 2 , W is a weighting variable corresponding to how much weight one places on Bostrom's or Goertzel's position for a given variable. Thus, for example, W3B is how much weight one places on Bostrom's position for P3, i.e. how much one believes that an AI would conduct a treacherous turn. For simplicity, we assume WnB + WnG = 1 for n = {3, 4, 5}. This is to assume that for each of {P3, P4, P5}, either Bostrom or Goertzel holds the correct position. This is a significant assumption: it could turn out to be the case that they are both mistaken. The assumption is made largely for analytical and expository convenience. This much is easy. The hard part is quantifying each of the P and W variables in Equation 2 . What follows is an attempt to specify how we would quantify these variables. We estimate the P variables by relating the arguments of Bostrom and Goertzel to the variables and taking into account any additional aspects of the variables. We aim to be faithful to Bostrom's and Goertzel's thinking. We estimate the W variables by making our own (tentative) judgments about the strength of Bostrom's and Goertzel's arguments as we currently see them. Thus, the P estimations aim to represent Bostrom's and Goertzel's thinking and the W estimations represent our own thinking. Later in the paper we also explore the implications of giving both experts' arguments equal weighting (i.e., WnB = WnG = 0.5 for each n) and of giving full weighting to exclusively one of the two experts. We make no claims to having the perfect or final estimations of any of these parameters. To the contrary, we have low confidence in our current estimations, in the sense that we expect we would revise our estimations significantly in the face of new evidence and argument. But there is value in having some initial estimations to stimulate thinking on the matter. We thus present our estimations largely for sake of illustration and discussion. We invite interested readers to make their own.
 P3 and W3: containment fails The human evaluation of AI values is only one aspect of containment. Other aspects include takeoff speed (faster takeoff means less opportunity to contain AI during recursive self-improvement) and ASI containment (measures to prevent an ASI from gaining decisive strategic advantage). Therefore, the Bostrom-Goertzel disagreement about human evaluation of AI values should only produce a relatively small difference on P3. Bostrom and Goertzel may well disagree on other aspects of P3, but those are beyond the scope of this paper. Bostrom's position, the treacherous turn, corresponds to a higher probability of containment failure and thus a higher value of P3 relative to Goertzel's position, the sordid stumble. We propose a 10% difference in P3 between Bostrom and Goertzel, i.e. P3B -P3G = 0.1. The absolute magnitude of P3B and P3G will depend on various case-specific details-for example, a seed AI launched on a powerful computer is more likely to have a fast takeoff and thus less likely to be contained. For simplicity, we will use P3B = 0.6 and P3G = 0.5, while noting that other values are also possible. Regarding W3B and W3G, our current view is that the sordid stumble is significantly more plausible. We find it relevant that AIs are already capable of learning complex tasks like face recognition, yet such AIs are nowhere near capable of outwitting humans with a web of lies. Additionally, it strikes us as much more likely that an AI would exhibit human-undesirable behavior before it becomes able to deceive humans, and indeed long enough in advance to give humans plenty of time to contain the situation. Therefore, we estimate W3B = 0.1 and W3G = 0.9.
 P4 and W4: humans fail to give AI safe goals The Bostrom-Goertzel disagreement about human creation of human-desirable AI values is relevant to the challenge of humans giving AI safe goals. Therefore, the disagreement can yield large differences in P4. Bostrom's position, the difficulty thesis, corresponds to a higher probability of humans failing to give the AI safe goals and thus a higher value of P4 relative to Goertzel's position, the weak difficulty thesis. The values of P4B and P4G will depend on various case-specific details, such as how hard humans try to give the AI safe goals. As representative estimates, we propose P4B = 0.9 and P4G = 0.4. Regarding W4B and W4G, our current view is that the weak difficulty thesis is significantly more plausible. The fact that AIs are already capable of learning complex tasks like face recognition suggests that learning human values is not a massively intractable task. An AI would not please everyone all the time-this is impossible-but it could learn to have broadly human-desirable values and behave in broadly human-desirable ways. However, we still see potential for the complexities of human values to pose AI training challenges that go far beyond what exists for tasks like face recognition. Therefore, we estimate W4B = 0.3 and W4G = 0.7.
 P5 and W5: AI fails to give itself safe goals The Bostrom-Goertzel disagreement about AI creation of human-desirable AI values is relevant to the challenge of the AI giving itself safe goals. Therefore, the disagreement can yield large differences in P5. Bostrom's position, goal-content integrity, corresponds to a higher probability of the AI failing to give itself safe goals and thus a higher value of P5 relative to Goertzel's position, ultimate value convergence. Indeed, an AI with perfect goal-content integrity will never change its goals. For ultimate value convergence, the key factor is the relation between ultimate values and human-desirable values; a weak relation suggests a high probability that the AI will end up with human-undesirable values. Taking these considerations into account, we propose P5B = 0.95 and P5G = 0.5. Regarding W5B and W5G, our current view is that goal-content integrity is significantly more plausible. While it is easy to imagine that an AI would not have perfect goal-content integrity, due to a range of realworld complications, we nonetheless find it compelling that this would be a general tendency of AIs. In contrast, we see no reason to believe that AIs would all converge towards some universal set of values. To the contrary, we believe that an agent's values derive mainly from its cognitive architecture and its interaction with its environment; different architectures and interactions could lead to different values. Therefore, we estimate W5B = 0.9 and W5G = 0.1.
 The probability of ASI catastrophe Table 1 summarizes the various parameter estimates in Sections 3.1-3.3. Using these estimates, recalling the assumption {P1, P2, P6} = 1, and following Equation 2 gives P = (0.1\*0.6 + 0.9\*0.5) \* (0.3\*0.9 + 0.7\*0.4) \* (0.9\*0.95 + 0.1\*0.5) ≈ 0.25. In other words, this set of parameter estimates implies an approximately 25% probability of ASI catastrophe. For comparison, giving equal weighting to Bostrom's and Goertzel's positions (i.e., setting each WB = WG = 0.5) yields P ≈ 0.26; using only Bostrom's arguments (i.e., setting each WB = 1) yields P ≈ 0.51; and using only Goertzel's arguments (i.e., setting each WG = 1) yields P = 0.1. Catastrophe probabilities of 0.1 and 0.51 may diverge by a factor of 5, but they are both still extremely high. Even "just" a 0.1 chance of major catastrophe could warrant extensive government regulation and/or other risk management. Thus, however much Bostrom and Goertzel may disagree with each other, they would seem to agree that ASI constitutes a major risk.
 PB However, an abundance of caveats is required. First, the assumption {P1, P2, P6} = 1 was made without any justification. Any thoughtful estimates of these parameters would almost certainly be lower. Our intuition is that ASI from AI takeoff is likely to be possible, and ASI deterrence seems unlikely to occur, suggesting {P1, P6} ≈ 1, but that the creation of seed AI is by no means guaranteed, suggesting P2 << 1. This implies P ≈ 0.25 is likely an overestimate. Second, the assumption that the correct position was either Bostrom's or Goertzel's was also made without any justification. They could both be wrong, or the correct position could be some amalgam of both of their positions, or an amalgam of both of their positions plus other position(s). Bostrom and Goertzel are both leading thinkers about ASI, but there is no reason to believe that their range of thought necessarily corresponds to the breadth of potential plausible thought. To the contrary, the ASI topic remains sufficiently unexplored that it is likely that many other plausible positions can be formed. Accounting for these other positions could send P to virtually any value in [0, 1]. Third, the estimates in Table 1 were made with little effort, largely for illustration and discussion purposes. Many of these estimates could be significantly off, even by several orders of magnitude. Given the form of Equation 1, a single very low value for Wn\*Pn would also make P very low. This further implies that P ≈ 0.25 is likely an overestimate, potentially by several orders of magnitude. Fourth, the estimates in
 A practical application: AI confinement A core motivation for analyzing ASI risk is to inform practical decisions aimed at reducing the risk. Risk analysis can help identify which actions would reduce the risk and by how much. Different assessments of the risk-such as from experts' differing viewpoints-can yield different results in terms of which actions would best reduce the risk. Given the differences observed in the viewpoints of Bostrom and Goertzel about ASI risk, it is possible that different practical recommendations could follow. To illustrate this, we apply the above risk analysis to model the effects of decisions on a proposed ASI risk reduction measure known as AI confinement: AI confinement: The challenge of restricting an artificially intelligent entity to a confined environment from which it can't exchange information with the outside environment via legitimate or covert channels if such information exchange was not authorized by the confinement authority (Yampolskiy 2012, p.196) . AI confinement is a type of containment and thus relates directly to the P3 (containment fails) variable in the ASI-PATH model (Figure 1 ). Stronger confinement makes it less likely that an AI takeoff would result in an ASI gaining decisive strategic advantage. Confinement might be achieved, for example, by disconnecting the AI from the internet and placing it in a Faraday cage. Superficially, strong confinement would seem to reduce ASI risk by reducing P3. However, strong confinement could increase ASI risk in other ways. In particular, by limiting interactions between the AI and the human populations, strong confinement could limit the AI's capability to learn human-desirable values, thereby increasing P4 (failure of human attempts to make ASI goals safe). For comparison, AIs currently learn to recognize key characteristics of images (e.g., faces) by examining large data sets of images, often guided by human trainers to help the AI correctly identify image features. Similarly, an AI may be able to learn humandesirable values by observing large data sets of human decision-making, human ethical reflection, or other phenomena, and may further improve via the guidance of human trainers. Strong confinement could limit the potential for the AI to learn human-desirable values, thus increasing P4. Bostrom and Goertzel have expressed divergent views on confinement. Bostrom has favored strong confinement, even proposing a single international ASI project in which "the scientists involved would have to be physically isolated and prevented from communicating with the rest of the world for the duration of the project, except through a single carefully vetted communication channel (Bostrom 2014, p. 253)". Goertzel has explicitly criticized this proposal (Goertzel 2015, p.71-73) and instead argued that an open project would be safer, writing that "The more the AGI system is engaged with human minds and other AGI systems in the course of its self-modification, presumably the less likely it is to veer off in an undesired and unpredictable direction" (Goertzel and Pitt 2012, p.13). Each expert would seem to be emphasizing different factors in ASI risk: P3 for Bostrom and P4 for Goertzel. The practical question here is how strong to make the confinement for an AI. Answering this question requires resolving the tradeoff between P3 and P4. This in turn requires knowing the size of P3 and P4 as a function of confinement strength. Estimating that function is beyond the scope of this paper. However, as an illustrative consideration, suppose that it is possible to have strong confinement while still giving the AI good access to human-desirable values. For example, perhaps a robust dataset of human decisions, ethical reflections, etc. could be included inside the confinement. In this case, the effect of strong confinement on P4 may be small. Meanwhile, if there is no arrangement that could shrink the effect of confinement on P3, such that this effect would be large, then perhaps strong confinement would be better. This and other practical ASI risk management questions could be pursued in future research.
 Conclusion Estimates of the risk of ASI catastrophe can depend heavily on which expert makes the estimate. A neutral observer should consider arguments and estimates from all available experts and any other sources of information. This paper analyzes ASI catastrophe risk using arguments from two experts, Nick Bostrom and Ben Goertzel. Applying their arguments to an ASI risk model, we calculate that their respective ASI risk estimates vary by a factor of five: P ≈ 0.51 for Bostrom and P = 0.1 for Goertzel. Our estimates, combining both experts' arguments, is P ≈ 0.25. Weighting both experts equally gave a similar result of P ≈ 0.26. These numbers come with many caveats and should be used mainly for illustration and discussion purposes. More carefully considered estimates could easily be much closer to either 0 or 1. These numbers are interesting, but they are not the only important part, or even the most important part, of this analysis. There is greater insight to be obtained from the details of the analysis than from the ensuing numbers. This is especially case for this analysis of ASI risk because the numbers are so tentative and the underlying analysis so comparatively rich. This paper is just an initial attempt to use expert judgment to quantify ASI risk. Future research can and should do the following: examine Bostrom's and Goertzel's arguments in greater detail so as to inform the risk model's parameters; consider arguments and ideas from a wider range of experts; conduct formal expert surveys to elicit expert judgments of risk model parameters; explore different weighting techniques for aggregating across expert judgment, as well as circumstances in which weighted aggregation is inappropriate; conduct sensitivity analysis across spaces of possible parameter values, especially in the context of the evaluation of ASI risk management decision options; and do all of this for a wider range of model parameters, including {P1, P2, P6} as well as more detailed components of {P3, P4, P5}, such as modeled in Barrett and Baum (2017a; 2017b) . Future research can also explore the effect on overall ASI risk when multiple ASI systems are launched: perhaps some would be riskier than others, and it may be important to avoid catastrophe from all of them. One overarching message of this paper is that more detailed and rigorous analysis of ASI risk can be achieved when the risk is broken into constituent parts and modeled, such as in Figure 1 . Each component of ASI risk raises a whole host of interesting and important details that are worthy of scrutiny and debate. Likewise, aggregate risk estimates are better informed and generally more reliable when they are made from detailed models. To be sure, it is possible for models to be too detailed, burdening experts and analysts with excessive minutiae. However, given the simplicity of the risk models at this early stage of ASI risk analysis, we believe that, at this time, more detail is better. A final point is that the size of ASI risk depends on many case-specific factors that in turn depend on many human actions. This means that the interested human actor has a range of opportunities available for reducing the probability of ASI catastrophe. Risk modeling is an important step towards identifying which opportunities are most effective at reducing the risk. ASI catastrophe is by no means a foregone conclusion. The ultimate outcome may well be in our hands.
 Acknowledgement We thank Ben Goertzel, Miles Brundage, Kaj Sotala, Steve Omohundro, Allan Dafoe, Stuart Armstrong, Ryan Carey, Nell Watson, and Matthijs Maas for helpful comments on an earlier draft. Any remaining errors are the authors' alone. Work for this paper is funded by Future of Life Institute grant 2015-143911. The views in this paper are those of the authors and do not necessarily reflect the views of the Global Catastrophic Risk Institute or the Future of Life Institute. Figure 1 : 1 Figure 1: ASI catastrophe fault tree. Adapted from Barrett and Baum (2017a).
 Bostrom 2014, p.119) Such an AI would not have durable values in the sense that it would go from acting in human-desirable ways to acting in human-undesirable ways. A key detail of the treacherous turn theory is that the AI has values that are similar to, but ultimately different from, humandesirable values. As the AI gains intelligence, it goes through a series of stages: 1. At low levels of intelligence, the AI acts in ways that humans consider desirable. At this stage, the differences between the AI's values and human values are not important because the AI can only complete simple tasks that are human-desirable. 2. At an intermediate level of intelligence, the AI realizes that its values differ from human-desirable values and that it if it tried deviating from humandesirable values, humans would reprogram the AI or shut it down. Furthermore, the AI discovers that it can successfully pretend to have human-desirable values until it is more intelligent. 3. At a high level of intelligence, the AI takes control of the world from humanity so that humans cannot reprogram it or shut it down, and then pursues its actual, human-undesirable values.
 Table 1 : 1 Summary of parameter estimates in Sections 3.1-3.3. PG WB WG 3 0.6 0.5 0.1 0.9 4 0.9 0.4 0.3 0.7 5 0.95 0.5 0.9 0.1
 Table 1 depend on a range of case-specific factors, including what other containment measures are used, how much effort humans put into giving the AI human-desirable values, and what cognitive architecture the AI has. Therefore, different seed AIs self-improving under different conditions would yield different values of P, potentially including much larger and much smaller values.
 On expert opinion of ASI, see Baum et al. (2011), Armstrong and Sotala (2012), Armstrong et al. (2014), and Müller and Bostrom (2014).

Seth Baum
Anthony Barrett
Roman V. Yampolskiy