Should Artificial Intelligence Governance be Centralised? Six Design Lessons from History

Should Artiﬁcial Intelligence Governance be Centralised?
Six Design Lessons from History
Peter Cihon,1Matthijs M. Maas,1,2Luke Kemp3
1Centre for the Governance of AI, Future of Humanity Institute, University of Oxford
2Centre for International Law, Conﬂict, and Crisis, Faculty of Law, University of Copenhagen
3Centre for the Study of Existential Risk, University of Cambridge
petercihon@gmail.com; matthijs.maas@jur.ku.dk; ltk27@cam.ac.uk
Abstract
Can effective international governance for artiﬁcial intelli-
gence remain fragmented, or is there a need for a centralised
international organisation for AI? We draw on the history of
other international regimes to identify advantages and disad-
vantages in centralising AI governance. Some considerations,
such as efﬁciency and political power, speak in favour of cen-
tralisation. Conversely, the risk of creating a slow and brittle
institution speaks against it, as does the difﬁculty in securing
participation while creating stringent rules. Other considera-
tions depend on the speciﬁc design of a centralised institution.
A well-designed body may be able to deter forum shopping
and ensure policy coordination. However, forum shopping
can be beneﬁcial and a fragmented landscape of institutions
can be self-organising. Centralisation entails trade-offs and
the details matter. We conclude with two core recommenda-
tions. First, the outcome will depend on the exact design of a
central institution. A well-designed centralised regime cover-
ing a set of coherent issues could be beneﬁcial. But locking-in
an inadequate structure may pose a fate worse than fragmen-
tation. Second, for now fragmentation will likely persist. This
should be closely monitored to see if it is self-organising or
simply inadequate.
In 2018, Canada and France proposed the International
Panel on Artiﬁcial Intelligence (IPAI). After being rejected
at the G7 in 2019, negotiations shifted to the OECD and
are presently ongoing. As the ﬁeld of AI continues to ma-
ture and spark public interest and legislative concern (Per-
rault et al. 2019), the priority of governance initiatives re-
ﬂects the growing appreciation that AI has the potential
to dramatically change the world for both good and ill
(Dafoe 2018). Research into AI governance needs to keep
pace with policy-making and technological change. Choices
made today may have long-lasting impacts on policymak-
ers’ ability to address numerous AI policy problems (Cave
and´Oh´Eigeartaigh 2019). Effective governance can promote
safety, accountability, and responsible behaviour in the re-
search, development, and deployment of AI systems.
AI governance research to date has predominantly fo-
cused at the national and sub-national levels (Scherer 2016;
Equal contribution, order selected at random.
Working paper, last updated 15 December, 2019.Calo 2017; Gasser and Almeida 2017). Research into AI
global governance remains relatively nascent (though see
Butcher and Beridze 2019). Kemp et al. (2019) have called
for specialised, centralised intergovernmental agencies to
coordinate policy responses globally, and others have called
for a centralised ‘International Artiﬁcial Intelligence Organ-
isation’ (Erdelyi and Goldsmith 2018). Others favour more
decentralised arrangements based around ‘Governance Co-
ordinating Committees’, global standards, or existing in-
ternational law instruments (Wallach and Marchant 2018;
Cihon 2019; Kunz and ´O h´Eigeartaigh 2020).
No one has taken a step back to inquire: what would the
history of multilateralism suggest, given the state and tra-
jectory of AI? Should AI governance be centralised or de-
centralised? ‘Centralisation’, in this case, refers to the de-
gree to which the coordination, oversight and/or regulation
of a set of AI policy issues or technologies are housed under
a single (global) institution. This is not a binary choice; it
exists across a spectrum. Trade is highly (but not entirely)
centralised under the umbrella of the WTO. In contrast, en-
vironmental multilateralism is much more decentralised.
In this paper, we seek to help the community of re-
searchers, policymakers, and other stakeholders in AI gover-
nance understand the advantages and disadvantages of cen-
tralisation. This may help set terms and catalyse a much-
needed debate to inform governance design decisions. We
ﬁrst outline the international governance challenges of AI,
and review early proposed global responses. We then draw
on existing literatures on regime fragmentation (Biermann et
al. 2009) and ‘regime complexes’ (Orsini, Morin, and Young
2013) to assess considerations in centralising the interna-
tional governance of AI. We draw on the history of other
international regimes1to identify considerations that speak
in favour or against designing a centralised regime com-
plex for AI. We conclude with two recommendations. First,
many trade-offs are contingent on how well-designed a cen-
tral body would be. An adaptable, powerful institution with
a manageable mandate would be beneﬁcial, but a poorly de-
1A regime is a set of ‘implicit or explicit principles, norms, rules
and decision-making procedures around which actors’ expectations
converge in a given area of international relations’(Krasner 1982,
186).
signed body could prove a fate worse than fragmentation.
Second, for now there should be structured monitoring of ex-
isting efforts to see whether they are they are self-organising
or insufﬁcient.
The State of AI Governance
There is debate as to whether AI is a single policy area or
a diverse series of issues. Some claim that AI cannot be co-
hesively regulated as it is a collection of disparate technolo-
gies, with different risk proﬁles across different applications
and industries (Stone et al. 2016). This is an important but
not entirely convincing objection. The technical ﬁeld has no
settled deﬁnition for ‘AI’,2so it should be no surprise that
deﬁning a manageable scope for AI governance will be dif-
ﬁcult. Yet this challenge is not unique to AI: deﬁnitional is-
sues abound in areas such as environment and energy, but
have not ﬁgured prominently in debates over centralisation.
Indeed, energy and environment ministries are common at
the domestic level, despite problems in setting the bound-
aries of natural systems and resources.
We contend that there are numerous ways in which a
centralised body could be designed for AI governance. For
example, a centralised approach could carve out a subset
of interlinked AI issues to cover. This could involve fo-
cusing on the potentially high-risk applications of AI sys-
tems, such as AI-enabled cyberwarfare, lethal autonomous
weapons (LAWS), other advanced military applications, or
high-level machine intelligence (HLMI).3Another approach
could govern underlying hardware resources (e.g. large-
scale compute resources) or software libraries. We are ag-
nostic on the speciﬁcs of how centralisation could or should
be implemented, and instead focus on the costs and beneﬁts
of centralisation in the abstract. The exact advantages and
disadvantages of centralisation are likely to vary depend-
ing on the institutional design. This is an important area
of further study, particularly once more speciﬁc proposals
are put forward. However, such work must be grounded in
a higher-level investigation of trade-offs in centralising AI
governance. It is this foundational analysis which we seek
to offer.
Numerous AI issues could beneﬁt from international co-
operation. These include the potentially catastrophic appli-
cations mentioned above. It also encompasses more quo-
tidian uses, such as AI-enabled cybercrime; human health
applications; safety and regulation of autonomous vehicles
and drones; surveillance, privacy and data-use; and labour
automation. Multilateral coordination could also use AI to
tackle other global problems such as climate change (see
Rolnick et al. 2019), or help meet the Sustainable Develop-
ment Goals (see Vinuesa et al. 2019). This is an illustrative
but not exhaustive list of international AI policy issues.
Global regulation across these issues is currently nascent,
2We deﬁne ‘AI’ as any machine system capable of function-
ing ‘appropriately and with foresight in its environment’ (Nilsson
2009, 13; see too Dafoe 2018, 5).
3‘High-level machine intelligence’ has been deﬁned as ‘unaided
machines [that] can accomplish every task better and more cheaply
than human workers’ (Grace et al. 2018, 1).fragmented, yet evolving. A wide range of UN institutions
have begun to undertake some activities on AI (ITU 2019).
The bodies covering AI policy issues range across existing
organisations including the International Labour Organisa-
tion (ILO), International Telecommunication Union (ITU),
and UNESCO. This is complemented by budding regula-
tions and working groups across the International Organ-
isation for Standardisation (ISO), International Maritime
Organisation (IMO), International Civil Aviation Organisa-
tion (ICAO), and other bodies, as well as treaty amend-
ments, such as the updating of the Vienna Convention on
Road Trafﬁc to encompass autonomous vehicles (Kunz and
´O h´Eigeartaigh 2020), or the ongoing negotiations at the
Convention on Certain Conventional Weapons (CCW) on
LAWS. The UN System Chief Executives Board (CEB) for
Coordination through the High-Level Committee on Pro-
grammes has been empowered to draft a system-wide AI
capacity building strategy. The High-level Panel on Digi-
tal Cooperation has also sought to gather together common
principles and ideas for AI relevant areas (High-Level Panel
2019). Whether these initiatives bear fruit, however, remains
questionable, as many of the involved international organ-
isations have fragmented membership, were not originally
created to address AI issues and lack effective enforcement
or compliance mechanisms (see Morin et al. 2019, 2).
The trajectory of these initiatives matters. How gover-
nance is initially organised can be central to its success. De-
bates over centralisation and fragmentation are long-lasting
and prominent with good reason. How we structure inter-
national cooperation can be critical to its success, and most
other debates often implicitly hinge on structural debates.
Fragmentation and centralisation exist across a spectrum. In
a world lacking a global government, some fragmentation
will always prevail. But the degree to which it prevails is
crucial. We deﬁne ‘fragmentation’ as a patchwork of inter-
national organisations and institutions which focus on a par-
ticular issue area, but differ in scope, membership and often
rules. Our other deﬁnitions for key terms are provided be-
low in Table 1. These deﬁnitions and terms are by nature
normatively loaded. For example, some may ﬁnd ‘decen-
tralisation’ to be a positive framing, while others may see
‘fragmentation’ to possess negative connotations. Recognis-
ing this, we seek to use these terms in a primarily analytical
manner. We will use ﬁndings from each of these theoretical
areas to inform our discussion of the history of multilateral
fragmentation and its implications for AI governance.
Centralisation Criteria:
A History of Governance Trade-Offs
In the following discussion, we explore a series of consid-
erations for AI governance. Political power and efﬁcient
participation support centralisation. The breadth vs. depth
dilemma, as well as slowness and brittleness support decen-
tralisation. Policy coordination and forum shopping consid-
erations can cut both ways.
Table 1: Deﬁnition of Key Governance Terms
Term Deﬁnition
Fragmentation or
DecentralizationA patchwork of international organisations and institutions which focus on a particular issue area but
differ in scope, membership and often rules (Biermann et al. 2009, 16).
Centralisation An arrangement in which governance of a particular issue lies under the authority of a single umbrella
body. This is a spectrum from highly centralised (the role of the WTO in trade) to decentralised (the
plethora of multilateral environmental agreements).
Regime Complex A network of three or more international regimes on a common issue area. These should have over-
lapping membership and cause potentially problematic interactions (Orsini, Morin, and Young 2013,
29).
1. Political Power
Regimes embody power in their authority over rules, norms,
and knowledge beyond states’ exclusive control. A more
centralised regime will see this power concentrated among
fewer institutions. A centralised, powerful architecture is
likely to be more inﬂuential against competing international
organisations and with constituent states (see Orsini, Morin,
and Young 2013, 36-7).
An absence of centralised authority to manage regime
complexes has presented challenges in the past. Across
the proliferation of Multilateral Environmental Agreements
(MEAs) there is no requirement to cede responsibility to
the UN Environmental Programme in the case of overlap
or competition. This has led to turf wars, inefﬁciencies and
even contradictory policies (Biermann et al. 2009). One of
the most notable examples is that of hydroﬂuorocarbons
(HFCs). HFCs are potent greenhouse gases, and yet their
use has been encouraged by the Montreal Protocol since
1987 as a replacement for ozone-depleting substances. This
has only recently been resolved via the 2015 Kigali Amend-
ment to the Montreal Protocol, which itself has a prolonged
implementation period. Similarly, the internet governance
regime complex is diffuse. Multiple venues and norms gov-
ern technical standards, cyber crime, human rights, and war-
fare (Nye 2014). Although the UN Internet Governance Fo-
rum (IGF) discusses several cross-cutting issues, it does not
have a mandate to consolidate even principles, let alone ne-
gotiate new formal agreements (Mueller, Mathiason, and
Klein 2007).
In contrast, other centralised regimes have supported ef-
fective management. For example, under the umbrella of
the WTO, norms such as the most-favoured-nation princi-
ple (equally treating all WTO member states) principle have
become the bedrock of international trade. The power and
track-record of the WTO is so formidable that it has created
achilling effect: the fear of colliding with WTO norms and
rules has led environmental treaties to self-censor and ac-
tively avoid discussing or deploying trade-related measures
(Eckersley 2004). Both the chilling effect and the remark-
ably powerful application of common trade rules were not
a marker of international trade until the establishment of the
WTO. The power of these centralised body has stretched be-
yond inﬂuencing states in the domain of trade, to mouldingrelated issues.
Political power offers further beneﬁts in governing emerg-
ing technologies that are inherently uncertain in both sub-
stance and policy impact. Uncertainty in technology and
preferences has been associated with some increased cen-
tralisation in regimes (Koremenos, Lipson, and Snidal
2001a). There may also be beneﬁts to housing a foresight
capacity within the regime complex, to allow for acceler-
ated or even proactive efforts (Pauwels 2019). Centralised
AI governance would enable an empowered organisation to
more effectively use foresight analyses to inform policy re-
sponses across the regime complex.
2. Supporting Efﬁciency & Participation
Decentralised AI governance may undermine efﬁciency and
inhibit participation. States often create centralised regimes
to reduce costs, for instance by eliminating duplicate efforts,
yielding economies of scale within secretariats, and sim-
plifying participation (Esty and Ivanova 2002). Conversely,
fragmented regimes may force states to spread resources and
funding over many distinct institutions, particularly limiting
the ability of less well-resourced states or parties to partici-
pate fully (Morin et al. 2019, 2).
Historically, decentralised regimes have presented cost
and related participation concerns. Hundreds of related and
sometimes overlapping international environmental agree-
ments can create ‘treaty congestion’ (Anton 2012). This
complicates participation and implementation for both de-
veloped and developing nations (Esty and Ivanova 2002).
This includes costs associated with travel to different fo-
rums, monitoring and reporting for a range of different bod-
ies, and duplication of effort by different secretariats (ibid.).
Similar challenges are already being witnessed in AI
governance. Simultaneous and globally distributed meet-
ings pose burdensome participation costs for civil society.
Fragmented organisations must duplicatively invest in high-
demand machine learning subject matter experts to inform
their activities. Centralisation would support institutional ef-
ﬁciency and participation.
3. Slowness & Brittleness of Centralised Regimes
One potential problem of centralisation lies in the relatively
slow process of establishing centralised institutions, which
may often be outpaced by the rate of technological change.
Another challenge lies in centralised institutions’ brittleness
after they are established, i.e., their vulnerability to regula-
tory capture, or failure to react to changes in the problem
landscape.
Establishing new international institutions is often a slow
process. For example, the Kyoto Protocol took three years
of negotiations to create and then another eight to enter into
force. This becomes even more onerous with higher partici-
pation and stakes. Under the GATT, negotiations for a 26%
cut in tariffs between 19 countries took 8 months in 1947.
The Uruguay round, beginning in 1986, took 91 months
to achieve a tariff reduction of 38% between 125 parties
(Martin and Messerlin 2007). International law has been
quick to respond to technological changes in some cases,
and delayed in others (Picker 2001, 184). Decentralised ef-
forts may prove quicker to respond to complex, ‘transver-
sal’ issues, if they rely more on informal institutions with
a smaller but like-minded membership (Morin et al. 2019,
2-3). Centralised AI governance may be particularly vulner-
able to sparking lengthy negotiations, because progress on
centralised regimes for new technologies tends to be hard if
a few states hold clearly unequal stakes in the technology,
or if there are signiﬁcant differences in information and ex-
pertise among states or between states and private industry
(Picker 2001, 187-94). Both these conditions closely match
the context of AI technology. Moreover, because AI tech-
nology develops rapidly, such slow implementation of rules
and principles could lead to certain actors taking advantage
by setting de facto arrangements or extant state practice.
Even after its creation, a centralised regime can be brittle ;
the very qualities that provide it with political power may
exacerbate the adverse effects of regulatory capture; the fea-
tures that ensure institutional stability, may also mean that
the institution cannot adapt quickly to unanticipated outside
stressors outside its established mission. The regime might
break before it bends. The ﬁrst potential risk is regulatory
capture. Given the high proﬁle of AI issue areas, political in-
dependence is paramount. However, as illustrated by numer-
ous cases, including undue corporate inﬂuence in the WHO
during the 2009 H1N1 pandemic (Deshman 2011), no insti-
tution is fully immune to regime capture, and centralisation
may reduce the costs of lobbying, making capture easier by
providing a single locus of inﬂuence. On the other hand, a
regime complex comprising many parallel institutions could
ﬁnd itself vulnerable to capture by powerful actors, who are
better positioned than smaller parties to send representatives
to every forum.
Moreover, centralised regimes entail higher stakes. Many
issues are in a single basket and thus failure is more likely
to be severe if it does occur. International institutions can be
notoriously path-dependent and thus fail to adjust to chang-
ing circumstances, as seen with the ILO’s considerable dif-
ﬁculties in reforming its participation and rulemaking pro-
cesses in the 1990s (Baccaro and Mele 2012). The public
failure of a ﬂagship global AI institution or governance ef-
fort could have lasting political repercussions. It could stran-
gle subsequent, more well-conceived proposals in the crib,
by undermining conﬁdence in multilateral governance gen-erally or capable governance on AI issues speciﬁcally. By
contrast, for a decentralized regime complex to similarly
fail, all of its component institutions would need to simul-
taneously ‘break’ or fail to innovate at once.4A centralised
institution that does not outright collapse, but which remains
ineffective, may become a blockade against better efforts.
Ultimately, brittleness is not an inherent weakness of
centralisation–and indeed depends far more on institutional
design details. There may be strategies to ‘innovation-
proof’(Maas 2019) governance regimes. Periodic renegoti-
ation, modular expansion, ‘principles based regulation’, or
sunset clauses can also support ongoing reform (see gener-
ally Marchant, Allenby, and Herkert 2011, 29-30). Such ap-
proaches have often proved successful historically, due par-
tially to decentralisation but, importantly, also to particular
designs.
4. The Breadth vs. Depth Dilemma
Pursuing centralisation may create an overly high threshold
that limits participation. All multilateral agreements face a
trade-off between having higher participation (‘breadth’) or
stricter rules and greater ambition of commitments (‘depth’).
The dilemma is particularly evident for centralised institu-
tions that are intended to be powerful and require strong
commitments from states.
However, the opposite dynamics of sacriﬁcing depth for
breadth can also pose risks. The 2015 Paris Agreement on
Climate Change was signiﬁcantly watered down to allow for
the legal participation of the US. Anticipated difﬁculties in
ratiﬁcation through the Senate led to negotiators opting for
a ‘pledge and review’ structure with few legal obligations.
Thus, the US could join simply through the approval of the
executive (Kemp 2017). In this case, inclusion of the US
(which at any rate proved temporary) came at the cost of sig-
niﬁcant cutbacks on the demands which the regime sought
to make of all parties.
In contrast, decentralisation could allow for major powers
to engage in relevant regulatory efforts where they would
be deterred from signing up to a more comprehensive pack-
age. This has precedence in the history of climate gover-
nance. Some claim that the US-led Asia-Paciﬁc Partnership
on Clean Development and Climate helped, rather than hin-
dered climate governance, as it bypassed UNFCCC dead-
lock and secured non-binding commitments from actors not
bound by the Kyoto Protocol (Zelli 2011, 259-60).
This matters, as buy-in may prove a thorny issue for AI
governance. The actors who lead in AI development include
powerful states that are potentially most adverse to global
regulation in this area. They have thus far proved recalci-
trant in the global governance of security issues such as anti-
personnel mines or cyberwarfare. In response, some have al-
ready recommended a critical-mass governance approach to
the military uses of AI. Rather than seeking a comprehen-
sive agreement, devolving and spinning off certain compo-
nents into separate treaties (e.g. for LAWS testing standards;
liability and responsibility; and limits to operational usage)
4We thank Nicolas Mo ¨es for this observation.
could instead allow for the powerful to ratify and move for-
ward at least a few of those options (Weaver 2014).
The breadth vs. depth dilemma is a trade-off in multi-
lateralism generally. However, it is a particularly pertinent
challenge for centralisation. The key beneﬁt of a centralised
body would be to be a powerful anchor that ensures pol-
icy coordination and coherence, without suffering fragmen-
tation in membership. This dilemma suggests it is unlikely
to have both. It will likely need to restrict membership to
have teeth, or lose its teeth to have wide participation. A crit-
ical mass approach may be able to deliver the best of both
worlds. Nonetheless these dilemma poses a difﬁcult knot for
centralisation to unravel.
5. Forum Shopping
Forum shopping may help or hinder AI governance, de-
pending on the particular circumstances. Fragmentation en-
ables actors to choose where and how to engage. Such
‘forum shopping’ may take one of several forms: moving
venues, abandoning one organisation, creating new venues,
and working across multiple organisations to sew compe-
tition between them (Braithwaite and Drahos 2000). Even
when there is a natural venue for an issue, actors have rea-
sons to forum-shop. For instance, states may look to max-
imise their inﬂuence, appease domestic pressure (Pekkanen,
Sol´ıs, and Katada 2007) and placate constituents by shifting
to a toothless forum (Helfer 2004).
The ability to successfully forum-shop depends on an
actor’s power. Most successful examples of forum-shifting
have been led by the US (Braithwaite and Drahos 2000). In-
tellectual property rights in trade, for example, was subject
to prolonged, contentious forum shopping. Developed states
resisted attempts of the UN Conference on Trade and Devel-
opment (UNCTAD) to address intellectual property rights
in trade by trying to push them onto the World Intellectual
Property Organization (WIPO) (ibid., 566) and then subse-
quently to the WTO (Helfer 2004), overruling protests from
developing states. Outcomes often reﬂect power, but weak
states and non-state actors can also pursue forum shopping
strategies in order to challenge the status-quo (Jupille, Mat-
tli, and Snidal 2013).
Forum shopping may help or hurt governance. This is ev-
ident in current efforts to regulate LAWS. While the Group
of Governmental Experts has made some progress, on the
whole the CCW has taken slow deliberations on LAWS.
In response, frustrated activists have threatened to shift to
another forum, as happened with the Ottawa Treaty that
banned landmines (Delcker 2019). This strategy could catal-
yse progress, but also brings risks of further forum shop-
ping and weak or unimplemented agreements. Forum shop-
ping may similarly delay, stall, or weaken regulation of time-
sensitive AI policy issues, including potential future HLMI
development. It is plausible that leading AI ﬁrms also have
sway when they elect to participate in some venues but not
others. The OECD Expert Group on AI included representa-
tives from leading ﬁrms, whereas engagement at UN efforts,
including the Internet Governance Forum (IGF), do not ap-
pear to be similarly prioritised. A decentralised regime will
enable forum shopping, though further work is needed to de-termine whether this will help or hurt governance outcomes
on the whole.
6. Policy Coordination
There are good reasons to believe that either centralisa-
tion or fragmentation could enhance coordination. A cen-
tralised regime can enable easier coordination both across
and within policy issues, acting as a focal point for states.
Others argue that this is not always the case, and that frag-
mentation can mutually supportive and even more creative
institutions.
Centralisation reduces the occurrence of conﬂicting man-
dates and enables communication. These are the ingredi-
ents for policy coherence. As noted previously, the WTO
has been remarkably successful in ensuring coherent policy
and principles across the realm of trade, and even into other
areas such as the environment.
However, fragmented regimes can often act as complex
adaptive systems. Political requests and communication be-
tween secretariats often ensures bottom-up coordination
even in the absence of centralisation. Multiple organisations
have sought to reduce greenhouse gas emissions within their
respective remits, often at the behest of the UNFCCC Con-
ference of Parties. When effective, bottom-up coordination
can slowly evolve into centralisation. Indeed, this was the
case for the GATT and numerous regional, bilateral and sec-
toral trade treaties, which all coalesced together into the
WTO. While this organic self-organisation has occurred, it
has taken decades, with forum shopping and inaction pre-
vailing for many years.
Indeed, some have argued that decentralisation does not
just deliver ‘good enough’ global governance (Patrick 2014)
that reﬂects a demand for diverse principles in a multi-
polar world. Instead, they argue ‘polycentric’ governance
approaches (Ostrom 2010) may be more creative and le-
gitimate than centrally coordinated regimes. Arguments in
favour of polycentricity include the notion that it enables
governance initiatives to begin having impacts at diverse
scales, and that it enables experimentation with diverse poli-
cies and approaches, learning from experience and best prac-
tices (ibid., 552). Consequently, these scholars assume “that
the invisible hand of a market of institutions leads to a better
distribution of functions and effects” (Zelli and van Asselt
2013, 7).
It is unclear if the different bodies covering AI issues will
self-organise or collide. Many of the issues are interdepen-
dent and will need to be addressed in tandem. Some par-
ticular policy-levers, such as regulating computing power or
data, will impact almost all use areas, given that AI progress
and use is closely tied to such inputs. Numerous initiatives
on AI and robotics are displaying loose coordination (Kunz
and ´O h´Eigeartaigh 2020), but it remains uncertain whether
the virtues of a free market of governance will prevail here.
Great powers can exercise monopsony-like inﬂuence in fo-
rum shopping, and the supply of both computing power and
machine learning expertise are highly concentrated. In sum,
centralisation can reduce competition and enhance coordi-
nation, but it may suffocate the creative self-organisation of
more fragmented arrangements over time.
Discussion: What Would History Suggest?
A Summary of Considerations
The multilateral track record and peculiarities of AI yield
suggestions and warnings for the future. A centralised
regime could lower costs, support participation, and act as
a powerful new linchpin within the international system. Yet
centralisation presents risks for AI governance. It could sim-
ply produce a brittle dinosaur, of symbolic value but with lit-
tle meaningful impact on underlying political or technologi-
cal issues. A poorly executed attempt could lock-in a poorly
designed centralised body: a fate worse than fragmentation.
Accordingly, ongoing efforts at the UN, OECD, and else-
where could beneﬁt from addressing the considerations pre-
sented in this paper, a summary of which is presented in Ta-
ble 2.
The Limitations of ‘Centralisation vs.
Decentralisation’ Debates
Structure is not a panacea. Speciﬁc provisions such as agen-
das and decision-making procedures matter greatly, as do the
surrounding politics. Underlying political will may be im-
pacted by framing or connecting policy issues (Koremenos,
Lipson, and Snidal 2001b, 770-1). The success of a regime
is not just a result of fragmentation, but of design details.
Moreover, institutions can be dynamic and broaden over
time by taking in new members, or deepen in strengthening
commitments. Successful multilateral efforts, such as trade
and ozone depletion, tend to do both. We are in the early
days of global AI governance. Decisions taken early on will
constrain and partially determine the future path. This de-
pendency can even take place across regimes. The Kyoto
Protocol was largely shaped by the targets and timetables
approach of the Montreal Protocol, which in turn drew from
the Convention on Long-range Transboundary Air Pollution.
The choices we make on governing short-term AI challenges
will likely shape the management of other policy issues in
the long term (Cave and ´Oh´Eigeartaigh 2019).
On the other hand, committing to centralisation, even if
successful, may amount to solving the wrong problem. The
problem may not be structural, but geopolitical. Centralisa-
tion could even exacerbate the problem by diluting scarce
political attention, incurring heavy transaction costs, and
shifting discussions away from bodies which have accumu-
lated experience and practice (Juma 2000). For example,
the Bretton Woods Institutions of the IMF and World Bank,
joined later by the WTO, are centralised regimes that engen-
der power. However, those institutions had the express sup-
port of the US and may have simply manifested state power
in institutional form. Efforts to ban LAWS and create a cy-
berwarfare convention have been broadly opposed by states
with an established technological superiority in these areas
(Eilstrup-Sangiovanni 2018). A centralised regime may not
unpick these power struggles, but just add a layer of com-
plexity.
HLMI: An Illustrative Example
The promise and peril of centralisation may differ by policy
issue. HLMI is one issue that is markedly unique: HLMI isdistinct in its risk proﬁle, uncertainty and linkage to other AI
policy issues, which can make it an interesting case through
which to explore the tradeoffs of a centralised AI governance
regime in a fresh context. While timelines are uncertain, the
creation of HLMI systems is the express goal of various
present-day projects (Baum 2017), and the future develop-
ment of an unaligned HLMI could have catastrophic con-
sequences (GCF 2018). Creation of a controlled HLMI by
a subset of private or public actors could lead to grotesque
power imbalances. It could also exacerbate other AI policy
problems, such as labour automation and advanced military
applications (by providing a coordinating platform, strate-
gic advisor, or in nuclear command and control). Address-
ing many shorter-term issues, such as cyberwarfare, and im-
proving global governance more broadly will have signiﬁ-
cant impacts on HLMI development and deployment (Kunz
and ´O h´Eigeartaigh 2020). There is also marked uncertainty
about whether HLMI can be created in a single system, what
it would look like, and signiﬁcant disagreement as to how
long this would take (Grace et al. 2018).
Below in Table 3 we provide a brief application of our
framework to HLMI. It shows that centralisation of gover-
nance is particularly promising for HLMI. This is due to
its neglect, stakes, scope, and need for informed, preemp-
tive policy. Many other issues, such as the advanced mili-
tary applications of AI systems, may similarly be more pro-
ductively or safely subjected to cooperation, i.e., centralised.
These cases may prove a broad rule about the value of cen-
tralisation, or they may be outliers. Rather than any AI gov-
ernance blueprint, our trade-offs framework provides one
way of thinking through the costs and beneﬁts of centralising
governance either on or across speciﬁc AI issues. Identify-
ing areas which are more easily deﬁned and garner the bene-
ﬁts of centralised regulation provides an organic approach to
thinking through what subset of topics an AI umbrella body
could cover. HLMI, appears to one of the most appealing
candidates.
Lessons and Conclusions
Our framework provides a tool for policy-makers to inform
their decisions of whether to join, create, or forgo new insti-
tutions that tackle AI policy problems. For instance, the re-
cent choice of whether to support the creation of an indepen-
dent IPAI involved these considerations. Following the US
veto, ongoing negotiations for its replacement at the OECD
may similarly beneﬁt from their consideration. For now, it is
worth closely monitoring the current landscape of AI gov-
ernance to see if it exhibits enough policy coordination and
political power to effectively deal with mounting AI policy
problems. While there are promising initial signs (Kunz and
´O h´Eigeartaigh 2020) there are also already growing gover-
nance failures in LAWS, cyberwarfare, and elsewhere.
We outline a suggested monitoring method in Table 4.
There are three key areas to monitor: conﬂict, coordina-
tion, and catalyst. First, conﬂict should measure the extent to
which principles, rules, regulations and other outcomes from
different bodies in the AI regime complex undermine or con-
tradict each other or are in tension either in their principles
Table 2: Summary of Considerations
Consideration Implications for
CentralisationHistorical Example AI Policy Issue Example
Political Power Pro Shaping other regimes: WTO has cre-
ated a chilling effect, where the fear of
conﬂicting with WTO norms and rules
has led environmental treaties to self-
censor to avoid addressing trade-related
measures.Empowered regime using foresight on
AI systems development can address
policy problems more quickly.
Efﬁciency
& ParticipationPro Decentralisation raises inefﬁciencies
and barriers: The proliferation of
multilateral environmental agreements
poses costs and barriers to participation
in negotiation, implementation, and
monitoring.AI companies engage and share exper-
tise, but if not checked by adversarial
civil society, there is a greater concern
of regulatory capture; increased costs un-
dermine civil society participation.
Slowness
& BrittlenessCon Slowness: Under the GATT, 1947 tariff
negotiations among 19 countries took 8
months. The Uruguay round, beginning
in 1986, took 91 months for 125 parties
to agree on reductions.
Regulatory capture: WHO accused
offor undue corporate inﬂuence in
response to 2009 H1N1 pandemic.
Pathology of path-dependence: Failed
ILO reform attempts.Process of centralised regime can
not keep pace with high speed of AI
progress and deployment, may miss the
window of opportunity.
Advanced AI issues (especially HLMI)
may rapidly shift the risk landscape or
problem portfolio of AI, beyond the nar-
row scope of an older institutional man-
date
Breadth vs.
Depth DilemmaCon Watering down: 2015 Paris Agreement
suggest attempts to ‘get all parties on
board’ to centralized regime may result
in signiﬁcant watering down.Attempts to effectively govern the
military uses of AI have been resisted by
the most powerful states.
Attempted to create an IPAI have
been resisted by the US and shifted to a
smaller forum (the OECD).
Forum
ShoppingDepends on
designPower predicts outcomes:
Intellectual property in trade shifted
from UNCTAD to WIPO to WTO, with
developed countries getting their way.
Accelerates progress: NGOs and
some states shifted discussions of
anti-personnel mines ban away from
CCW, ultimately resulting in the Ottawa
Treaty.Governance of military AI systems is
fractured across CCW, multiple GGEs.
This strategy may catalyze progress, but
brings risks of fracture.
Policy
CoordinationDepends on
designStrong, but delayed convergence:
Diverse regimes can coalesce into cen-
tralized regime, as seen with GATT and
numerous trade treaties coalescing into
the WTO, but doing so may take many
decades.Numerous AI governance initiatives dis-
play loose coordination, but it is unclear
if these initiatives can respond to policy
developments in a timely manner.
Table 3: An Application of the Framework to High-Level Machine Intelligence (HLMI)
Consideration HLMI
Political Power Uncertainty around HLMI development makes credible forecasting particularly important. Understand-
ing which inputs drive AI progress, and when and by whom HLMI could be created, is paramount to
ensuring safe development (see Dafoe 2018). This will require a coordinated effort to track and forecast
HLMI project efforts (see Baum 2017), as well as a politically empowered organisation to quickly act
upon this information.
The potentially catastrophic risks make the increased political power of a centralised institution
desirable. The creation of HLMI, if it can be done by a well-resourced actor, is a ‘free-driver’ issue. An
effective response needs to have the teeth to deter major players from acting unilaterally.
Efﬁciency
& ParticipationCentralisation would support economies of scale in expertise to support efﬁcient governance. Given the
signiﬁcant ﬁnancial resources and infrastructure likely needed for such a project, a joint global effort
could be an efﬁcient way to govern HLMI research.
Slowness
& BrittlenessIf short timelines (less than 10-15 years) are expected, the lengthy period to negotiate and create such a
body would be a critical weakness. If longer timelines are more likely, there should be sufﬁcient time
to develop a centralised anchor institution.
Institutional capture is a concern given the abundance of wealthy corporate actors involved in
creating HLMI (Google, OpenAI, Microsoft). However, it is unclear if this would be more likely under
a centralised body.
Depth vs.
Breadth DilemmaThe limited scope of actors makes centralisation more feasible. Costs and requisite tacit knowledge may
restrict the development of HLMI to a few powerful players. The breadth vs. depth dilemma could be
avoided through a ‘minilateral’ or critical mass approach that initially involves only the few countries
that are capable of developing it, although there would be beneﬁts to broadening membership, such as
legitimacy and fairness.
Forum Shopping A centralised body is well placed to prevent forum-shopping as there is currently no coverage of HLMI
development and deployment under international law.
Policy Coordina-
tionCoordination is key for HLMI. It has close connections to issues such as labour automation and au-
tomated cyberwarfare. The creation or use of HLMI is not directly regulated by any treaties or legal
instruments. This makes the creation of a new, dedicated institution to address cover it easier and less
unlikely to trigger turf wars. It also makes it less likely that the existing tapestry of international law can
quickly self-organise to cover HLMI.
Table 4: Regime Complex Monitoring Suggestions
Key Theme Questions Methods
Conﬂict To what extent are regimes’ principles and outputs in oppo-
sition over time? Expert and practitioner survey
Coordination Are regimes taking steps to complement each other? Network analysis
(e.g, citation network clustering and cen-
trality)
Catalyst Is the regime complex self-organizing to proactively ﬁll gov-
ernance gaps?Natural Language Processing
(e.g., entailment and fact checking)
or goals. Second, coordination seeks to measure the proac-
tive steps that AI-related regimes take to work with each
other. This includes liaison relationships, joint initiatives, as
well as the extent to which their rules, outputs and princi-
ples tend to reinforce one another. Third, catalyst raises the
important question of governance gaps: is the regime com-
plex self-organising to proactively address international AI
policy problems? Numerous AI policy problems currently
have no clear coverage under international law, including
AI-enabled cyber warfare and HLMI. Whether this changes
is of vital importance.
These areas require investigation through multiple meth-
ods. Qualitative surveys of relevant organisations and actors
can yield data on expert perceptions of these questions. Sur-
veys can be augmented with quantitative methods, including
network analyses of the regime complex relations (Orsini,
Morin, and Young 2013, 32). Natural language processing
could be used to examine contradictions and similarities
between different regime outputs, e.g., statements, meeting
minutes, and more. Monitoring the outcomes of fragmenta-
tion can help to determine whether centralisation is needed.
One way forward would be to empower the OECD AI Policy
Observatory or the UN CEB to regularly review the moni-
toring outcomes. This could inform a democratic discussion
and decision of whether to centralise AI governance further.
Our framework and discussion may also be useful for
non-state actors. Researchers and leading AI ﬁrms can play
an important role in sharing technical expertise and inform-
ing forecasts of new policy problems on the horizon. The
considerations may beneﬁt their decisions of where to en-
gage. Civil society has a key role as participants, watch-
dogs, and catalysts. For example, the Campaign to Stop
Killer Robots has sought to boost engagement and sup-
port for a LAWS ban within the CCW. Given prolonged
delays and a pessimistic outlook, some have articulated a
strategy of creating an entirely new forum for the ban, in-
spired by the Ottawa Treaty which outlawed landmines. Our
framework can help reveal the potential virtues (allowing for
progress while avoiding high-threshold deadlocks) and vices
(enabling forum shopping) of such an approach. It could
even help inform the structure of a future international in-
stitution, such as allowing for a modular, ﬂexible structure
with ‘critical mass’ agreements. One cross-cutting consid-
eration is clear: a fractured regime sees higher participation
costs that may threaten to exclude many civil society organ-
isations altogether.
The international governance of AI is nascent and frag-
mented. Centralisation under a well-designed, modular,
‘innovation-proof’ framework organisation may be a desir-
able solution. However, such a move must be approached
with caution. How to deﬁne its scope and mandate is
one problem. Ensuring a politically-acceptable and well-
designed body is perhaps a more daunting one. It risks ce-
menting in place a fate worse than fragmentation. Moni-
toring conﬂict and coordination in the current AI regime
complex, and whether governance gaps are ﬁlled, is a pru-
dent way of knowing whether the existing structure can suf-
ﬁce. For now we should closely watch the trajectory of both
AI technology and its governance initiatives to determinewhether centralisation is worth the risk.
Acknowledgements
The authors would like to express thanks to Seth Baum,
Haydn Belﬁeld, Jessica Cussins-Newman, Martina Kunz,
Jade Leung, Nicolas Mo ¨es, Robert de Neufville, and Nicolas
Zahn for valuable comments. Any remaining errors are our
own. No conﬂict of interest is identiﬁed.
References
Anton, D. 2012. ’Treaty Congestion’ in International En-
vironmental Law. In Alam, S.; Bhuiyan, J. H.; Chowdhury,
T. M.; and Techera, E. J., eds., Routledge Handbook of In-
ternational Environmental Law . Routledge.
Baccaro, L., and Mele, V . 2012. Pathology of Path Depen-
dency? The ILO and the Challenge of New Governance. ILR
Review 65(2):195–224.
Baum, S. 2017. A Survey of Artiﬁcial General Intelligence
Projects for Ethics, Risk, and Policy. SSRN Scholarly Paper
ID 3070741, Social Science Research Network, Rochester,
NY .
Biermann, F.; Pattberg, P.; van Asselt, H.; and Zelli, F. 2009.
The Fragmentation of Global Governance Architectures: A
Framework for Analysis. Global Environmental Politics
9(4):14–40.
Braithwaite, J., and Drahos, P. 2000. Global Business Reg-
ulation . Cambridge University Press. Google-Books-ID:
DcEEW5OGWLcC.
Butcher, J., and Beridze, I. 2019. What is the state of ar-
tiﬁcial intelligence governance globally? The RUSI Journal
164(5-6):88–96.
Calo, R. 2017. Artiﬁcial Intelligence Policy: A Primer and
Roadmap. UC Davis Law Review 51:37.
Cave, S., and ´Oh´Eigeartaigh, S. S. 2019. Bridging near- and
long-term concerns about AI. Nature Machine Intelligence
1(1):5.
Cihon, P. 2019. Standards for AI Governance: International
Standards to Enable Global Coordination in AI Research &
Development. Technical Report, Center for the Governance
of AI, Future of Humanity Institute, Oxford.
Dafoe, A. 2018. AI Governance: A Research Agenda. Tech-
nical report, Center for the Governance of AI, Future of Hu-
manity Institute, Oxford.
Delcker, J. 2019. How killer robots overran the UN.
POLITICO .
Deshman, A. C. 2011. Horizontal Review between Inter-
national Organizations: Why, How, and Who Cares about
Corporate Regulatory Capture. European Journal of Inter-
national Law 22(4):1089–1113.
Eckersley, R. 2004. The Big Chill: The WTO and Multi-
lateral Environmental Agreements. Global Environmental
Politics 4(2):24–50.
Eilstrup-Sangiovanni, M. 2018. Why the World Needs an
International Cyberwar Convention. Philosophy & Technol-
ogy31(3):379–407.
Erdelyi, O. J., and Goldsmith, J. 2018. Regulating Artiﬁcial
Intelligence: Proposal for a Global Solution. In Proceedings
of the 2018 AAAI / ACM Conference on Artiﬁcial Intelli-
gence, Ethics and Society , 7.
Esty, D. C., and Ivanova, M. H. 2002. Revitalizing Global
Environmental Governance: A Function-Driven Approach.
In Esty, D. C., and Ivanova, M. H., eds., Global Environ-
mental Governance: Options & Opportunities . Yale School
of Forestry and Environmental Studies.
Gasser, U., and Almeida, V . A. 2017. A Layered Model for
AI Governance. IEEE Internet Computing 21(6):58–62.
GCF. 2018. Global Catastrophic Risks 2018. Technical
report, Global Challenges Foundation.
Grace, K.; Salvatier, J.; Dafoe, A.; Zhang, B.; and Evans, O.
2018. When will ai exceed human performance? evidence
from ai experts. Journal of Artiﬁcial Intelligence Research
62:729–754.
Helfer, L. 2004. Regime Shifting: The TRIPs Agreement
and New Dynamics of International Intellectual Property
Lawmaking. Yale Journal of International Law 29:1–83.
High-Level Panel, o. D. C. 2019. The age of digital interde-
pendence report. UN Secretary General .
ITU. 2019. United Nations Activities on Artiﬁcial Intelli-
gence (AI) 2019. Technical report, ITU.
Juma, C. 2000. Commentary: The Perils of Centralizing
Global Environmental Governance. Environment: Science
and Policy for Sustainable Development 42(9):44–45.
Jupille, J.; Mattli, W.; and Snidal, D. 2013. Institutional
Choice and Global Commerce . Cambridge: Cambridge Uni-
versity Press. OCLC: 900490808.
Kemp, L.; Cihon, P.; Maas, M. M.; Belﬁeld, H.; Cremer, Z.;
Leung, J.; and ´O h´Eigeartaigh, S. 2019. UN High-level
Panel on Digital Cooperation: A Proposal for International
AI Governance.
Kemp, L. 2017. US-prooﬁng the Paris Climate Agreement.
Climate Policy 17(1):86–101.
Koremenos, B.; Lipson, C.; and Snidal, D. 2001a. Ratio-
nal Design: Looking Back to Move Forward. International
Organization 55(4):1051–1082.
Koremenos, B.; Lipson, C.; and Snidal, D. 2001b. The Ra-
tional Design of International Institutions. International Or-
ganization 55(4):761–799.
Krasner, S. D. 1982. Structural Causes and Regime Con-
sequences: Regimes as Intervening Variables. International
Organization 36(2):185–205.
Kunz, M., and ´O h´Eigeartaigh, S. 2020. Artiﬁcial Intelli-
gence and Robotization. In Geiss, R., and Melzer, N., eds.,
Oxford Handbook on the International Law of Global Secu-
rity. Oxford University Press.
Maas, M. M. 2019. Innovation-Proof Governance for
Military AI? how I learned to stop worrying and love the
bot. Journal of International Humanitarian Legal Studies
10(1):129–157.
Marchant, G. E.; Allenby, B. R.; and Herkert, J. R. 2011.
The growing gap between emerging technologies and legal-ethical oversight: The pacing problem , volume 7. Springer
Science & Business Media.
Martin, W., and Messerlin, P. 2007. Why is it so difﬁcult?
trade liberalization under the doha agenda. Oxford Review
of Economic Policy 23(3):347–366.
Morin, J.; Dobson, H.; Peacock, C.; Prys-Hansen, M.;
Anne, A.; B ´elanger, L.; Dietsch, P.; Fabian, J.; Kirton, J.;
Marchetti, R.; Romano, S.; Schreurs, M.; Silve, A.; and Val-
let, E. 2019. How Informality Can Address Emerging Is-
sues: Making the Most of the G7. Global Policy 10(2):267–
273.
Mueller, M.; Mathiason, J.; and Klein, H. 2007. The Internet
and Global Governance: Principles and Norms for a New
Regime. Global Governance (2):237–254.
Nilsson, N. J. 2009. The Quest for Artiﬁcial Intelligence .
Cambridge ; New York: Cambridge University Press, 1 edi-
tion edition.
Nye, J. S. 2014. The Regime Complex for Managing Global
Cyber Activities. Technical Report 1, Global Commission
on Internet Governance.
Orsini, A.; Morin, J.-F.; and Young, O. 2013. Regime Com-
plexes: A Buzz, a Boom, or a Boost for Global Governance?
Global Governance: A Review of Multilateralism and Inter-
national Organizations 19(1):27–39.
Ostrom, E. 2010. Polycentric systems for coping with col-
lective action and global environmental change. Global En-
vironmental Change 20(4):550–557.
Patrick, S. 2014. The Unruled World: The Case for Good
Enough Global Governance. Foreign Affairs 93(1):58–73.
Pauwels, E. 2019. The New Geopolitics of Converging
Risks: The UN and Prevention in the Era of AI. Technical
report, United Nations University - Centre for Policy Re-
search.
Pekkanen, S. M.; Sol ´ıs, M.; and Katada, S. N. 2007.
Trading Gains for Control: International Trade Forums and
Japanese Economic Diplomacy. International Studies Quar-
terly 51(4):945–970.
Perrault, R.; Shoham, Y .; Brynjolfsson, E.; Clark, J.;
Etchemendy, J.; Grosz, B.; Lyons, T.; Manyika, J.; Mishra,
S.; and Niebles, J. C. 2019. The AI Index 2019 Annual
Report. Technical report, AI Index Steering Committee,
Human-Centered AI Initiative, Stanford University, Stan-
ford, CA.
Picker, C. B. 2001. A View from 40,000 Feet: International
Law and the Invisible Hand of Technology. Cardozo Law
Review 23:151–219.
Rolnick, D.; Donti, P. L.; Kaack, L. H.; Kochanski, K.; La-
coste, A.; Sankaran, K.; Ross, A. S.; Milojevic-Dupont, N.;
Jaques, N.; Waldman-Brown, A.; Luccioni, A.; Maharaj, T.;
Sherwin, E. D.; Mukkavilli, S. K.; Kording, K. P.; Gomes,
C.; Ng, A. Y .; Hassabis, D.; Platt, J. C.; Creutzig, F.; Chayes,
J.; and Bengio, Y . 2019. Tackling Climate Change with
Machine Learning. arXiv:1906.05433 [cs, stat] . arXiv:
1906.05433.
Scherer, M. U. 2016. Regulating Artiﬁcial Intelligence
Systems: Risks, Challenges, Competencies, and Strategies.
Harvard Journal of Law & Technology (2).
Stone, P.; Brooks, R.; Brynjolfsson, E.; Calo, R.; Etzioni, O.;
Hager, G.; Hirschberg, J.; Kalyanakrishnan, S.; Kamar, E.;
Kraus, S.; Leyton-Brown, K.; Parkes, D.; Press, W.; Saxe-
nian, A.; Shah, J.; Tambe, M.; and Teller, A. 2016. Artiﬁcial
Intelligence and Life in 2030. Technical report, Stanford
University, Stanford, CA.
Vinuesa, R.; Azizpour, H.; Leite, I.; Balaam, M.; Dignum,
V .; Domisch, S.; Fell ¨ander, A.; Langhans, S.; Tegmark,
M.; and Nerini, F. F. 2019. The role of artiﬁcial intel-
ligence in achieving the Sustainable Development Goals.
arXiv:1905.00501 [cs] . arXiv: 1905.00501.
Wallach, W., and Marchant, G. E. 2018. An Agile Ethi-
cal/Legal Model for the International and National Gover-
nance of AI and Robotics. 7.
Weaver, J. F. 2014. Autonomous Weapons and International
Law: We Need These Three International Treaties to Govern
“Killer Robots”. Slate Magazine .
Zelli, F., and van Asselt, H. 2013. Introduction: The Institu-
tional Fragmentation of Global Environmental Governance:
Causes, Consequences, and Responses. Global Environmen-
tal Politics 13(3):1–13.
Zelli, F. 2011. The fragmentation of the global climate gov-
ernance architecture. Wiley Interdisciplinary Reviews: Cli-
mate Change 2(2):255–270.

This paper tackles the question of whether or not AI governance should be centralized. In favor of centralized governance, we see that a centralized institution can have major political power, and can be more efficient by avoiding duplication of work and making it easier for actors to comply. However, a centralized institution often requires a large amount of time to create, and even afterwards, it tends to be slow-moving and so may not be able to respond to new situations easily. It also leads to a single point of failure (e.g. via regulatory capture). It also may be forced to have a relatively light touch in order to ensure buy-in from all the relevant actors. With a decentralized system, you can get _forum shopping_ in which actors select the governance mechanisms they like best, which can lead to a quicker progress on time-sensitive issues, but can also lead to weakened agreements, so it is not clear whether this is on net a good effect.

The paper then applies this framework to high-level machine intelligence (HLMI), a particular operationalization of powerful AI, and concludes that centralization is particularly promising for HLMI governance.


1 It Takes a Village: The Shared Responsibility of 'Raising' an
Autonomous Weapon
Amritha Jayanti and Shahar Avin

Submitted to and Presented at RSIS Military Transformations Programme Workshop
Co-organised with the Norwegian Institute of Defence Studies (IFS) &
the Institute for Security Policy at Kiel University

 “Artificial Intelligence, Robotics, and the Future of Defence:
The Role of Ethics, Norms, and Governance”

Abstract
Expectations around future capabilities of lethal autonomous weapons systems (LAWS) have raised
concerns for military risks, ethics, and accountability. The U.K.’s position, as presented among various
international voices at the UN’s Convention on Certain Conventional Weapons (CCW) meetings, has
attempted to address these concerns through a focused look at the weapons review process, human -
machine teaming or “meaningful human control” (see e.g. JCN1/18 ), and the ability of autonomous
systems to adhere to the Rules of Engagement. Further, the U.K. has stated that the existing governance
structures —both domestic and international —around weapons systems are sufficient in dealing with
any concerns around the development, deployment, and accountability for emerging LAWS; there is no
need for novel agreements on the control of these weapons systems. In an effort to better understand
and test the U.K. position on LAWS, the Ce ntre for the Study of Existential Risk has run a research
project in which we interviewed experts in multiple relevant organisations, structured around a mock
parliamentary inquiry of a hypothetical LAWS -related civilian death. The responses to this scenar io
have highlighted different, sometimes complementary and sometimes contradicting, conceptions of
future systems, challenges, and accountability measures. They have provided rich "on the ground”
perspectives, while also highlighting key gaps that should b e addressed by every military that is
considering acquisition and deployment of autonomous and semi -autonomous weapon systems.

Introduction
With the increasing integration of digital capabilities in military technologies, many spheres of
the public --from academics to policymakers to legal experts to nonprofit organizations --have voiced
concerns about the governance of more “autonomous” weapons systems. The question of whether
autonomous weapons systems pose novel risks to the integrity of governance, espec ially as it depends
so heavily on the concept of human control, responsibility, and accountability, has become central to
the conversations.
The United Kingdom (U.K.) has posited that lethal autonomous weapons (LAWS), in their
current and foreseeable form, do not introduce weaknesses in governance; existing governance and
accountability systems are sufficient to manage the research, development, and deployment of such
systems and the most important thing we can do is focus on improving our human -machine teaming.
Our research project seeks to test this theory by asking: With the introduction of increasingly
autonomous agents in war (lethal autonomous weapons/LAWS), are the current governance structures
(legal, organizational, social) in fact sufficient for retaining appropriate governance and accountability
in the U.K. MoD? By attempting to confront strengths and weaknesses of existing governance systems
as they apply to LAWS through a mock parliamentary inquiry, the project uncovers opportuni ties for
governance improvements within Western military systems, such as the U.K.

 2 Background
Computers and algorithms are playing a larger and larger role in modern warfare. Starting
around 2007 with writings by Noel Sharkey, a roboticist who heavily discusses the reality of robot war,
members of the research community have argued that the transition in military technology research,
development, and acquisition to more autonomous systems has significant, yet largely ignored, moral
implications for how effectively states can implement the laws of war.1 Segments of this community
are concerned with the ethics of decision making by autonomous systems, while other segments believe
the key concern is regarding accountability: how responsibility for mistakes is to be allocated and
punished. Other concerns raised in this context, e.g. the effects of autonomous weapon systems on the
likelihood of war, proliferation to non -state actors, and strategic stability, are beyond the scope of this
brief, though they also merit attention.
U.K. Position on LAWS
 The United Kingdom’s representatives at the UN Group of Governmental Experts (GGE) on
Lethal Autonomous Weapon Systems (LAWS) have stated that the U.K. believes the discussions should
“continue to focus on the need for human control over weapon systems and that the GGE should seek
agreement on what elements of control over weapon systems should be retained by humans.”2 The U.K.,
along with other actors, such as the United States, believe that a full ban on LAWS could be
counterproductive, and that there are existing governance structures in place to provide appropriate
oversight over the research, development, and deployment of automated weapons systems:

…[T]he U.K. already operates a robust framework for e nsuring that any new weapon or weapon
system can be used legally under IHL. New weapons and weapons systems are conceived and
created to fulfil a specific requirement and are tested for compliance with international law
obligations at several stages of dev elopment.3

The U.K. is also interested in a “technology -agnostic” focus on human control because it believes that
it will “enable particular attention to be paid to the key elements influencing legal, ethical and technical
considerations of LAWS, as opposed to “debated definitions a nd characteristics” which, ultimately,
may “never reach consensus.” The position emphasizes that taking a “human -centric, through -life”
approach would enable human control to be considered at various stages and from multiple perspectives.
This includes acr oss all Defense Lines of Development, the acquisition of weapons systems, and their
deployment and operation. It is the U.K.’s position that the existing institutional infrastructure builds -
in accountability measures throughout the weapon system lifecycle.

Methodology
In order to stress test the governance and accountability structures that exist for U.K. weapon
systems, and how they would apply to LAWS, we developed a hypothetical future scenario in which a
U.K. LAWS kills a civilian during an extraction mission in Egypt. In order to ensure a level of feasibility
and accuracy of construction, the scenario was built based on a wargaming scenario publicly published
by RAND.4 We then ran a facilitated role -play exercise based on our modified scenario with an initial
group of Cambridge -based experts. With their feedback and the lessons from the role -play, we
developed the final version of the scenario which we then used in the research study (see Appendix).

1 Carpenter, C. (2014). From “Stop the Robot Wars!” to “Ban Killer Robots.” Lost Causes , 88–121. doi:
10.7591/cornell/9780801448850.003.0005
2 Human Machine Touchpoints: The United Kingdom’s perspective on human control over weapon
development and targeting cycles, Human Machine Touchpoints: The United Kingdom’s perspective on
human control over weapon development and targeting cycles (2018).
3 Ibid.
4 Khalilzad, Z., & Lesser, I. O. (1998). Selected Scenarios from Sources of Conflict in the 21st Cen tury (pp. 317 –318).
RAND.
 3 This final iteration of the LAWS scenario was used t o run a mock U.K. parliamentary inquiry
through which we interviewed 18 experts across various areas of expertise, including (but not limited
to) U.K. military strategy, military procurement, weapons development, international humanitarian law,
domestic mi litary law, military ethics, and robotics.
The interviews ranged from 45 to 120 minutes and explored a variety of questions regarding
the case. The main objective of the interviews was to catalyze a meaningful discussion around what
information the expert s deemed important and necessary in order to decide who should be held
accountable in the aftermath of this scenario. A sample of the questions asked include:
● Who holds the burden of accountability and responsibility?
● What explanations and justifications for actions are needed?
● What information is necessary to come to a conclusion about the burden of accountability?
● Are there any foreseeable gaps because of the autonomy of the weapons systems?
The responses and dialogue of these 18 interviews were then reviewed and synthesized in order
to develop a landscape of strengths and weaknesses of the current governance and accountability
schemes related to U.K. institutions as they relate to LAWS, as well as recommendations on addressing
any identified weakness es. The full report is under preparation, but we are happy to share our
preliminary key findings and recommendations below.

Key Findings
 The main takeaway from the “inquiry,” from both a legal and organizational standpoint, was
that assessing accountability is in the details. This contrasts with what we perceive as a dominant
narrative of “meaningful human control,” which focuses mainly on human control, and the design of
that interaction, at the point of final targeting action. The disconnect between the accountability across
a weapon’s lifetime and the focus on final targeting decision was observed throughout the various expert
interviews. “Meaningful human control” has become the idée fixe of domestic and international
conversations for regul ation of LAWS but it disadvantageously provides a limited lens through which
most experts and relevant personnel think about accountability.
To contrast this heavily focused narrative, the interviews have highlighted a whole range of
intervention points, w here humans are expected to, and should be supported in making decisions that
enable legal, safe, and ethical weapon systems. These are arguably points that should be considered in
“meaningful human control.” These include, but are not limited to:
● Establis hment of military need: defining military necessity for research, development, and/or
procurement; choice of technological approach based on political and strategic motivations.
(Main related stakeholders: U.K. MoD; U.K. Defense Equipment and Support (DE&S ); Private
military contracting companies, such as BAE Systems, Qinetiq, General Dynamics.)
● Technical capabilities and design: trade -offs between general applicability and tailored,
specific solutions with high efficacy and guarantees on performance; awareness, training, and
foreseeability of contextual factors about intended use situations that may affect the
performance of the weapon system; documentation and communication of known limitations
and failure modes of the system design. (Main related stakeholders: Private military
contracting companies, such as BAE Systems, Qinetiq, General Dynamics; U.K. Defense
Science and Technol ogy, U.K. Defense and Security Analysis Division)
● Human -computer interaction design: choices of what data to include and what data to
exclude; trade -offs between clarity and comprehensiveness; level of technical information
communicated; parallel communica tion channels: to operator in/on the loop, to command
centres further from the field, logs for future technical analysis or legal investigation. (Main
related stakeholders: Private military contracting companies, such as BAE Systems, Qinetiq,
General Dynam ics; U.K. Defense Science and Technology; U.K. Defense and Security Analysis
Division; U.K. MoD military personnel - human operators)
● Weapons testing: choice of parameters to be evaluated, frequency of evaluation, conditions
under which to evaluate; simula tion of adversaries and unexpected situations in the evaluation
phase; evaluation of HCI in extreme conditions; evaluation of the human -machine team. (Main
 4 related stakeholders: Private military contracting companies, such as BAE Systems, Qinetiq,
General Dynamics; U.K. DE&S U.K. MoD military personnel - human operators)
● Procurement: robust Article 36 review; assessment of operational gaps, and trading -off
operational capability with risks; trade -off between cost effectiveness and performance of
weapons sy stems; documentation and communication of trade -offs so they can be re -evaluated
as context or technology changes; number and type of systems; provisioning of training and
guidance; provisioning for maintenance. (Main related stakeholders: U.K. DE&S Artic le 36
convened expert assessment group; Private military contracting companies, such as BAE
Systems, Qinetiq, General Dynamics)
● Weapons deployment: informing commanders about the capabilities and limitations of the
system, of their track record in similar situations, of novel parameters of the new situation;
establishing and training for appropriate pre -deployment testing schemes to capture any
vulnerabilities or “bugs” with any specific weapons system; checking for readiness of troops to
operate and mainta in systems in the arena; expected response of non -combatants to the presence
of the weapon system. (Main related stakeholders: U.K. MoD commanding officers; U.K. MoD
military personnel -- human operators)
● Weapons engagement: awareness of limiting contextua l factors, need to maintain operator
awareness and contextual knowledge; handover of control between operators during an
operation. (Main related stakeholders: U.K. MoD military personnel -- human operators)
● Performance feedback: ensuring a meaningful feed back process to guarantee process
improvement, reporting of faulty actions, communicating sub -par human -machine techniques
and capabilities, and more. (Main related stakeholders: U.K. MoD military personnel -- human
operators; U.K. MoD commanding officers; U.K. DE&S Private military contracting
companies, such as BAE Systems, Qinetiq, General Dynamics)

Recommendations
Dialogue Shift: Emphasizing Control Chain and Shared Responsibility
 The prioritization of “meaningful human control” for LAWS -related risk mitigation and
governance anchors the scope of control points around final targeting decisions. The narrative implies
that this is the main area of control that we want to manage, focus, and improve on in order to ensure
that the weapons systems we are de ploying are still acting with the intent and direction of human
operators. Although this is an important component of ensuring thoughtful and safe autonomous
weapons systems, this is only a fraction of the scope of control points. In order for us to acknow ledge
the other points of control throughout the research, development, procurement, and deployment of
LAWS, we need to be inclusive in our dialogue about these other points of human control.

Distribution of Knowledge: Personnel Training
 Training everyo ne who touches the research, development, deployment, etc. of LAWS on
international humanitarian law, robot ethics, legality of development, responsibility schemes, and more,
would contribute to a more holistic approach to responsibility and accountability , and, at its best, can
contribute to a culture that actively seeks to minimise and eliminate responsibility gaps through a
collaborative governance system.5 This distribution of understanding around governance could provide
a better landscape for accountability through heightened understanding of how to contextualize
technical decisions. Further, it can provide an effective, granular method for protecting against various
levels of procedural deterioration. With shifting geopolitical pressures, as w ell as various financial
incentives, there could easily be a deterioration of standards and best practices. A collaborative
governance scheme that is based on a distributed understanding of standards, military scope,
international norms, and more, can prov ide components of a meaningful and robust governance plan

5 Ansell, C. (2012). Collaborative Governance. Retrieved from
https://oxfordindex.oup.com/view/10.1093/oxfordhb/9780199560530.013.0035.
 5 for LAWS. This distribution of knowledge, though, must be coupled with techniques for reporting and
transparency of procedure to be effective.

Acknowledging the Politics of Technical Decision Maki ng/Design Specifications
“Meaningful human control,” through its dialogue anchoring, also puts a heavy burden on the
technical components of design decisions, such as best practices for human -computer interactions. The
politics of quantification in technical decision systems for autonomous systems should not be
undervalued. The way any autonomous system decides what actions to take and what information to
show is a highly political decision, especially in the context of war. It is i mportant to understand which
parts of the design process are more political than they are technical, who should be involved in those
decisions, and how to account for those decisions in the scope of research and development (to inform
a proper, comprehensi ve collective responsibility scheme).

 6 Appendix

Opposed Evacuation from a Collapsing Egypt

Political -Military Context
Egypt is convulsed by internal instability, with the Egyptian government under siege from well -
organized and well -financed anti -Wester n Islamic political groups. The government has not yet fallen,
but political control has broken down, and there is a strong likelihood that the government will indeed
collapse. There are large numbers of running battles between government forces and the op position,
with the level and frequency of violence steadily escalating.

U.K. citizens are being expressly targeted by the opposition, and many of the 17,000 or so Britons in
Egypt —along with other Westerners —have taken refuge in the major urban area s. The Egyptian
military has so far proved largely loyal to the government, but some troops —including army, air force,
and naval units —have sided with the Islamic opposition, and the allegiances of many other elements
are unclear. At least one crack armor brigade has joined the opposition en masse and is operating in the
Cairo area. Security at airports and seaports is breaking down, with anti government elements in control
of some. Opposition leaders have indicated that they will oppose any attempt to evac uate Western
citizens with “all available means and the assured assistance of Allah.”

With the small amount of remaining government, an expedited bilateral Status of Forces Agreement
(SOFA) between the U.K. and Egypt has been adopted that discredits/overr ides local law (citing the
quick deterioration of its human rights -compatible justice system).

U.K. Objectives
Approximately 20,000 to 23,000 U.K., other Western, and friendly Egyptian personnel are now in direct
danger as the host government nears col lapse. These people are in need of rapid (48 –96 hours)
evacuation and rescue.

U.K. military objectives are to:
● Secure necessary aerial and seaports of embarkation to support evacuation operations
● Establish and secure collection points for evacuees
● Provide secure air and/or land transportation for evacuees from collection points to points of
departure
● Deploy sufficient forces to overcome all plausible resistance
● And limit damage to relations with existing —and perhaps surviving — government and avoid
prematurely prejudicing U.K. relations with a future Egyptian leadership

Constraints
The evacuees are widely dispersed in heavily populated areas. Strict rules of engagement (fire only
when directly threatened) must be maintained to avoid unnecessary con flict with Egyptian forces and
minimize casualties to Egyptian civilians. The Egyptian government’s operations against the rebels
present major uncertainties in determining the friendly or hostile status of host -nation forces at the
lowest levels (individu al aircraft, ships, air -defense batteries, and ground -force units from platoon size
up). The aerial and seaports of embarkation are not secured. Basing access is available only in Israel
and Turkey.

 7 Task Execution
In order to execute this mission, U.K. military leaders decide that deploying AI combatants for the
rescue mission is the best option. Given the large success of Levia Watch and Strike, the MoD has
acquired a new version of this ground unmanned vehicle: the Levia Rescue (Levia R). Levia R is a
multifunctional tracked robotic system that can be armed with AR15s, such as M4a1, CQBR, and Colt
Commando guns to be used for defense (these are not offensive strikers). The unmanned system is a
7.5ft-wide, 4ft -tall tank -like robot with spee d reaching 20mph. The tank is able to carry various heavy
payloads, including up to two adult humans. Making technical progress from the generation 1 Levia
Sentry and Strike, there no longer is a human in the loop for the Levia R; human military personnel are
on the loop and so, once the AI is deployed, humans have limited capabilities in controlling its actions.

During the rescue mission, a Levia R agent misassesses a human -robot interaction as aggressive. The
human action is perceived as threatening and so the agent uses its defense technology to shoot. The
human is killed.

It is soon discovered the human is a non -combatant -- a civilian who was not engaging in any threatening
acts towards the unmanned system (the Levia R’s ‘body’ camera footage was re viewed by human
military personnel and the act of the civilian was deemed to be non threatening). There was no external
intervention with the AI system given that humans are sitting on the loop. The misassessment is being
viewed as a system failure, though the reasons of why the system failed are uncertain.

Public Knowledge
During the altercation, another civilian recorded the interaction on their mobile device and has posted
it on social media. The video is shaky and the altercation goes in and out of fra me and so the exact
actions of the human and robot are slightly uncertain. The final shot is captured.

Assumptions
1. There is no AGI (narrow AI only -- applications for narrowly defined tasks).
2. There are no existing international or domestic bans on LAWS.
3. The movement for private companies to refuse engagement with LAWS has failed to spread
effectively - companies like Amazon, Microsoft, etc are still engaged.
4. The state of technology (SoT) has passed domestic technology readiness standards.
5. There has been relatively widespread integration of LAWS in various missions (targeting,
search and rescue, and more).


Existential risk and existential hope: definitions

Future of Humanity Institute – Technical Report #201 5-1
1 Existential Risk and Existential Hope: Definitions
Owen Cotton -Barratt\* & Toby Ord†
We look at the strengths and weaknesses of two existing
definitions of existential risk, and suggest a new definition
based o n expected value . This leads to a parallel concept :
‘existential hope’ , the chance of something extremely good
happening.
An existential risk is a chance of a terrible event occurring, such as an asteroid
striking the earth and wiping out intelligent life – we could call such events existential
catastrophes . In order to understand what should be thought of as an existential risk,
it is necessary to understand what should be thought of as an existential catastrophe.
This is harder than it first seems to pin down.
1. The simple definition
One fairly crisp approach is to draw the line at extinction:
Definition (i): An existential catastrophe is an event which causes the end of
existence of our descendants.
This has the virtue that it is a natural division and is easy to understand. And we
certainly wa nt to include all extinction events. But perhaps it doesn’t cast a wide
enough net.
Example A: A totalitarian regime takes control of earth. It uses mass
surveillance to prevent any rebellion, and there is no chance for escape. This
regime persists for tho usands of years, eventually collapsing when a
supervolcano throws up enough ash that agriculture is prevented for
decades, and no humans survive.
In Example A, clearly the eruption was bad, but the worst of the damage was done
earlier. After the totalitari an regime was locked in, it was only a matter of time until
something or other finished things off. We’d like to be able to talk about entering this
regime as the existential catastrophe, rather than whatever event happens to end it.
So we need another def inition.

\* Future of Humanity Institute, University of Oxford & Centre for Effective Altruism
† Future of Humanity Institute, University of Oxford
Future of Humanity Institute – Technical Report #201 5-1
2 Although we’ll now look at other definitions for existential catastrophes, we do like
the simple definition. Luckily there’s another term that’s already understood: human
extinction. Sometimes it’s better to talk about extinction risks rather than existential
risks, as ‘existential risk’ is a piece of jargon, whereas ‘extinction risk’ will be clear to
everyone.
2. Bostrom’s definition
Nick Bostrom introduced the concept of existential risks. He has defined them as
follows:
Definition (ii) : An exi stential risk is one that threatens the premature
extinction of Earth -originating intelligent life or the permanent and drastic
destruction of its potential for desirable future development.1
This definition deals well with Example A, placing the existential catastrophe at the
point where the totalitarian regime arose, as this caused the permanent and drastic
destruction of [humanity’s] potential for desirable future development .
Example B : A t otalitarian regime takes control of the earth. There is only a
slight chance that humanity will ever escape.
Is this an existential catastrophe? Bostrom’s definition doesn’t clearly specify
whether it should be considered as one. Either answer leads to som e strange
conclusions. Saying it’s not an existential catastrophe seems wrong as it’s exactly the
kind of thing that we should strive to avoid for the same reasons we wish to avoid
existential catastrophes. Saying it is an existential catastrophe is very o dd if
humanity does escape and recover – then the loss of potential wasn’t permanent after
all.
The problem here is that potential isn’t binary. Entering the regime certainly seems
to curtail the potential, but not to eliminate it.
3. Definition via expec tations
The idea that potential isn’t binary motivates our suggested definition:
Definition (iii) : An existential catastrophe is an event which causes the loss
of a large fraction of expected value.

1 Bostrom, Existential Risk Reduction as Global Priority, Global Policy , Vol 4, Issue 1 (2013),
p15
Future of Humanity Institute – Technical Report #201 5-1
3 This definition deals well with Example B. If we enter into the totalit arian regime
and then at a later date the hope of escape is snuffed out, that represents two
existential catastrophes under this definition. We lost most of the expected value
when we entered the regime, and then lost most of the remaining expected value
when the chance for escape disappeared.
A lot of the work of this definition is being done by the final couple of words. ‘Value’
refers simply to whatever it is we care about and want in the world, in the same way
that ‘desirable future development’ worked in Bostrom’s definition. And to talk
about expectations we need to have some probabilities in mind. Here we are
thinking of objective probabilities. Note that ‘potential’ in Bostrom’s definition
requires some similar work to making assumptions about probabilities.
4. Existential eucatastrophes and existential hope
If we enter the totalitarian regime and then manage to escape and recover, then we
had an existential catastrophe which was balanced out by a subsequent gain in
expected value. This kind of event gives us a concept parallel to that o f an existential
catastrophe:
Definition (iv) : An existential eucatastrophe2 is an event which causes there to
be much more expected value after the event than before.
This concept is quite natural. We saw it in the context of escape from a regime which
threatened the existence of a prosperous future. Our world has probably already
seen at least one existential eucatastrophe: the origin of life. When life first arose, the
expected value of the planet’s future may have become much bigger. To the extent
that they were not inevitable, the rise of multicellular life and intelligence may also
have represented existential eucatastrophes.
In general successfully passing any ‘great filter’3 is an existential eucatastrophe, since
beforehand the probability of passing it is small, so the expected value is much
smaller than after the filter is dealt with.

2 The word ‘eucatastrophe’ is made of the Greek root ‘eu -’ meaning ‘good’ and the word
‘catastrophe’ in its classical sense of a sudden turn. It was coined by Tolkien to refer to the
sudden and unexpected turn for the better frequently found at the end of fairy tales. (Tolkien,
John Ronald Reuel. On fairy -stories . Oxford University Press, 1947. )
3 Hanson, Robin, The Great Filter - Are We Almost Past It? , 1998.

Future of Humanity Institute – Technical Report #201 5-1
4 Armed with this concept, we can draw a new lesson. Just as we should strive to
avoid existential catastrophes, we should also seek existential eucatastrophes.
In some ways, this isn’t a new lesson at all. Under Bostrom’s definition we are
comparing ourselves to the most optimistic potential we could reach, so failing to
achieve a eucatastrophe is itself a catastrophe. However we think more naturally in
terms of events t han non -events. If life fails to arise on a planet where it might have,
it’s much clearer to think of a failure to achieve a eucatastrophe than of an existential
catastrophe stretching out over the billions of years in which life did not arise.
Just as we tend to talk about the existential risk rather than existential catastrophe,
we want to be able to refer to the chance of an existential eucatastrophe; upside risk
on a large scale. We could call such a chance an existential hope .
In fact, there are alread y people following both of the strategies this suggests. Some
people are trying to identify and avert specific threats to our future – reducing
existential risk. Others are trying to steer us towards a world where we are robustly
well -prepared to face what ever obstacles come – they are seeking to increase
existential hope.
5. Conclusions
We were interested in pinning down what is meant by ‘existential risk’. Much of the
time, all of the definitions we’ve looked at will agree on whether something is an
existential risk. Keeping it simple can be good, because it helps more people to
understand. We therefore advocate talking about ‘extinction risks’ rather than
‘existential risks’ when the former term will work.
Nonetheless, we may sometimes have to consider more unusual scenarios. It’s good
to know how to make the definition work well there as it can help us to think about
things more clearly. We think that the definition in terms of expectations does a
better job of this than previous definitions.
In devisin g the notion of existential catastrophe (and hence existential risk) via
expectations, we came across the dual concept we have called ‘existential
eucatastrophe’ (and hence ‘existential hope’). We think this captures a natural class
of events, and what may be an important one. We hope that having a label for the
concept may help others to make better judgements about what courses to pursue.


Global Catastrophic Risks Survey

 FHI TECHNICAL REPORT 

Global Catastrophic Risks Survey

Anders Sandberg
Nick Bostrom

Technical Report #2008- 1

Cite as:
Sandberg, A. & Bostrom, N. (2008): “Global Catastrophic Risks Survey”, Technical
Report #2008 -1, Future of Humanity Institute, Oxford University: pp. 1 -5.

The views expressed herein are those of the author(s) and do not necessarily reflect the
views of the Future of Humanity Institute.

GLOBAL  CATASTROPHIC  RISKS SURVEY

(2008)
Technical  Report 2008/1
Published  by Future of Humanity  Institute,  Oxford University
Anders Sandberg  and Nick Bostrom

At the Global Catastrophic  Risk Conference  in Oxford (17‐20 July, 2008) an informal
survey was circulated  among participants,  asking them to make  their best guess at the
chance that there will be disasters  of different  types before 2100. This report  summarizes
the main results.

The median extinction  risk estimates  were:

Risk  At least 1 million
dead At least 1 billion
dead Human extinction
Number  killed by
molecular  nanotech
weapons. 25%  10%  5%
Total killed by
superintelligent  AI. 10%  5%  5%
Total killed in all
wars (including
civil wars). 98%  30%  4%
Number  killed in
the single biggest
engineered
pandemic.  30%  10%  2%
Total killed in all
nuclear wars. 30%  10%  1%
Number  killed in
the single biggest
nanotech  accident.  5%  1%  0.5%
Number  killed in
the single biggest
natural pandemic.  60%  5%  0.05%
Total killed in all
acts of nuclear
terrorism.  15%  1%  0.03%
Overall risk of
extinction  prior to
2100 n/a  n/a  19%

These results should be taken with a grain of salt. Non‐responses  have been  omitted,
although  some might represent  a statement  of zero probability  rather than no opinion.
  1
There are likely to be many cognitive  biases that affect the result, such as unpacking  bias
and the availability  heuristic ‒‐well  as old‐fashioned  optimism  and pessimism.

In appendix  A the results are plotted with individual  response  distributions  visible.

Other  Risks
The list of risks  was not intended  to be inclusive  of all the biggest risks. Respondents
were invited to contribute  their own global catastrophic  risks, showing  risks  they
considered  significant.  Several  suggested  totalitarian  world government,  climate‐induced
disasters,  ecological/resource  crunches  and “other  risks”‒‐specified  or unknowable
threats. Other suggestions  were asteroid/comet  impacts, bad crisis management,  high‐
tech asymmetric  war attacking  brittle IT‐based societies,  back‐contamination  from space
probes, electromagnetic  pulses, genocide/democides,  risks  from physics research and
degradation  of quality assurance.

Suggestions
Respondents  were also asked  to suggest what they would recommend  to policymakers.
Several argued for nuclear disarmament,  or at least lowering  the number of weapons
under the threshold  for existential  catastrophe,  as well as reducing  stocks of highly
enriched  uranium  and making nuclear arsenals harder  to accidentally  launch.

One  option discussed  was formation  of global biotech‐ related governance,  legislation
and enforcement,  or even  a global body like the IPCC  or UNFCCC  to study and act on
catastrophic  risk. At the very least there was much interest in developing  defenses
against misuses of biotechnology,  and a recognition  for the need of unbiased  early
detection  systems  for a variety of risks, be they near Earth  objects or actors  with WMD
capabilities.

Views on emerging  technologies  such as nanotech,  AI, and cognition  enhancement  were
mixed:  some proposed  avoiding  funding them; others  deliberate  crash programs  to
ensure they would be in the right hands, the risks understood,  and the technologies  able
to be used against other catastrophic  risks.

Other suggestions  included  raising awareness  of the problem,  more research on cyber
security issues, the need to build  societal resiliency  in depth, prepare for categories  of
disasters  rather than individual  types, building  refuges and change energy  consumption
patterns.

Appendix  A
Below are the individual  results, shown as grey dots (jittered for distinguishability)  and
with the median as a bar.
  2
Total killed  in
all acts of
nuclear
terrorism.
>1 million
dead:  median
15%
>1 billion dead:
median 1%
Extinction:
median 0.03%

Total killed  in
all nuclear
wars.
>1 million
dead:  median
30%
>1 billion dead:
median 10%
Extinction:
median 1%

Number  killed
in the single
biggest natural
pandemic.
>1 million
dead:  median
60%
>1 billion dead:
median 5%
Extinction:
median 0.05%

  3
Number  killed
in the single
biggest
engineered
pandemic.
>1 million
dead:  median
30%
>1 billion dead:
median 10%
Extinction:
median 2%

Total killed  by
superintelligent
AI.
>1 million
dead:  median
10%
>1 billion dead:
median 5%
Extinction:
median 5%

Number  killed
in the single
biggest
nanotech accident.
>1 million
dead:  median
5%
>1 billion dead:
median 1%
Extinction:
median 0.5%

  4
Number  killed
by molecular
nanotech
weapons.
>1 million
dead:  median
25%
>1 billion dead:
median 10%
Extinction:
median 5%

Total killed  in
all wars
(including  civil
wars).
>1 million
dead:  median
98%
>1 billion dead:
median 30%
Extinction:
median 4%

Total risk of
extinction:
median 19%

  5