The vulnerable world hypothesis

The Vulnerable World Hypothesis
Nick Bostrom
Future of Humanity Institute, University of Oxford
Abstract
Scienti ﬁc and technological progress might change people ’s capabilities or incentives in ways that would destabilize civiliza-
tion. For example, advances in DIY biohacking tools might make it easy for anybody with basic training in biology to kill mil-lions; novel military technologies could trigger arms races in which whoever strikes ﬁrst has a decisive advantage; or some
economically advantageous process may be invented that produces disastrous negative global externalities that are hard to
regulate. This paper introduces the concept of a vulnerable world : roughly, one in which there is some level of technological
development at which civilization almost certainly gets devastated by default, i.e. unless it has exited the ‘semi-anarchic
default condition ’. Several counterfactual historical and speculative future vulnerabilities are analyzed and arranged into a
typology. A general ability to stabilize a vulnerable world would require greatly ampli ﬁed capacities for preventive policing
and global governance. The vulnerable world hypothesis thus offers a new perspective from which to evaluate the risk-bene ﬁt
balance of developments towards ubiquitous surveillance or a unipolar world order.
Policy Implications
Technology policy should not unquestioningly assume that all technological progress is bene ﬁcial, or that complete scien-
tiﬁc openness is always best, or that the world has the capacity to manage any potential downside of a technology after
it is invented.Some areas, such as synthetic biology, could produce a discovery that suddenly democratizes mass destruction, e.g. by
empowering individuals to kill hundreds of millions of people using readily available materials. In order for civilization tohave a general capacity to deal with “black ball ”inventions of this type, it would need a system of ubiquitous real-time
worldwide surveillance. In some scenarios, such a system would need to be in place before the technology is invented.
Partial protection against a limited set of possible black balls is obtainable through more targeted interventions. For exam-
ple, biorisk might be mitigated by means of background checks and monitoring of personnel in some types of biolab, bydiscouraging DIY biohacking (e.g. through licencing requirements), and by restructuring the biotech sector to limit access
to some cutting-edge instrumentation and information. Rather than allow anybody to buy their own DNA synthesis
machine, DNA synthesis could be provided as a service by a small number of closely monitored providers.
Another, subtler, type of black ball would be one that strengthens incentives for harmful use —e.g. a military technology
that makes wars more destructive while giving a greater advantage to the side that strikes ﬁrst. Like a squirrel who uses
the times of plenty to store up nuts for the winter, we should use times of relative peace to build stronger mechanismsfor resolving international disputes.
Is there a black ball in the urn of possible
inventions?
One way of looking at human creativity is as a process of
pulling balls out of a giant urn.1The balls represent possible
ideas, discoveries, technological inventions. Over the courseof history, we have extracted a great many balls –mostly
white (bene ﬁcial) but also various shades of gray (moder-
ately harmful ones and mixed blessings). The cumulativeeffect on the human condition has so far been overwhelm-ingly positive, and may be much better still in the future(Bostrom, 2008). The global population has grown aboutthree orders of magnitude over the last ten thousand years,and in the last two centuries per capita income, standardsof living, and life expectancy have also risen.
2
What we haven ’t extracted, so far, is a black ball: a tech-
nology that invariably or by default destroys the civilizationthat invents it. The reason is not that we have beenparticularly careful or wise in our technology policy. We
have just been lucky.
It does not appear that any human civilization has been
destroyed –as opposed to transformed –by its own inven-
tions.
3We do have examples of civilizations being destroyed
by inventions made elsewhere. For example, the European
inventions that enabled transoceanic travel and force projec-tion could be regarded as a black-ball event for the indige-nous populations of the Americas, Australia, Tasmania, andsome other places. The extinction of archaic hominid popu-lations, such as the Neanderthals and the Denisovans, wasprobably facilitated by the technological superiority ofHomo sapiens. But thus far, it seems, we have seen no suf ﬁ-
ciently auto-destructive invention to count as a black ballfor humanity.
4
What if there is a black ball in the urn? If scienti ﬁc and
technological research continues, we will eventually reach it
and pull it out. Our civilization has a considerable ability to
Global Policy (2019) 10:4 doi: 10.1111/1758-5899.12718 ©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.
This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use,
distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.Global Policy Volume 10 . Issue 4 . November 2019455Research Article
pick up balls, but no ability to put them back into the urn.
We can invent but we cannot un-invent. Our strategy is tohope that there is no black ball.
This paper develops some concepts that can help us
think about the possibility of a technological black ball,and the different forms that such a phenomenon could
take. We also discuss some implications for policy from a
global perspective, particularly with respect to how oneshould view developments in mass surveillance and movestowards more effectual global governance or a moreunipolar world order. These implications by no means set-tle questions about the desirability of changes in thosemacrostrategic variables –for there indeed are other
strongly relevant factors, not covered here, which wouldneed to be added to the balance. Yet they form an impor-tant and under-appreciated set of considerations thatshould be taken into account in future debates on these
issues.
Before getting to the more conceptual parts of the paper,
it will be useful to paint a more concrete picture of what atechnological black ball could be like. The most obviouskind is a technology that would make it very easy tounleash an enormously powerful destructive force. Nuclearexplosions are the most obviously destructive force we havemastered. So let us consider what would have happened ifit had been very easy to unleash this force.
A thought experiment: easy nukes
On the morning of 12 September 1933, Leo Szilard was
reading the newspaper when he came upon a report of an
address recently delivered by the distinguished Lord Ruther-ford, now often considered the father of nuclear physics(Rhodes, 1986). In his speech, Rutherford had dismissed theidea of extracting useful energy from nuclear reactions as‘moonshine ’. This claim so annoyed Szilard that he went out
for a walk. During the walk, he got the idea of a nuclearchain reaction –the basis for both nuclear reactors and
nuclear bombs. Later investigations showed that making anatomic weapon requires several kilograms of plutonium orhighly enriched uranium, both of which are very dif ﬁcult
and expensive to produce. However, suppose it had turned
out otherwise: that there had been some really easy way tounleash the energy of the atom –say, by sending an electric
current through a metal object placed between two sheetsof glass.
So let us consider a counterfactual history in which Szilard
invents nuclear ﬁssion and realizes that a nuclear bomb
could be made with a piece of glass, a metal object, and abattery arranged in a particular con ﬁguration. What happens
next? Szilard becomes gravely concerned. He sees that hisdiscovery must be kept secret at all costs. But how? His
insight is bound to occur to others. He could talk to a few
of his physicist friends, the ones most likely to stumbleupon the idea, and try to persuade them not to publish any-thing on nuclear chain reactions or on any of the reasoningsteps leading up to the dangerous discovery. (That is whatSzilard did in actual history.)Here Szilard faces a dilemma: either he doesn ’t explain
the dangerous discovery, but then he will not be effectivein persuading many of his colleagues to stop publishing; orhe tells them the reason for his concern, but then hespreads the dangerous knowledge further. Either way he isﬁghting a losing battle. The general advance of scienti ﬁc
knowledge will eventually make the dangerous insight more
accessible. Soon, ﬁguring out how to initiate a nuclear chain
reaction with pieces of metal, glass, and electricity will nolonger take genius but will be within reach of any STEM stu-dent with an inventive mindset.
Let us roll the tape a little further. The situation looks
hopeless, but Szilard does not give up. He decides to take afriend into his con ﬁdence, a friend who is also the world ’s
most famous scientist –Albert Einstein. He successfully per-
suades Einstein of the danger (again following actual his-tory). Now, Szilard has the support of a man who can get
him a hearing with any government. The two write a letter
to President Franklin D. Roosevelt. After some committeewranglings and report-writing, the top levels of the US gov-ernment are eventually suf ﬁciently convinced to be ready to
take serious action.
What action can the United States take? Let us ﬁrst con-
sider what actually happened (Rhodes, 1986). What the USgovernment did, after having digested the information pro-vided by Einstein and Szilard, and after having receivedsome further nudging from the British who were also look-ing into the matter, was to launch the Manhattan Project in
order to weaponize nuclear ﬁssion as quickly as possible. As
soon as the bomb was ready, the US Air Force used it todestroy Japanese population centers. Many of the Manhat-tan scientists had justi ﬁed their participation by pointing to
the mortal danger that would arise if Nazi Germany got thebomb ﬁrst; but they continued working on the project after
Germany was defeated.
5Szilard advocated unsuccessfully
for demonstrating ‘the gadget ’over an unpopulated area
rather than in a city (Franck et al., 1945). After the warended, many of the scientists favored the international con-trol of atomic energy and became active in the nuclear dis-
armament movement; but their views carried little weight,
as nuclear policy had been taken out of their hands. Fouryears later, the Soviet Union detonated its own atomicbomb. The Soviet effort was aided by spies in the Manhat-tan Project, yet even without espionage it would have suc-ceeded within another year or two (Holloway, 1994). TheCold War followed, which at its peak saw 70,000 nuclearwarheads ready to unleash global destruction at a moment ’s
notice, with a trembling ﬁnger hovering over the ‘red but-
ton ’on either side (Norris and Kristensen, 2010).
6
Fortunately for human civilization, after the destruction of
Hiroshima and Nagasaki, no other atomic bomb has been
detonated in anger. Seventy-three years later, partly thanksto international treaties and anti-proliferation efforts, onlynine states possess nuclear weapons. No non-state actor isbelieved ever to have possessed nuclear weapons.
7
But how would things have played out if there had been
aneasy way to make nukes? Maybe Szilard and Einstein
could persuade the US government to ban all research in
©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd. Global Policy (2019) 10:4Nick Bostrom456
nuclear physics (outside high-security government facilities)?
Such a ban on basic science would be subjected to enor-mous legal and political challenges –the more so as the
reason for the ban could not be publicly disclosed in anydetail without creating an unacceptable information haz-ard.
8
Let us suppose, however, that President Roosevelt could
somehow mobilize enough political support to drivethrough a ban, and that the US Supreme Court could some-how ﬁnd a way of regarding it as constitutionally valid. We
then confront an array of formidable practical dif ﬁculties. All
university physics departments would have to be closed,and security checks initiated. A large number of faculty andstudents would be forced out. Intense speculations wouldswirl around the reason for all these heavy-handed mea-sures. Groups of physics PhD students and faculty bannedfrom their research ﬁeld would sit around and speculate
about what the secret danger might be. Some of them
would ﬁgure it out. And among those who ﬁgured it out,
some would feel compelled to use the knowledge toimpress their colleagues; and those colleagues would wantto tell yet others, to show they were in the know. Alterna-tively, somebody who opposed the ban would unilaterallydecide to publish the secret, maybe in order to supporttheir view that the ban is ineffective or that the bene ﬁts of
publication outweigh the risks.
91 0Careless or disgruntled
employees at the government labs would eventually also letslip information, and spies would carry the secret to foreign
capitals. Even if, by some miracle, the secret never leaked in
the United States, scientists in other countries would inde-pendently discover it, thereby multiplying the sources fromwhich it could spread. Sooner or later –probably sooner –
the secret would be a secret no more.
In the present age, when one can publish instantaneously
and anonymously on the Internet, it would be even moredifﬁcult to limit the spread of scienti ﬁc secrets (Cf. Green-
berg, 2012; Swire, 2015).
An alternative approach would be to eliminate all glass,
metal, or sources of electrical current (save perhaps in a few
highly guarded military depots). Given the ubiquity of these
materials, such an undertaking would be extremely daunt-ing. Securing political support for such measures would beno easier than shutting down physics education. However,after mushroom clouds had risen over a few cities, the polit-ical will to make the attempt could probably be mustered.Metal use is almost synonymous with civilization, and wouldnot be a realistic target for elimination. Glass productioncould be banned, and existing glass panes con ﬁscated; but
pieces of glass would remain scattered across the landscapefor a long time. Batteries and magnets could be seized,
though some people would have stashed away these mate-
rials before they could be collected by the authorities. Manycities would be destroyed by nihilists, extortionists, revan-chists, or even folk who just want to ‘see what would hap-
pen ’.
11People would ﬂee urban areas. In the end, many
places would be destroyed by nuclear fallout, cities wouldbe abandoned, there would be no use of electricity or glass.Possession of proscribed materials, or equipment that couldbe used to make them, would be harshly punished, such as
by on-the-spot execution. To enforce these provisions, com-munities would be subjected to strict surveillance –infor-
mant networks incentivized by big rewards, frequent policeraids into private quarters, continuous digital monitoring,and so forth.
That is the optimistic scenario. In a more pessimistic sce-
nario, law and order would break down entirely and soci-eties might split into factions waging civil wars with nuclearweapons, producing famine and pestilence. The disintegra-tion might end only when society has been so reduced thatnobody is able any longer to put together a bomb and adelay detonator from stored materials or the scrap of cityruins. Even then, the dangerous insight –once its impor-
tance had been so spectacularly demonstrated –would be
remembered and passed down the generations. If civiliza-tion began to rise from the ashes, the knowledge would lie
in wait, ready to pounce as soon as people learned once
again how to make sheet glass and electric current genera-tors. And even if the knowledge were forgotten, it would berediscovered once nuclear physics research was resumed.
We were lucky that making nukes turned out to be hard.
The vulnerable world hypothesis
We now know that one cannot trigger a nuclear explosionwith just a sheet of glass, some metal, and a battery. Makingan atomic bomb requires several kilograms of ﬁssile mate-
rial, which is dif ﬁcult to produce. We pulled out a gray ball
that time. Yet with each act of invention, we reach into the
urn anew.
Let us introduce the hypothesis that the urn of creativity
contains at least one black ball. We can refer to this as thevulnerable world hypothesis (VWH). Intuitively, the hypothesis
is that there is some level of technology at which civilizationalmost certainly gets destroyed unless quite extraordinaryand historically unprecedented degrees of preventive polic-ing and/or global governance are implemented. More pre-cisely:
VWH: If technological development continues then
a set of capabilities will at some point be attainedthat make the devastation of civilization extremelylikely, unless civilization suf ﬁciently exits the semi-
anarchic default condition.
By the ‘semi-anarchic default condition ’I mean a world
order characterized by three features
12:
1.Limited capacity for preventive policing . States do not have
sufﬁciently reliable means of real-time surveillance and
interception to make it virtually impossible for any indi-vidual or small group within their territory to carry outillegal actions –particularly actions that are very strongly
disfavored by >99 per cent of the population.
2.Limited capacity for global governance . There is no reliable
mechanism for solving global coordination problems andprotecting global commons –particularly in high-stakes
Global Policy (2019) 10:4 ©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.The Vulnerable World Hypothesis457
situations where vital national security interests are
involved.
3.Diverse motivations . There is a wide and recognizably
human distribution of motives represented by a largepopulation of actors (at both the individual and statelevel) –in particular, there are many actors motivated, to
a substantial degree, by perceived self-interest (e.g.
money, power, status, comfort and convenience) andthere are some actors ( ‘the apocalyptic residual ’) who
would act in ways that destroy civilization even at highcost to themselves.
3
The term ‘devastation of civilization ’in the above de ﬁni-
tion could be interpreted in various ways, yielding differ-ent versions of VWH. For example, one could de ﬁne an
existential-risk vulnerable world hypothesis (x-VWH), whichwould state that at some level of technology, by default,an existential catastrophe occurs, involving the extinctionof Earth-originating intelligent life or the permanentblighting of our future potential for realizing value. How-ever, here we will set the bar lower. A key concern in thepresent context is whether the consequences of civiliza-tion continuing in the current semi-anarchic default condi-
tion are catastrophic enough to outweigh reasonable
objections to the drastic developments that would berequired to exit this condition. If this is the criterion, thena threshold short of human extinction or existential catas-trophe would appear suf ﬁcient. For instance, even those
who are highly suspicious of government surveillancewould presumably favour a large increase in such surveil-lance ifit were truly necessary to prevent occasional
region-wide destruction. Similarly, individuals who valueliving in a sovereign state may reasonably prefer to liveunder a world government given the assumption that the
alternative would entail something as terrible as a nuclear
holocaust. Therefore, we stipulate that the term ‘civiliza-
tional devastation ’in VWH refers (except where otherwise
speci ﬁed) to any destructive event that is at least as bad
as the death of 15 per cent of the world population or areduction of global GDP by >50 per cent per cent lasting
for more than a decade.
13
It is not a primary purpose of this paper to argue that
VWH is true. (I regard that as an open question, though itwould seem to me unreasonable, given the available evi-dence, to be at all con ﬁdent that VWH is false .) Instead, the
chief contribution claimed here is that VWH, along with
related concepts and explanations, is useful in helping us
surface important considerations and possibilities regardinghumanity ’s macrostrategic situation. But those considera-
tions and possibilities need to be further analyzed, and com-bined with other considerations that lie outside the scopeof this paper, before they could deliver any de ﬁnitive policy
implications.
A few more clari ﬁcations before we move on. This paper
uses the word ‘technology ’in its broadest sense. Thus, in
principle, we count not only machines and physical devicesbut also other kinds of instrumentally ef ﬁcacious templates
and procedures –including scienti ﬁc ideas, institutionaldesigns, organizational techniques, ideologies, concepts, and
memes –as constituting potential technological black
balls.
14
We can speak of vulnerabilities opening and closing. In
the ‘easy nukes ’scenario, the period of vulnerability begins
when the easy way of producing nuclear explosions is dis-
covered. It ends when some level of technology is attained
that makes it reasonably affordable to stop nuclear explo-sions from causing unacceptable damage –or that again
makes it infeasible to produce nuclear explosions (becauseof technological regress).
15If no protective technology is
possible (as in, e.g., the case of nuclear weapons it may notbe) and technological regress does not occur, then theworld becomes permanently vulnerable.
We can also speak of the world being stabilized (with
respect to some vulnerability) if the semi-anarchic defaultcondition is exited in such a way as to prevent the vulnera-
bility from leading to an actual catastrophe. The ways in
which the semi-anarchic default condition would have to bealtered in order to achieve stabilization depend on the spe-ciﬁcs of the vulnerability in question. In a later section, we
will discuss possible means by which the world could bestabilized. For now, we simply note that VWH does notimply that civilization is doomed.
T ypology of vulnerabilities
We can identify four types of civilizational vulnerability.
T ype-1 ( ‘easy nukes ’)
The ﬁrst type is one where, as in the ‘easy nukes ’scenario,
it becomes too easy for individuals or small groups to cause
mass destruction:
Type-1 vulnerability : There is some technology
which is so destructive and so easy to use that,given the semi-anarchic default condition, theactions of actors in the apocalyptic residual make
civilizational devastation extremely likely.
Note that in determining whether a scenario presents a
Type-1 vulnerability, there is an inverse relationship betweenthe ease with which it becomes possible to cause an inci-
dent and the destructiveness of incident. The greater the
destructiveness of a single incident, the less easy it needs to
be to cause such an incident in order for us to diagnose thepresence of a Type-1 vulnerability.
Thus, consider a ‘very easy nukes ’scenario, in which any
halfwit can create an easily portable thermonuclear weaponat the kitchen sink over the course of an afternoon: thiswould de ﬁnitely qualify as a civilizational vulnerability. Con-
trast this with a ‘moderately easy nukes ’scenario, in which it
takes a ﬁve-person team of semi-skilled individuals toiling
for an entire year to produce a single bulky few-kilotondevice: that might not quite rise to the level of a civiliza-
tional vulnerability. It seems possible, in the ‘moderately
easy nukes ’scenario, that the great majority of cities would
escape destruction, although the threat posed by a well-
©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd. Global Policy (2019) 10:4Nick Bostrom458
resourced terrorist organization, such as Aum Shinrikyo anno
1995 or Al-Qaeda anno 2001, would increase substantially.However, consider yet another scenario, ‘moderately easy
bio-doom ’, in which again it requires a semi-skilled ﬁve-per-
son team working for a year to put the black-ball technol-ogy into effect, except that this time it is a biological agent,
a single point release of which is suf ﬁcient to kill billions. In
‘moderately easy bio-doom ’, the threshold for a Type-1 vul-
nerability would be reached. If destroying civilisation
required only that a single group succeed with a task at themoderately-easy level, civilization would probably bedestroyed within a few years in the semi-anarchic defaultcondition. Indeed, both Aum Shinrikyo and Al-Qaeda soughtto obtain nuclear and biological weapons, and would likelyhave chosen to use them (see e.g. Danzig et al., 2011; Olson,1999; Mowatt-Larssen and Allison, 2010).
So a Type-1 vulnerability exists if it is either extremely easy
to cause a moderate amount of harm or moderately easy to
cause an extreme amount of harm.
16The reason why a
black-ball technology that enables only moderate amounts ofharm per incident could count as a Type-1 vulnerability isthat –if the technology is suf ﬁciently easy to use –al a r g e
number of such incidents would be almost certain to occur.Take the scenario where it is easy for an average individualto make a metropolis-busting H-bomb. This is not necessarilya scenario in which a single individual could devastate civi-lization. Building hundreds of bombs and transporting themto hundreds of cities without getting caught would still be a
formidable endeavor even if making a single bomb were
fairly easy. The ‘easy nukes ’scenario nevertheless presents a
civilizational vulnerability because it is plausible that therewould in fact be hundreds of individuals who would eachdestroy at least one city under those circumstances.
That this is so almost follows from the law of large num-
bers combined with the plausible assumption that for anyrandomly selected person there is some small but apprecia-ble chance that they would be motivated to trigger thiskind of destruction –whether out of ideological hatred,
nihilistic destructiveness, revenge for perceived injustices, as
part of some extortion plot, or because of delusions or men-
tal illness, or perhaps even just to see what would happen.Given the diversity of human character and circumstance,for any ever so imprudent, immoral, or self-defeating action,there is some residual fraction of humans who wouldchoose to take that action. This is especially plausible if theaction in question represents a culturally salient affordance–as it everywhere would after one such nuke attack had
taken place. In other words, ‘easy nukes ’is an illustration of
a vulnerable world because it looks like the apocalypticresidual has a large enough intersection with the set of
empowered actors that one would expect a civilization-dev-
astating amount of destruction to result.
T ype-2a ( ‘safe ﬁrst strike ’)
A technology that ‘democratizes ’mass destruction is not the
only kind of black ball that could be hoisted out of the urn.Another kind would be a technology that strongly incen-
tivizes powerful actors to use their powers to cause massdestruction. Again we can turn to nuclear history for illustra-tion.
After the invention of the atomic bomb and a short-lived
American nuclear monopoly, an arms race ensued between
the US and the USSR. The rival superpowers amassed stag-
gering arsenals, topping out at 70,000 nuclear warheads in1986, more than enough to devastate civilization (Norris andKristensen, 2010). While public awareness of the perils ofthe Cold War seems to have faded since its peaceful conclu-sion in 1991, the academic community –bene ﬁting from
the opening of formerly classi ﬁed archives and the testi-
mony of retired policy makers, of ﬁcers, and analysts –has
uncovered a disconcerting array of practices and incidentswhich seem to have repeatedly brought the world to thebrink.
17Just how close we came remains a topic of dispute.
Some scholars have argued that it was only thanks to a
good deal of luck that nuclear holocaust was avoided.18
Whether surviving the Cold War required much luck or
just a little, we can easily imagine a counterfactual in whichthe odds of avoiding a nuclear con ﬂagration would be sub-
stantially worse. This holds even if we assume that nuclearweapons can be produced only by large technologicallyadvanced states (thus distinguishing the case from the type-1 vulnerability of ‘easy nukes ’). The counterfactual could
involve changes in the technological possibility frontier thatwould have made the arms race less stable.
For example, it is widely believed among nuclear strate-
gists that the development of a reasonably secure second-strike capability by both superpowers by the mid-1960s cre-ated the conditions for ‘strategic stability ’(Colby and Ger-
son, 2013). Prior to this period, American war plans re ﬂected
a much greater inclination, in any crisis situation, to launcha preemptive nuclear strike against the Soviet Union ’s
nuclear arsenal. The introduction of nuclear submarine-based ICBMs was thought to be particularly helpful forensuring second-strike capabilities (and thus ‘mutually
assured destruction ’) since it was widely believed to be
practically impossible for an aggressor to eliminate the
adversary ’s boomer ﬂeet in the initial attack.
19Other strate-
gies for ensuring a second-strike capability could also beemployed, but they had drawbacks. For example, oneoption, brie ﬂy used by the United States, was to have a con-
tingent of long-range nuclear bombers on continuous air-borne alert (Sagan, 1995). This program was very costly andincreased the risk of accidental or unauthorized attacks.Another option was to build hardened land-based missilesilos: in suf ﬁcient numbers, these could in principle provide
the assurance of a second-strike capability to one side; how-
ever, such a large arsenal would then threaten to provide
the capacity of a safe ﬁrst strike against the other side, thus
again destabilizing any crisis. Road-mobile ICBM launchers,which are harder to attack than silo-based missiles, eventu-ally provided some stabilization when they were deployedby the Soviet Union in 1985, a few years before the end ofCold War (Brower, 1989).
Global Policy (2019) 10:4 ©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.The Vulnerable World Hypothesis459
So consider a counterfactual in which a preemptive coun-
terforce strike is more feasible. Imagine some technologythat makes it easy to track ballistic missile submarines. Wecan also imagine that nuclear weapons were a bit morefragile, so that the radius within which a nuclear weaponwould be destroyed by the detonation of another nuclear
weapon was substantially larger than it actually is.
20Under
those circumstances, it might have been impossible toensure a second-strike capability. Suppose, further, thattechnology had been such as to make it very hard to detectmissile launches, rendering a launch-on-warning strategycompletely unworkable. The crisis instability of the Cold Warwould then have been greatly ampli ﬁed. Whichever side
struck ﬁrst would survive relatively unscathed (or might at
least have believed that it would, since the possibility of a
nuclear winter was largely ignored by war planners at thetime; Badash, 2001; Ellsberg, 2017).
21The less aggressive
side would be utterly destroyed. In such a situation, mutual
fear could easily trigger a dash to all-out war (Schelling,1960).
Other technological parameter changes could similarly
increase the probability of attacks. In the real world, themain ‘attraction ’of a nuclear ﬁrst strike is that it would alle-
viate the fear that one might otherwise oneself become thevictim of such a strike; but we can imagine a counterfactualin which there are also bene ﬁtsto nuclear aggression,
beyond the removal of a negative. Suppose it were some-how possible to derive great economic gains from initiating
a large-scale nuclear assault.
22It might be hard to see how
this could be the case, yet one can imagine some auto-mated manufacturing technology or energy technologymaking physical resources more valuable; or technology-en-abled population growth could again make agricultural landa more vital resource (Drexler, 1986)). Some internationalrelations scholars believe that the net economic bene ﬁts of
conquest have declined substantially in the post-industrialera and that this decline has been a major contributor topeace.
23If powerful national economic motives were again
added to other causes for war (such as concern for one ’s
own security, disputes over non-economic values, mainte-
nance of national reputation, in ﬂuence of particularly belli-
cose special interest groups, inter alia) then armed con ﬂicts
might become more common and large-scale nuclear warmore likely.
In these examples, the vulnerability arises not from
destruction getting easier, but from the actions leading todestruction coming to be supported by stronger incentives.We shall call these Type-2 vulnerabilities. Speci ﬁcally, a sce-
nario like ‘safe ﬁrst strike ’, in which some enormously
destructive action becomes incentivized, we shall refer to as
Type-2a:
Type-2a vulnerability : There is some level of technol-
ogy at which powerful actors have the ability toproduce civilization-devastating harms and, in thesemi-anarchic default condition, face incentives touse that ability.We will see some more examples of Type-2a vulnerabili-
ties below, where the ‘civilization-devastating harms ’take
the form of risk externalities.
T ype-2b ( ‘worse global warming ’)
There is yet another way in which the world could be vul-
nerable; one that we can illustrate with a counterfactual
related to climate change.
In the real world, we observe a secular rise in global
mean temperature, widely believed to be driven primarilyby human-caused emissions of greenhouse gases such ascarbon dioxide, methane, and nitrous oxide (Stocker et al.,2014).Projections vary, depending on the emissions scenarioand modelling assumptions, but forecasts that imply anaverage temperature rise of between 3
oC and 4.5oCi n
2100 (compared to 2000), in the absence of any signi ﬁcant
action to reduce emissions, are quite typical (See Stockeret al. (2014, table 12.2)). The effects of such warming –on
sea levels, weather patterns, ecosystems, and agriculture –
are usually expected to be net negative for human welfare(See Field et al. (2014, ﬁgure 10-1)). Greenhouse gases are
emitted by wide range of activities, including in industry,transport, agriculture, and electricity production, and fromall around the world, though especially from industrializedor industrializing countries. Efforts to curb emissions have sofar failed to achieve much global-scale impact (Friedlingsteinet al., 2014)).
Now, we could imagine a situation in which the problem
of global warming would be far more dire than it actually
seems to be. For example, the transient climate sensitivity (a
measure of the medium-term change in mean global sur-face temperature of the Earth that results from some kindof forcing, such as a doubling of atmospheric CO
2) could
have turned out to be much greater than it is (Shindell,2014). If it had been several times larger than its actualvalue, we would have been in for a temperature rise of, say,15
oor 20oC instead of 3o–a prospect with far greater civi-
lization-destroying potential than the actual expectation.24
We can also imagine other deviations from reality that
would have made global warming a worse problem. Fossil
fuels could have been even more abundant than they are,
and available in more cheaply exploitable deposits, whichwould have encouraged greater consumption. At the sametime, clean energy alternatives could have been moreexpensive and technologically challenging. Global warmingcould also have been a worse problem if there were stron-ger positive feedback loops and nonlinearities, such as aninitial phase in which the atmosphere is gradually loaded upwith greenhouse gases without much observable or detri-mental effect, followed by a second phase in which temper-atures shoot up abruptly. To get a truly civilizational threat
from global warming, it may also be necessary to stipulate,
counterfactually, that mitigation through geoengineering isinfeasible.
The vulnerability illustrated by such a ‘worse global warm-
ing ’scenario is different from that of a Type-2a scenario like
©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd. Global Policy (2019) 10:4Nick Bostrom460
‘safe ﬁrst strike ’. In a Type-2a vulnerability, some actor has
the ability to take some action –such as launching a
nuclear ﬁrst strike –that is destructive enough to devastate
civilization. In the ‘worse global warming’ scenario, no such
actor need exist. Instead, in what we will call a Type-2b vul-
nerability, there is a large number of individually insigni ﬁ-
cant actors who is each incentivized (under the semi-anarchic default condition) to take some action that con-tributes slightly to what cumulatively becomes a civilization-devastating problem:
Type-2b vulnerability : There is some level of technol-
ogy at which, in the semi-anarchic default condi-tion, a great many actors face incentives to takesome slightly damaging action such that the com-bined effect of those actions is civilizational devas-tation.
What Type-2a and Type-2b have in common is that, in both
cases, the damage-capable actors face incentives that wouldencourage a wide range of normally motivated actors in theirsituation to pursue the course of action that leads to damage.Global warming would not be a problem if only some smallfraction of those actors who can drive cars or chop down a fewtrees chose to do so; the problem arises only because many
actors make these choices. And in order for many actors tomake those choices, the choices must be supported by incen-tives that have wide appeal (such as money, status, and conve-nience). Similarly, if only one in a million actors who couldlaunch a nuclear ﬁrst strike would actually choose to do so, then
it would not be so alarming if there are a handful of actors pos-sessing that capability; but it does get worrisome if launching anuclear strike is strongly supported by incentives that appeal tonormally-motivated actors (such as the motive of preempting astrike by one’ s adversary). This is in contrast to a Type-1 vulnera-
bility, where the problem arises from the very widespread pro-liferation of destructive capability. Only an actor with quiteunusual values would choose, at great cost and risk to himself,to blow up a city or unleash a doomsday pathogen; the troublein that case is that if suf ﬁciently many actors possess such a
capability, then the subset of them who also have apocalyptic
m o t i v e si sn o te m p t y .
T ype-0 ( ‘surprising strangelets ’)
In 1942, it occurred to Edward Teller, one of the Manhattan
scientists, that a nuclear explosion would create a tempera-ture unprecedented in Earth ’s history, producing conditions
similar to those in the center of the sun, and that this couldconceivably trigger a self-sustaining thermonuclear reactionin the surrounding air or water Rhodes, 1986). The impor-tance of Teller ’s concern was immediately recognized by
Robert Oppenheimer, the head of the Los Alamos lab.Oppenheimer noti ﬁed his superior and ordered further cal-
culations to investigate the possibility. These calculationsindicated that atmospheric ignition would not occur. Thisprediction was con ﬁrmed in 1945 by the Trinity test, which
involved the detonation of the world ’sﬁrst nuclear explo-
sive.
25In 1954, the US carried out another nuclear test, the Cas-
tle Bravo test, which was planned as a secret experimentwith an early lithium-based thermonuclear bomb design.Lithium, like uranium, has two important isotopes: lithium-6and lithium-7. Ahead of the test, the nuclear scientists calcu-lated the yield to be 6 megatons (with an uncertainty range
of 4 –8 megatons). They assumed that only the lithium-6
would contribute to the reaction, but they were wrong. The
lithium-7 contributed more energy than the lithium-6, andthe bomb detonated with a yield of 15 megaton –more
than double of what they had calculated (and equivalent toabout 1,000 Hiroshimas). The unexpectedly powerful blastdestroyed much of the test equipment. Radioactive falloutpoisoned the inhabitants of downwind islands and the crewof a Japanese ﬁshing boat, causing an international incident.
We may regard it at as lucky that it was the Castle Bravo
calculation that was incorrect, and not the calculation of
whether the Trinity test would ignite the atmosphere. Coun-
terfactually, if the atmosphere had been susceptible to igni-tion by a nuclear detonation, and if this fact had beenrelatively easy to overlook –let us say as easy as it was to
overlook the contribution of the lithium-7 in the CastleBravo test –then the human story (and that of all terrestrial
life) would have come to an end in 1945. We can call thisscenario ‘Castle Bravissimo ’.
Whenever we pull a ball from the urn of invention, there
could conceivably be a possibility of accidental devastation.Usually, this risk is negligible; but in some cases it could be
signi ﬁcant, especially when the technology in question gen-
erates some kind of novel perturbation of nature or intro-
duces historically unprecedented conditions. This suggeststhat we should add to our typology one more category, thatof technology-fated accidental civilizational devastation:
Type-0 vulnerability : There is some technology that
carries a hidden risk such that the default outcomewhen it is discovered is inadvertent civilizationaldevastation.
26
It is instructive to note, however, that ‘Castle Bravissimo ’ is
not a perfect illustration of a Type-0 vulnerability. Suppose that
careful calculations had shown that there was a 1 per cent
probability that a nuclear detonation would ignite the atmo-
sphere and the oceans and thereby extinguish life on Earth.
Suppose, further, that it had been known that to resolve the
matter further and prove that the chance was zero (or alterna-
tively, that the chance was one) would take another 10 years
of meticulous study. It is unclear, under those circumstances,
what the leaders of the Manhattan project would have
decided. They would presumably have thought it greatly desir-
able that humanity hold off on developing nuclear weapons
for at least another 10 years.27 On the other hand, they would
have feared that Germany might have an advanced bomb
project and that Hitler maybe would not pull the brakes
because of a 1 per cent risk of destroying the world.28 They
might have concluded that the risk of testing a nuclear
bomb was worth taking in order to reduce the probability of
Nazi Germany ending up with a nuclear monopoly.
Global Policy (2019) 10:4 ©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.The Vulnerable World Hypothesis461
In this version of ‘Castle Bravissimo ’, civilization gets blown
up by accident: nobody sought to cause a destructive event.Yet the key actors were locked in a strategic situation thatincentivized them to proceed despite the risk. In this respect,the scenario ﬁts as a Type-2a vulnerability; only, the civiliza-
tion-devastating harm it involves is probabilistic. When
nuclear technology becomes possible, powerful actors face
incentives, in the semi-anarchic default condition, to use thattechnology in ways that produce civilization-destroying harms(which here take the form of risk externalities).
29
Accordingly, in order for us to diagnose a Type-0 vulnera-
bility, we require that a stronger condition be met thanmerely that the key actors did not intend destruction. Westipulate that ‘inadvertent ’should here mean that the
adverse outcome sprang from bad luck, not coordinationfailure. In a Type-0 vulnerability, the key actors would, evenif they were adequately coordinated, decide to proceed with
using the technology, in the belief that the bene ﬁts would
outweigh costs –but they would be wrong, and the costs
would be larger than expected, enough so as to cause civi-lizational devastation.
30
Since ‘Castle Bravissimo ’only ambiguously satis ﬁes this crite-
rion (it being unclear in the original counterfactual to whatextent the disaster would have resulted from coordination fail-ure and to what extent from miscalculation/bad luck), it maybe useful to introduce a cleaner example of a Type-0 vulnera-bility. Thus, consider a ‘surprising strangelets ’scenario in which
some modern high-energy physics experiment turns out to ini-
tiate a self-catalyzing process in which ordinary matter gets
converted into strange matter, with the result that our planet isdestroyed. This scenario, and variations thereof in which accel-erator experiments generate stable black holes or trigger thedecay of a metastable vacuum state, have been analyzed inthe literature (Jaffe et al., 2000; Tegmark and Bostrom, 2005).Such outcomes would indeed be very surprising, since analysisindicates that they have a completely negligible chance ofoccurring. Of course, with sufﬁciently bad luck, a negligible-
chance event could occur. But alternatively (and far more likelyin this case), the analysis could have a hidden ﬂaw, like the Cas-
tle Bravo calculations did; in which case the chance might not
be so negligible after all (Ord et al., 2010).
31
Achieving stabilization
The truth of VWH would be bad news. But it would notimply that civilization will be devastated. In principle atleast, there are several responses that could stabilize theworld even if vulnerability exists. Recall that we de ﬁned the
hypothesis in terms of a black-ball technology making civi-lizational devastation extremely likely conditional on techno-
logical development continuing and the semi-anarchic defaultcondition persisting . Thus we can theoretically consider the
following possibilities for achieving stabilization:
1. Restrict technological development.
2. Ensure that there does not exist a large population of
actors representing a wide and recognizably human dis-tribution of motives.3. Establish extremely effective preventive policing.
4. Establish effective global governance.
We will discuss (3) and (4) in subsequent sections. Here
we consider (1) and (2). We will argue they hold only limitedpromise as ways of protecting against potential civilizationalvulnerabilities.
T echnological relinquishment
In its general form, technological relinquishment looks
exceedingly unpromising. Recall that we construed the word‘technology ’broadly; so that completely stopping technolog-
ical development would require something close to a cessa-
tion of inventive activity everywhere in the world. That is
hardly realistic; and if it could be done, it would be extre-mely costly –to the point of constituting an existential
catastrophe in its own right (Namely, ‘permanent stagnation ’
(Bostrom, 2013)).
That general relinquishment of scienti ﬁc and technologi-
cal research is a non-starter does not, however, imply thatlimited curtailments of inventive activities could not be a
good idea. It can make sense to forego particularly perilousdirections of advancement. For instance, recalling our ‘easy
nukes ’scenario, it would be sensible to discourage research
into laser isotope separation for uranium enrichment (Kemp,
2012). Any technology that makes it possible to produceweapons-grade ﬁssile material using less energy or with a
smaller industrial footprint would erode important barriersto proliferation. It is hard to see how a slight reduction inthe price of nuclear energy would compensate. On the con-trary, the world would probably be better off if it somehowbecame harder and more expensive to enrich uranium. What
we would ideally want in this area is not technological pro-gress but technological regress .
While targeted regress might not be in the cards, we
could aim to slow the rate of advancement towards risk-in-
creasing technologies relative to the rate of advancement in
protective technologies. This is the idea expressed by theprinciple of differential technological development. In itsoriginal formulation, the principle focuses on existential risk;but we can apply it more broadly to also encompass tech-nologies with ‘merely ’devastational potential:
Principle of Differential Technological Development .
Retard the development of dangerous and harmfultechnologies, especially ones that raise the level ofexistential risk; and accelerate the development ofbene ﬁcial technologies, especially those that reduce
the existential risks posed by nature or by othertechnologies Bostrom, 2002).
The principle of differential technological development is
compatible with plausible forms of technological determin-ism. For example, even if it were ordained that all technolo-gies that can be developed will be developed, it can still
matter when they are developed. The order in which they
arrive can make an important difference –ideally, protective
technologies should come before the destructive
©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd. Global Policy (2019) 10:4Nick Bostrom462
technologies against which they protect; or, if that is not
possible, then it is desirable that the gap be minimized sothat other countermeasures (or luck) may tide us over untilrobust protection become available. The timing of an inven-tion also in ﬂuences what sociopolitical context the technol-
ogy is born into. For example, if we believe that there is a
secular trend toward civilization becoming more capable of
handling black balls, then we may want to delay the mostrisky technological developments, or at least abstain fromaccelerating them. Even if we suppose that civilizationaldevastation is unavoidable, many would prefer it to takeplace further into the future, at a time when maybe theyand their loved ones are no longer alive anyway.
32
Differential technological development doesn ’t really
make sense in the original urn-of-creativity model, wherethe color of each ball comes as a complete surprise. If wewant to use the urn model in this context, we must modify
it. We could stipulate, for example, that the balls have differ-
ent textures and that there is a correlation between textureand color, so that we get clues about the color of a ballbefore we extract it. Another way to make the metaphormore realistic is to imagine that there are strings or elasticbands between some of the balls, so that when we pull onone of them we drag along several others to which it islinked. Presumably the urn is highly tubular, since certaintechnologies must emerge before others can be reached(we are not likely to ﬁnd a society that uses jet planes and
ﬂint axes). The metaphor would also become more realistic
if we imagine that there is not just one hand daintily explor-
ing the urn: instead, picture a throng of scuf ﬂing prospec-
tors reaching in their arms in hopes of gold and glory, andcitations.
Correctly implementing differential technological develop-
ment is clearly a dif ﬁcult strategic task (Cf. Collingridge,
1980). Nevertheless, for an actor who cares altruisticallyabout long-term outcomes and who is involved in someinventive enterprise (e.g. as a researcher, funder, entrepre-neur, regulator, or legislator) it is worth making the attempt.Some implications, at any rate, seem fairly obvious: for
instance, don ’t work on laser isotope separation, don ’t work
on bioweapons, and don ’t develop forms of geoengineering
that would empower random individuals to unilaterallymake drastic alterations to the Earth ’s climate. Think twice
before accelerating enabling technologies –such as DNA
synthesis machines –that would directly facilitate such omi-
nous developments.
33But boost technologies that are pre-
dominantly protective; for instance, ones that enable moreefﬁcient monitoring of disease outbreaks or that make it
easier to detect covert WMD programs.
Even if it is the case that all possible ‘bad ’technologies
are bound to be developed eventually, it can still be helpful
to buy a little time.
34However, differential technological
development does not on its own offer a solution for vul-nerabilities that persist over long periods –ones where ade-
quately protective technologies are much harder to developthan their destructive counterparts, or where destructionhas the advantage even at technological maturity.
35Preference modiﬁcation
Another theoretically possible way of achieving civilizational
stabilization would be to change the fact that there exists alarge population of actors representing a wide and recogniz-ably human distribution of motives. We reserve for later dis-cussion of interventions that would reduce the effectivenumber of independent actors by increasing various formsof coordination. Here we consider the possibility of modify-ing the distribution of preferences (within a more or lessconstant population of actors).
The degree to which this approach holds promise
depends on which type of vulnerability we have in mind.
In the case of a Type-1 vulnerability, preference modi ﬁca-
tion does not look promising, at least in the absence ofextremely effective means for doing so. Consider that someType-1 vulnerabilities would result in civilizational devasta-tion if there is even a single empowered person anywherein the world who is motivated to pursue the destructiveoutcome. With that kind of vulnerability, reducing the num-ber of people in the apocalyptic residual would do nothingto forestall devastation unless the number could be reducedall the way to zero, which may be completely infeasible. It is
true that there are other possible Type-1 vulnerabilities that
would require a somewhat larger apocalyptic residual inorder for civilizational devastation to occur: for example, in ascenario like ‘easy nukes ’, maybe there would have to be
somebody from the apocalyptic residual in each of severalhundred cities. But this is still a very low bar. It is dif ﬁcult to
imagine an intervention –short of radically re-engineering
human nature on a fully global scale –that would suf ﬁ-
ciently deplete the apocalyptic residual to entirely eliminateor even greatly reduce the threat of Type-1 vulnerabilities.
Note that an intervention that halves the size of the apoc-
alyptic residual would not (at least not through any ﬁrst-
order effect) reduce the expected risk from Type-1 vulnera-
bilities by anywhere near as much. A reduction of 5 per centor 10 per cent of Type-1 risk from halving the apocalypticresidual would be more plausible. The reason is that there iswide uncertainty about how destructive some new black-ball technology would be, and we should arguably use afairly uniform prior in log space (over several orders of mag-nitude) over the size of apocalyptic residual that would berequired in order for civilizational devastation to occur con-ditional on a Type-1 vulnerability arising. In other words,conditional on some new technology being developed that
makes it easy for an average individual to kill at least one
million people, it may be (roughly) as likely that the technol-ogy would enable the average individual to kill one millionpeople, ten million people, a hundred million people, a bil-lion people, or every human alive.
These considerations notwithstanding, preference modi ﬁ-
cation could be helpful in scenarios in which the set ofempowered actors is initially limited to some small de ﬁnable
subpopulation. Some black-ball technologies, when they ﬁrst
emerge from the urn, might be dif ﬁcult to use and require
specialized equipment. There could be a period of several
years before such a technology has been perfected to the
Global Policy (2019) 10:4 ©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.The Vulnerable World Hypothesis463
point where an average individual could master it. During
this early period, the set of empowered actors could bequite limited; for example, it might consist exclusively ofindividuals with bioscience expertise working in a particulartype of lab. Closer screening of applicants to positions insuch labs could then make a meaningful dent in the risk
that a destructive individual gains access to the biotech
black ball within the ﬁrst few years of its emergence.
36And
that reprieve may offer an opportunity to introduce othercountermeasures to provide more lasting stabilization, inanticipation of the time when the technology gets easyenough to use that it diffuses to a wider population.
For Type-2a vulnerabilities, the set of empowered actors
is much smaller. Typically what we are dealing with here arestates, perhaps alongside a few especially powerful non-state actors. In some Type-2a scenarios, the set might con-sist exclusively of two superpowers, or a handful of states
with special capabilities (as is currently the case with nuclear
weapons). It could thus be very helpful if the preferences ofeven a few powerful states were shifted in a more peace-loving direction. The ‘safe ﬁrst strike ’scenario would be a
lot less alarming if the actors facing the security dilemmahad attitudes towards one another similar to those prevail-ing between Finland and Sweden. For many plausible setsof incentives that could arise for powerful actors as a conse-quence of some technological breakthrough, the prospectsfor a non-devastational outcome would be signi ﬁcantly
brightened if the actors in question had more irenic disposi-
tions. Although this seems dif ﬁcult to achieve, it is not as
difﬁcult as persuading almost all the members in the apoca-
lyptic residual to alter their dispositions.
Lastly, consider Type-2b. Recall that such a vulnerability
entails that ‘by default ’a great many actors face incentives
to take some damaging action, such that the combinedeffects add up to civilizational devastation. The incentivesfor using the black-ball technology must therefore be onesthat have a grip on a substantial fraction of the world popu-lation –economic gain being perhaps being the prime
example of such a near-universal motivation. So imagine
some private action, available to almost every individual,
which saves each person who takes it a fraction X of his orher annual income, while producing a negative externalitysuch that if half the world ’s population takes the action then
civilization gets devastated. At X =0, we can assume that
few people would take the antisocial action. But the greaterX is, the larger the fraction of the population that wouldsuccumb to temptation. Unfortunately, it is plausible thatthe value of X that would induce at least half of the popula-tion to take the action is small, perhaps less than 1 percent.
37While it would be desirable to change the distribu-
tion of global preferences so as to make people more altru-
istic and raise the value of X, this seems dif ﬁcult to achieve.
(Consider the many strong forces already competing forhearts and minds –corporate advertisers, religious organiza-
tions, social movements, education systems, and so on.)Even a dramatic increase in the amount of altruism in theworld –corresponding, let us say, to a doubling of X from 1
per cent to 2 per cent –would prevent calamity only in arelatively narrow band of scenarios, namely those in which
the private bene ﬁt of using the destructive technology is in
the 1 –2 per cent range. Scenarios in which the private gain
exceeds 2 per cent would still result in civilizational devasta-tion.
In sum, modifying the distribution of preferences within
the set of actors that would be destructively empowered by
a black-ball discovery could be a useful adjunct to othermeans of stabilization, but it can be dif ﬁcult to implement
and would at best offer only very partial protection (unlesswe assume extreme forms of worldwide re-engineering ofhuman nature).
38
Some speciﬁc countermeasures and their limitations
Beside in ﬂuencing the direction of scienti ﬁc and technologi-
cal progress, or altering destruction-related preferences,there are a variety of other possible countermeasures thatcould mitigate a civilizational vulnerability. For example, one
could try to:
prevent the dangerous information from spreading;
restrict access to requisite materials, instruments, and
infrastructure;
deter potential evildoers by increasing the chance of their
getting caught;
be more cautious and do more risk assessment work; and
establish some kind of surveillance and enforcement
mechanism that would make it possible to interdictattempts to carry out a destructive act
It should be clear from our earlier discussion and exam-
ples that the ﬁrst four of these are not general solutions.
Preventing information from spreading could easily be infea-sible. Even if it could be done, it would not prevent the dan-gerous information from being independently rediscovered.Censorship seems to be at best a stopgap measure.
39
Restricting access to materials, instruments, and infrastruc-ture is a great way to mitigate some kinds of (gray-ball)
threats, but it is unavailing for other kinds of threats –such
as ones in which the requisite ingredients are needed in toomany places in the economy or are already ubiquitouslyavailable when the dangerous idea is discovered (such as
glass, metal, and batteries in the ‘easy nukes ’scenario).
Deterring potential evildoers makes good sense; but for suf-ﬁciently destructive technologies, the existence of an apoca-
lyptic residual renders deterrence inadequate even if everyperpetrator were certain to get caught.
Exercising more caution and doing more risk assessment
is also a weak and limited strategy. One actor unilaterallydeciding to be more cautious may not help much withrespect to a Type-2a vulnerability, and would do basicallynothing for one of Type-2b or Type-1. In the case of a Type-0 vulnerability, it could help if the pivotal actor were more
cautious –though only if the ﬁrst cautiously tiptoeing actor
were not followed by an onrush of incautious actors gettingaccess to the same risky technology (unless the world hadsomehow, in the interim, been stabilized by other means).
40
©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd. Global Policy (2019) 10:4Nick Bostrom464
And as for risk assessment, it could lower the risk only if it
led to some other countermeasure being implemented.41
The last countermeasure in the list –surveillance –does
point towards a more general solution. We will discuss it inthe next section under the heading of ‘preventive policing ’.
But we can already note that on its own it is not suf ﬁcient.
For example, consider a Type-2b vulnerability such as ‘worse
global warming ’. Even if surveillance made it possible for a
state to perfectly enforce any environmental regulation itchooses to impose, there is still the problem of getting asufﬁcient plurality of states to agree to adopt the requisite
regulation –something which could easily fail to happen.
The limitations of surveillance are even more evident in thecase of Type-2a vulnerability, such as ‘safe ﬁrst strike ’, where
the problem is that states (or other powerful actors) arestrongly incentivized to perform destructive acts. The abilityof those states to perfectly control what goes on within
their own borders does not solve this problem. What is
needed to reliably solve problems that involve challenges ofinternational coordination, is effective global governance.
Governance gaps
The limitations of technological relinquishment, preference
modi ﬁcation, and various speci ﬁc countermeasures as
responses to a potential civilizational vulnerability shouldnow be clear. To the extent, therefore, that we are con-cerned that VWH may be true, we must consider theremaining two possible ways of achieving stabilization:1.Create the capacity for extremely effective preventive polic-
ing. Develop the intra-state governance capacity needed
to prevent, with extremely high reliability, any individualor small group –including ones that cannot be deterred
–from carrying out any action that is highly illegal; and
2.Create the capacity for strong global governance . Develop
the inter-state governance capacity needed to reliablysolve the most serious global commons problems andensure robust cooperation between states (and otherstrong organizations) wherever vital security interests areat stake –even where there are very strong incentives to
defect from agreements or refuse to sign on in the ﬁrst
place.
The two governance gaps re ﬂected by (1) and (2), one at
the micro-scale, the other at the macro-scale, are twoAchilles ’heels of the contemporary world order. So long as
they remain unprotected, civilization remains vulnerable to apotential technological black ball that would enable a strike
to be directed there. Unless and until such a discovery
emerges from the urn, it is easy to overlook how exposedwe are.
In the following two sections, we will discuss how ﬁlling
in these governance gaps is necessary to achieve a generalability to stabilize potential civilizational vulnerabilities. Itgoes without saying that there are great dif ﬁculties, and also
very serious potential downsides, in seeking progresstowards (1) and (2). In this paper, we will say little about thedifﬁculties and almost nothing about the potentialdownsides –in part because these are already rather well
known and widely appreciated. However, we emphasize thatthe lack of discussion about arguments against (1) and (2)should not be interpreted as an implicit assertion that thesearguments are weak or that they do not point to importantconcerns. They would, of course, have to be taken into
account in an all-things-considered evaluation. But such an
evaluation is beyond the scope of the present contribution,which focuses speci ﬁcally on considerations ﬂowing from
VWH.
Preventive policing
Suppose that a Type-1 vulnerability opens up. Somebodydiscovers a really easy way to cause mass destruction. Infor-mation about the discovery spreads. The requisite materialsand instruments are ubiquitously available and cannotquickly be removed from circulation. Of course it is highlyillegal for any non-state actor to destroy a city, and anybody
caught doing so would be subject to harsh penalties. But it
is plausible that more than one person in a million belongsto an undeterrable apocalyptic residual. Though small in rel-ative terms, if each such person creates a city-destroyingevent, the absolute number is still too large for civilizationto endure. So what to do?
If we suddenly found ourselves in such a situation, it may
be too late to prevent civilization from being destroyed.However, it is possible to envisage scenarios in whichhuman society would survive such a challenge intact –and
the even harder challenge where individuals can single-
handedly destroy not just one city but the entire world.
What would be required to stabilize such vulnerabilities is
anextremely well-developed preventive policing capacity.
States would need the ability to monitor their citizens clo-sely enough to allow them to intercept anybody who beginspreparing an act of mass destruction.
The feasibility of such surveillance and interception
depend on the speci ﬁcs of the scenario: How long does it
take to deploy the black-ball technology destructively? howobservable are the actions involved? can they be distin-guished from behavior that we don ’t want to prohibit? But
it is plausible that a considerable chunk of the Type-1 vul-
nerability spectrum could be stabilized by a state thatdeploys currently available technologies to the fullest extent.And expected advances in surveillance technology willgreatly expand the achievable protection.
For a picture of what a really intensive level of surveil-
lance could look like, consider the following vignette:
High-tech Panopticon
Everybody is ﬁtted with a ‘freedom tag ’–a
sequent to the more limited wearable surveillancedevices familiar today, such as the ankle tag used
in several countries as a prison alternative, the
bodycams worn by many police forces, the pockettrackers and wristbands that some parents use tokeep track of their children, and, of course, theubiquitous cell phone (which has been
Global Policy (2019) 10:4 ©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.The Vulnerable World Hypothesis465
characterized as ‘a personal tracking device that
can also be used to make calls ’).42The freedom tag
is a slightly more advanced appliance, worn aroundthe neck and bedecked with multidirectional cam-eras and microphones. Encrypted video and audiois continuously uploaded from the device to the
cloud and machine-interpreted in real time. AI algo-
rithms classify the activities of the wearer, his handmovements, nearby objects, and other situationalcues. If suspicious activity is detected, the feed isrelayed to one of several patriot monitoring sta-tions. These are vast of ﬁce complexes, staffed 24/7.
There, a freedom of ﬁcer reviews the video feed on
several screens and listens to the audio in head-phones. The freedom of ﬁcer then determines an
appropriate action, such as contacting the tag-wearer via an audiolink to ask for explanations or
to request a better view. The freedom of ﬁcer can
also dispatch an inspector, a police rapid responseunit, or a drone to investigate further. In the smallfraction of cases where the wearer refuses to desistfrom the proscribed activity after repeated warn-ings, an arrest may be made or other suitablepenalties imposed. Citizens are not permitted toremove the freedom tag, except while they are inenvironments that have been out ﬁtted with ade-
quate external sensors (which however includesmost indoor environments and motor vehicles).
The system offers fairly sophisticated privacy pro-
tections, such as automated blurring of intimatebody parts, and it provides the option to redactidentity-revealing data such as faces and name tagsand release it only when the information is neededfor an investigation. Both AI-enabled mechanismsand human oversight closely monitor all theactions of the freedom of ﬁcers to prevent abuse.
43
Creating and operating the High-tech Panopticon would
require substantial investment, but thanks to the fallingprice of cameras, data transmission, storage, and computing,and the rapid advances in AI-enabled content analysis, itmay soon become both technologically feasible and afford-able. For example, if the cost of applying this to one individ-ual for 1 year falls to around US$140, then the entire worldpopulation could be continuously monitored at a cost ofless than 1 per cent of world GDP. At that price, the system
would plausibly represent a net saving –even setting aside
its use in preventing civilization-scale cataclysms –because
of its utility for regular law enforcement. If the system worksas advertised, many forms of crime could be nearly elimi-nated, with concomitant reductions in costs of policing,courts, prisons, and other security systems. It might alsogenerate growth in many bene ﬁcial cultural practices that
are currently inhibited by a lack of social trust.
If the technical barriers to High-tech Panopticon are
rapidly coming down, how about its political feasibility? Onepossibility is that society gradually drifts towards total social
transparency even absent any big shock to the system. Itmay simply become progressively easier to collect and ana-
lyze information about people and objects, and it may provequite convenient to allow that to be done, to the pointwhere eventually something close to full surveillancebecomes a reality –close enough that with just one more
turn of the screw it can be turned into High-tech Panopti-
con.
44An alternative possibility is that some particular Type-
1 vulnerability comes suf ﬁciently starkly into view to scare
states into taking extreme measures, such as launching acrash program to create universal surveillance. Otherextreme measures that could be attempted in the absenceof a fully universal monitoring system might include adopt-ing a policy of preemptive incarceration, say whenever someset of unreliable indicators suggest a greater than1 per centprobability that some individual will attempt a city-destroy-ing act or worse.
45Political attitudes to such policies would
depend on many factors, including cultural traditions and
norms about privacy and social control; but they would also
depend on how clearly the civilizational vulnerability wasperceived. At least in the case of vulnerabilities for whichthere are several spectacular warning shots, it is plausiblethat the risk would be perceived very clearly. In the ‘easy
nukes ’scenario, for example, after the ruination of a few
great cities, there would likely be strong public support fora policy which, for the sake of forestalling another attack,would involve incarcerating a hundred innocent people forevery genuine plotter.
46In such a scenario, the creation of a
High-tech Panopticon would probably be widely supported
as an overwhelmingly urgent priority. However, for vulnera-
bilities not preceded or accompanied by such incontrovert-ible evidence, the will to robust preventive action maynever materialize.
Extremely effective preventive policing, enabled by ubiq-
uitous real-time surveillance, may thus be necessary to stabi-lize a Type-1 vulnerability. Surveillance is also relevant tosome other types of vulnerability, although not so centrallyas in the case of Type-1.
In a Type-2b vulnerability, the bad outcome is brought
about by the combined actions of a mass of independent
actors who are incentivized to behave destructively. But
unless the destructive behaviours are very hard to observe,intensi ﬁcation of surveillance or preventive policing would
not be needed to achieve stabilization. In ‘worse global
warming ’, for instance, it is not essential that individual
actions be preempted. Dangerous levels of emissions taketime to accumulate, and polluters can be held accountableafter the fact; and it is tolerable if a few of them slipthrough the cracks.
For other Type-2b vulnerabilities, however, enhanced
methods of surveillance and social control could be impor-
tant. Consider ‘runaway mob ’, a scenario in which a mob
forms that kills anybody it comes into contact with whorefuses to join, and which grows ever bigger and more for-midable (Cf. Munz et al., 2009). The ease with which suchbad social equilibria can form and propagate, the feasibilityof reforming them once they have taken hold, and the tollthey exact on human welfare, depend on parameters thatcould be changed by technological innovations, potentially
©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd. Global Policy (2019) 10:4Nick Bostrom466
for the worse. Even today, many states struggle to subdue
organized crime. A black-ball invention (perhaps some cle-ver cryptoeconomic mechanism design) that makes criminalenterprises much more scalable or more damaging in theirsocial effects might create a vulnerability that could only bestabilized if states possessed unprecedented technological
powers of surveillance and social control.
As regards to Type-2a vulnerabilities, where the problem
arises from the incentives facing state powers or othermighty actors, it is less clear how domestic surveillancecould help. Historically, stronger means for social controlmay even have worsened inter-state con ﬂict–the bloodiest
inter-state con ﬂicts have depended on the highly effective
governance capacities of the modern state, for tax collec-tion, conscription, and war propaganda. It is conceivablethat improved surveillance could indirectly facilitate the sta-bilization of a Type-2a vulnerability, such as by changing
sociocultural dynamics or creating new options for making
arms-reduction treaties or non-aggression pacts more veri ﬁ-
able. But it seems equally plausible that the net effect ofstrengthened domestic surveillance and policing powers onType-2a vulnerabilities would, in the absence of reliablemechanisms for resolving international disputes, be in theopposite direction (i.e. tending to produce or exacerbatesuch vulnerabilities rather than to stabilize them).
Global governance
Consider again ‘safe ﬁrst strike ’:s t a t e sw i t ha c c e s st ot h e
black-ball technology by default face strong incentives to use
it destructively even though it would be better for everybody
that no state did so. The original example involved a counter-factual with nuclear weapons, but looking to the future wemight get this kind of black ball from advances in biologicalweapons, or atomically precise manufacturing, or the creationof vast swarms of killer drones, or arti ﬁcial intelligence, or
something else. The set of state actors then confronts a col-lective action problem. Failure to solve this problem meansthat civilization gets devastated in a nuclear Armageddon oranother comparable disaster. It is plausible that, absent effec-tive global governance, states would in fact fail to solve this
problem. By assumption, the problem confronting us here
presents special challenges; yet states have frequently failedto solve easier collective action problems. Human history is
covered head to foot with the pockmarks of war.
With effective global governance, however, the solution
becomes trivial: simply prohibit all states from wielding theblack-ball technology destructively. In the case of ‘safe ﬁrst
strike ’, the most obvious way to do this would be by order-
ing that all nuclear weapons be dismantled and an inspec-tion regime set up, with whatever level of intrusiveness isnecessary to guarantee that nobody recreates a nuclear
capability. Alternatively, the global governance institution
itself could retain an arsenal of nuclear weapons as a bufferagainst any breakout attempt.
To deal with Type-2a vulnerabilities, what civilization
requires is a robust ability to achieve global coordination,speci ﬁcally in matters where state actions have extremelylarge externalities. Effective global governance would also
help with those Type-1 and Type-2b scenarios where somestates are reluctant to institute the kind of preventive polic-ing that would be needed to reliably prevent individualswithin their territories from carrying out a destructive act.
Consider a biotechnological black ball that is powerful
enough that a single malicious use could cause a pandemic
that would kill billions of people, thus presenting a Type-1vulnerability. It would be unacceptable if even a single statefails to put in place the machinery necessary for continuoussurveillance and control of its citizens (or whatever othermechanisms are necessary to prevent malicious use with vir-tually perfect reliability). A state that refuses to implementthe requisite safeguards –perhaps on grounds that it values
personal freedom too highly or accords citizens a constitu-tionally inscribed right to privacy –would be a delinquent
member of the international community. Such a state, even
if its governance institutions functioned admirably in other
respects, would be analogous to a ‘failed state ’whose inter-
nal lack of control makes it a safe haven for pirates andinternational terrorists (though of course in the present casethe risk externality it would be imposing on the rest of theworld would be far larger). Other states certainly would haveground for complaint.
A similar argument applies to Type-2b vulnerabilities, such
as a ‘worse global warming ’scenario in which some states
are inclined to free-ride on the costly efforts of others to cutemissions. An effective global governance institution could
compel every state to do its part.
We thus see that while some possible vulnerabilities can
be stabilized with preventive policing alone, and some othervulnerabilities can be stabilized with global governancealone, there are some that would require both. Extremelyeffective preventive policing would be required becauseindividuals can engage in hard-to-regulate activities thatmust nevertheless be effectively regulated, and strong glo-bal governance would be required because states may haveincentives notto effectively regulate those activities even if
they have the capability to do so. In combination, however,
ubiquitous-surveillance-powered preventive policing and
effective global governance would be suf ﬁcient to stabilize
most vulnerabilities, making it safe to continue scienti ﬁc and
technological development even if VWH is true.
47
Discussion
Comprehensive surveillance and global governance wouldthus offer protection against a wide spectrum of civiliza-tional vulnerabilities. This is a considerable reason in favorof bringing about those conditions. The strength of this rea-son is roughly proportional to the probability that the vul-nerable world hypothesis is true.
It goes without saying that a mechanism that enables
unprecedentedly intense forms of surveillance, or a globalgovernance institution capable of imposing its will on anynation, could also have bad consequences. Improved capa-bilities for social control could help despotic regimes protectthemselves from rebellion. Ubiquitous surveillance could
Global Policy (2019) 10:4 ©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.The Vulnerable World Hypothesis467
enable a hegemonic ideology or an intolerant majority view
to impose itself on all aspects of life, preventing individualswith deviant lifestyles or unpopular beliefs from ﬁnding
refuge in anonymity. And if people believe that everythingthey say and do is, effectively, ‘on the record ’, they might
become more guarded and blandly conventional, sticking
closely to a standard script of politically correct attitudes
and behaviours rather than daring to say or do anythingprovocative that would risk making them the target of anoutrage mob or putting an indelible disqualifying mark ontheir r /C19esum /C19e. Global governance, for its part, could reduce
bene ﬁcial forms of inter-state competition and diversity, cre-
ating a world order with single point of failure: if a worldgovernment ever gets captured by a suf ﬁciently pernicious
ideology or special interest group, it could be game over forpolitical progress, since the incumbent regime might neverallow experiments with alternatives that could reveal that
there is a better way. Also, being even further removed
from individuals and culturally cohesive ‘peoples ’than are
typical state governments, such an institution might bysome be perceived as less legitimate, and it may be moresusceptible to agency problems such as bureaucratic sclero-sis or political drift away from the public interest.
48
It also goes without saying that stronger surveillance and
global governance could have various good consequencesaside from stabilizing civilizational vulnerabilities (see alsoRe, 2016)) ; Bostrom, 2006; cf. Torres, 2018)). More effectivemethods of social control could reduce crime and alleviate
the need for harsh criminal penalties. They could foster a cli-
mate of trust that enables bene ﬁcial new forms of social
interaction and economic activity to ﬂourish. Global gover-
nance could prevent interstate wars, including ones that donot threaten civilizational devastation, and reduce militaryexpenditures, promote trade, solve various global environ-mental and other commons problems, calm nationalistichatreds and fears, and over time perhaps would foster anenlarged sense of cosmopolitan solidarity. It may also causeincreased social transfers to the global poor, which somewould view as desirable.
Clearly, there are weighty arguments both for and against
moving in these directions. This paper offers no judgmentabout the overall balance of these arguments. The ambitionhere is more limited: to provide a framework for thinkingabout potential technology-driven civilizational vulnerabili-ties, and to point out that greatly expanded capacities forpreventive policing and global governance would be neces-sary to stabilize civilization in a range of scenarios. Yes, thisanalysis provides an additional reason in favor of developingthose capacities, a reason that does not seem to have beenplaying a signi ﬁcant role in many recent conversations
about related issues, such as debates about government
surveillance and about proposed reforms of internationaland supranational institutions.
49When this reason is added
to the mix, the evaluation should therefore become more
favourable than it otherwise would have been towards poli-cies that would strengthen governance capacities in theseways. However, whether or not this added reason is suf ﬁ-
ciently weighty to tip the overall balance would depend onother considerations that fall outside the scope of this
paper.
It is worth emphasizing that the argument in this paper
favors certain speci ﬁc forms of governance capacity
strengthening. With respect to surveillance and preventivepolicing, VWH-concerns point speci ﬁcally to the desirability
of governance capacity that makes it possible to extremely
reliably suppress activities that are very strongly disap-proved of by a very large supermajority of the population(and of power-weighted domestic stakeholders). It providessupport for other forms of governance strengthening onlyinsofar as they help create this particular capacity. Similarly,with respect to global governance, VWH-based argumentssupport developing institutions that are capable of reliablyresolving very high-stakes international coordination prob-lems, ones where a failure to reach a solution would resultin civilizational devastation. This would include having the
capacity to prevent great power con ﬂicts, suppress arms
races in weapons of mass destruction, regulate developmentraces and deployment of potential black-ball technologies,and successfully manage the very worst kinds of tragedy ofthe commons. It need not include the capacity to make
states cooperate on a host of other issues, nor does it nec-essarily include the capacity to achieve the requisite stabi-lization using only fully legitimate means. While thosecapacities may be attractive for other reasons, they do notimmediately emerge as desiderata simply from taking VWHseriously. For example, so far as VWH is concerned, it would
theoretically be satisfactory if the requisite global gover-
nance capacity comes into existence via the rise of onesuperpower to a position of suf ﬁcient dominance to give it
the ability, in a suf ﬁciently dire emergency, unilaterally to
impose a stabilization scheme on the rest of the world.
One important issue that we still need to discuss is that
of timing. Even if we became seriously concerned that theurn of invention may contain a black ball, this need notmove us to favor establishing stronger surveillance or globalgovernance now, if we thought that it would be possible to
take those steps later , if and when the hypothesized vulner-
ability came clearly into view. We could then let the world
continue its sweet slumber, in the con ﬁdent expectation
that as soon as the alarm goes off it will leap out of bedand undertake the required actions. But we should questionhow realistic that plan is.
Some historical re ﬂection is useful here. Throughout the
Cold War, the two superpowers (and the entire northernhemisphere) lived in continuous fear of nuclear annihilation,which could have been triggered at any time by accident oras the result of some crisis spiralling out of control. The real-ity of the threat was accepted by all sides. This risk could
have been substantially reduced simply by getting rid of all
or most nuclear weapons (a move which, as a nice sideeffect, could also have saved more than ten trillion dol-lars).
5051Yet, after several decades of effort, only limited
nuclear disarmament and other risk-reduction measureswere implemented. Indeed the threat of nuclear annihilationremains with us to this day. In the absence of strong globalgovernance that can enforce a treaty and compel disputants
©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd. Global Policy (2019) 10:4Nick Bostrom468
to accept a compromise, the world has so far been unable
to solve this most obvious collective action problem.52
But perhaps the reason why the world has failed to elimi-
nate the risk of nuclear war is that the risk was insuf ﬁciently
great? Had the risk been higher, one could eupepticallyargue, then the necessary will to solve the global gover-
nance problem would have been found. Perhaps –though
it does seem rather shaky ground on which to rest the fateof civilization. We should note that although a technologyeven more dangerous than nuclear weapons may stimulatea greater will to overcome the obstacles to achieving stabi-lization, other properties of a black ball could make the glo-bal governance problem more challenging than it was
during the Cold War. We have already illustrated this possi-bility in scenarios such as ‘safe ﬁrst strike ’and ‘worse global
warming ’. We saw how certain properties of a technology
set could generate stronger incentives for destructive use or
for refusing to join (or defecting from) any agreement to
curb its harmful applications.
53
Even if one felt optimistic that an agreement could even-
tually be reached, the question of timing should remain a
serious concern. International collective action problems,even within a restricted domain, can resist solution for along time, even when the stakes are large and indisputable.
It takes time to explain why an arrangement is needed andto answer objections, time to negotiate a mutually accept-able instantiation of the cooperative idea, time to hammerout the details, and time to set up the institutional mecha-
nisms required for implementation. In many situations, hold-
out problems and domestic opposition can delay progressfor decades; and by the time one recalcitrant nation is readyto come on board, another who had previously agreedmight have changed its mind. Yet at the same time, theinterval between a vulnerability becoming clearly visible toall and the point when stabilization measures must be inplace could be short . It could even be negative, if the nature
of the vulnerability leaves room for denialism or if speci ﬁc
explanations cannot be widely provided because of informa-tion hazards. These considerations suggest that it is prob-
lematic to rely on spontaneous ad hoc international
cooperation to save the day once a vulnerability comes intoview.
54
The situation with respect to preventive policing is in
some respects similar, although we see a much faster andmore robust trend –driven by advances in surveillance
technology –towards increasing state capacities for moni-
toring and potentially controlling the actions of their owncitizens than any trend towards effective global governance.At least this is true if we look at the physical realm. In thedigital information realm the outlook is somewhat less clear,
owing to the proliferation of encryption and anonymization
tools, and the frequency of disruptive innovation whichmakes the future of cyberspace harder to foresee. Suf ﬁ-
ciently strong capabilities in physical space would, however,spill over into strong capabilities in the digital realm as well.In High-tech Panopticon, there would be no need for theauthorities to crack ciphers, since they could directlyobserve everything that users type into their computers and
everything that is shown on their screens.
One could take the position that we should not develop
improved methods of surveillance and social control unlessand until a speci ﬁc civilizational vulnerability comes clearly
into view –one that looks suf ﬁciently serious to justify the
sacriﬁce of some types of privacy and the risk of inadver-
tently facilitating a totalitarian nightmare. But as with thecase of international cooperation, we confront a question oftiming. A highly sophisticated surveillance and response sys-tem, like the one depicted in ‘High-tech Panopticon ’, cannot
be conjured up and made fully reliable overnight. Realisti-cally, from our current starting point, it would take manyyears to implement such a system, not to mention the timerequired to build political support. Yet the vulnerabilitiesagainst which such a system might be needed may notoffer us much advance warning. Last week a top academic
biolab may have published an article in Science ; and as you
are reading these words, a popular blogger somewhere inthe world, in hot pursuit of pageviews, might be uploadinga post that explains some clever way in which the lab ’s
result could be used by anybody to cause mass destruction.
In such a scenario, intense social control may need to be
switched on almost immediately. In an unfavorable scenario,the lead time could be as short as hours or days. It wouldthen be too late to start developing a surveillance architec-ture when the vulnerability comes clearly into view. If devas-tation is to be avoided, the mechanism for stabilization
would need to have been put in place beforehand.
What may theoretically be feasible is to develop the capa-
bilities for intrusive surveillance and real-time interception in
advance, but not initially to use those capabilities to any-
thing like their full extent. This would be one way to satisfythe requirement for stabilizing a Type-1 vulnerability (andother vulnerabilities that require highly reliable monitoringof individual actions). By giving human civilization thecapacity for extremely effective preventive policing, wewould have exited one of the dimensions of the semi-anar-chic default condition.
Admittedly, constructing such a system and keeping it in
standby mode would mean that some of the downsides ofactually instituting intense forms social control would beincurred. In particular, it may make oppressive outcomesmore likely:
"[The] question is whether the creation of a system
of surveillance perilously alters that balance too far
in the direction of government control ...We might
imagine a system of compulsory cameras installedin homes, activated only by warrant, being usedwith scrupulous respect for the law over many years.The problem is that such an architecture of surveil-lance, once established, would be dif ﬁcult to dis-
mantle, and prove too potent a tool of control if itever fell into the hands of people who –whether
through panic, malice, or a misguided con ﬁdence in
their own ability to secretly judge the public good –
would seek to use it against us (Sanchez, 2013)."
Global Policy (2019) 10:4 ©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.The Vulnerable World Hypothesis469
Developing a system for turnkey totalitarianism means
incurring a risk, even if one does not intend for the key tobe turned.
One could try to reduce this risk by designing the system
with appropriate technical and institutional safeguards. Forexample, one could aim for a system of ‘structured trans-
parency ’that prevents concentrations of power by organiz-
ing the information architecture so that multipleindependent stakeholders must give their permission inorder for the system to operate, and so that only the speci-ﬁc information that is legitimately needed by some deci-
sion-maker is made available to her, with suitable redactionsand anonymization applied as the purpose permits. Withsome creative mechanism design, some machine learning,and some fancy cryptographic footwork, there might be nofundamental barrier to achieving a surveillance system thatis at once highly effective at its of ﬁcial function yet also
somewhat resistant to being subverted to alternative uses.
How likely this is to be achieved in practice is of course
another matter, which would require further exploration.
55
Even if a signi ﬁcant risk of totalitarianism would inevitably
accompany a well-intentioned surveillance project, it wouldnot follow that pursuing such a project would increase therisk of totalitarianism. A relatively less risky well-intentionedproject, commenced at a time of comparative calm, mightreduce the risk of totalitarianism by preempting a less-well-intentioned and more risky project started during a crisis.But even if there were some net totalitarianism-risk-increas-
ing effect, it might be worth accepting that risk in order to
gain the general ability to stabilize civilization againstemerging Type-1 threats (or for the sake of other bene ﬁts
that extremely effective surveillance and preventive policingcould bring).
Conclusions
This paper has introduced a perspective from which we canmore easily see how civilization is vulnerable to certaintypes of possible outcomes of our technological creativity –
our drawing a metaphorical black ball from the urn ofinventions, which we have the power to extract but not to
put back in. We developed a typology of such potential vul-
nerabilities, and showed how some of them result fromdestruction becoming too easy, others from perniciouschanges in the incentives facing a few powerful state actorsor a large number of weak actors.
We also examined a variety of possible responses and
their limitations. We traced the root cause of our civiliza-tional exposure to two structural properties of the contem-porary world order: on the one hand, the lack of preventivepolicing capacity to block, with extremely high reliability,individuals or small groups from carrying out actions that
are highly illegal; and, on the other hand, the lack of global
governance capacity to reliably solve the gravest interna-tional coordination problems even when vital national inter-ests by default incentivize states to defect. Generalstabilization against potential civilizational vulnerabilities –
in a world where technological innovation is occurringrapidly along a wide frontier, and in which there are large
numbers of actors with a diverse set of human-recognizablemotivations –would require that both of these governance
gaps be eliminated. Until such a time as this is accom-plished, humanity will remain vulnerable to drawing a tech-nological black ball.
Clearly, these re ﬂections provide a pro tanto reason to
support strengthening surveillance capabilities and preven-tive policing systems and for favoring a global governanceregime that is capable of decisive action (whether based onunilateral hegemonic strength or powerful multilateral insti-tutions). However, we have not settled whether these thingswould be desirable all-things-considered, since doing sowould require analyzing a number of other strong consider-ations that lie outside the scope of this paper.
Because our main goal has been to put some signposts
up in the macrostrategic landscape, we have focused our
discussion at a fairly abstract level, developing concepts that
can help us orient ourselves (with respect to long-term out-comes and global desirabilities) somewhat independently ofthe details of our varying local contexts.
In practice, were one to undertake an effort to stabilize
our civilization against potential black balls, one might ﬁnd
it prudent to focus initially on partial solutions and low-hanging fruits. Thus, rather than directly trying to bringabout extremely effective preventive policing or strong glo-bal governance, one might attempt to patch up particulardomains where black balls seem most likely to appear. One
could, for example, strengthen oversight of biotechnology-
related activities by developing better ways to track keymaterials and equipment, and to monitor scientists withinlabs. One could also tighten know-your-customer regulationsin the biotech supply sector, and expand the use of back-ground checks for personnel working in certain kinds of labsor involved with certain kinds of experiment. One canimprove whistleblower systems, and try to raise biosecuritystandards globally. One could also pursue differential tech-nological development, for instance by strengthening thebiological weapons convention and maintaining the global
taboo on biological weapons. Funding bodies and ethical
approval committees could be encouraged to take broaderview of the potential consequences of particular lines ofwork, focusing not only on risks to lab workers, test animals,and human research subjects, but also on ways that thehoped-for ﬁndings might lower the competence bar for
bioterrorists down the road. Work that is predominantly pro-tective (such as disease outbreak monitoring, public healthcapacity building, improvement of air ﬁltration devices)
could be differentially promoted.
Nevertheless, while pursuing such limited objectives, one
should bear in mind that the protection they would offer
covers only special subsets of scenarios, and might be tem-porary. If one ﬁnds oneself in a position to in ﬂuence the
macroparameters of preventive policing capacity or globalgovernance capacity, one should consider that fundamentalchanges in those domains may be the only way to achievea general ability to stabilize our civilization against emergingtechnological vulnerabilities.
©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd. Global Policy (2019) 10:4Nick Bostrom470
Notes
For comments, discussion, and critique, I ’m grateful to Sonja Also ﬁ, Stu-
art Armstrong, Andrew Snyder-Beattie, Chris Anderson, Nick Beckstead,Miles Brundage, Ben Buchanan, Owen Cotton-Barratt, Niel Bowerman,
Paul Christiano, Allan Dafoe, Jeff Ding, Eric Drexler, Peter Eckersley,
Owain Evans, Thomas Homer-Dixon, Thomas Inglesby, John Leslie, Gre-gory Lewis, Matthijs Maas, Jason Matheny, Michael Montague, LukeMuehlhauser, Toby Ord, Ben Pace, Richard Re, Anders Sandberg, Julian
Savulescu, Stefan Schubert, Carl Shulman, Tanya Singh, Helen Toner,
and to the audiences of several workshops and lectures where earlierversions of this work were presented), and to three anonymous referees;and I thank Carrick Flynn, Christopher Galias, Ben Gar ﬁnkel, and Rose
Hadshar for help with the manuscript and many useful suggestions.
This work has received funding from the European Research Council(ERC) under the European Union ’s Horizon 2020 research and innovation
program (grant agreement No. 669751).
1. Obviously, the urn metaphor has important limitations. We will dis-
cuss some of them later.
2. The net effect on the conditions of non-human animals is harder to
assess. In particular, modern factory farming involves the mistreat-
ment of large numbers of animals.
3. There are, however, examples of ‘cultures ’or local populations
whose demise may have been brought about (at least partially) by
their own technological practices, such as Easter Island (Rapa Nui)
people and the Ancestral Puebloans in Mesa Verde (Anasazi), who,according to Diamond (2005), cut down their own forests and then
suffered environmental collapse.
4. Examples may however be found in other species, if we consider
evolutionary adaptations as inventions. For instance, there is a liter-ature exploring how evolutionary dead ends, leading to the extinc-
tion of a species or a population, may ensue from advantageous
evolutionary changes such as ones involved in specialization (inwhich adaptation to a narrow niche may entail an irreversible lossof traits needed to survive in a wider range of environments) (Day
et al., 2016), the emergence of inbred social systems (among e.g.
social spiders) (Aviles and Purcell, 2012), or a switch to sel ﬁng (e.g.
among ﬂowering plant species transitioning from outcrossing to
self-fertilization) (Igic and Busch, 2013).
5. Although most scientists involved in the project favored proposals
such as the Baruch Plan, which would have placed nuclear energyunder international control, they retained little decision-making
power at this point.
6. Metaphorically, of course. But arguably in the metaphor, there should
be more than one trembling ﬁnger on each side, given the widely
delegated command and control (Ellsberg, 2017; Schlosser, 2013).
7. However, within a given state, the number of actors who are
empowered to launch nuclear attacks may be quite large. Ellsberg(2017) claims that, for at least a signi ﬁcant portion of the Cold War,
the authority to launch nuclear weapons was delegated multiple
rungs down the American chain of command. The number of of ﬁ-
cers with the physical ability to launch nuclear weapons, althoughnot the authority to do so, was also necessarily larger. In the Soviet
Union, at one point during the coup against Mikhail Gorbachev in
August 1991, all three of the USSR ’s Chegets ( ‘nuclear briefcases ’)
were in the hands of coup leaders (Sokoski and Tertrais, 2013;Stevenson, 2008).
8. An ‘information hazard ’is a risk arising from the dissemination of
true information, for instance because the information could enablesome agents to cause harm. Bostrom (2011) discusses informationhazards more generally.
9. They might argue that openness would be a bene ﬁt since it would
allow more people to work on countermeasures (cf. the debatearound gain of function work on ﬂu viruses; (Duprex et al., 2015;
Fauci et al., 2011; Sharp, 2005)). They might also argue that, so longas the government continues to justify draconian actions by refer-
encing secret information, it will be dangerously unaccountable toits citizens. A similar belief motivated the American magazine The
Progressive ’s decision in the late 1970s to publish secrets about the
hydrogen bomb, even in the face of a legal challenge by the USDepartment of Energy. The author of the piece, Howard Morland,
wrote: ‘Secrecy itself, especially the power of a few designated ‘ex-
perts ’to declare some topics off-limits, contributes to a political cli-
mate in which the nuclear establishment can conduct business asusual, protecting and perpetuating the production of these horror
weapons ’Morland (1979, p. 3).
10. Generally, in cases where multiple actors each have some indepen-
dent probability of taking an action unilaterally, the probability thatthe action will be taken tends to one as the number of actors
increases. When this phenomenon arises for actors with shared
goals but discordant judgments, due to randomness in the evi-dence they are exposed to or the reasoning they carry out, therearises a ‘unilateralist ’s curse ’(Bostrom et al., 2016). The curse implies
that even a very unwise decision, such as the decision to publish
nuclear weapons designs, is likely to be made if enough actors arein a position to take it unilaterally.
11. Many of these same motivations are evident today among ‘black
hat ’hackers who carry out malicious cyber attacks. For instance, as
a method of extortion, some anonymous hackers have proventhemselves willing to remove cities ’abilities to provide vital services
to their residents (Blinder and Perlroth, 2018). Motivations for eco-
nomically damaging cyber attacks have also seemed to includeboth political ideology and curiosity. Since contemporary cyberattacks are dramatically less destructive than attacks with nuclear
weapons, the set of actors that would be willing to use nuclear
weapons or threaten their use is surely much smaller than the setof actors willing to engage in malicious hacking. Nevertheless, the
social and psychological factors relevant to both cases may be simi-
lar.
12. This concept is distinct from that of international anarchy in the
ﬁeld of international relations. The present concept emphasizes that
anarchy is a matter of degree and is meant to be relatively neutral
as between different schools of thought in IR (cf. Lechner, 2017)).More importantly, it encompasses a lack of governance not just ‘at
the top ’but also ‘at the bottom ’. That is to say, the semi-anarchic
default condition refers to the fact that in the current world order,
not only is there a degree of anarchy at the international level,because of lack of global governance or other fully effective means
of constraining the actions of state and solving global coordination
problems, but there is also a degree of anarchy at the level of indi-viduals (and other sub-state actors) in that even highly functionalstates currently lack the ability to perfectly regulate the actions of
those small actors. For example, despite many states seeking to pre-
vent rape and murder within their territory, rape and murder con-tinue to occur with non-zero frequency. The consequences of thisdegree of anarchy at the bottom could be vastly magni ﬁed if indi-
viduals obtained much greater destructive capabilities.
13. For comparison, a death toll of 15 per cent of the present world
population is more than double the combined effects of World WarI, the Spanish Flu, and World War II as a percentage of global popu-
lation (and the difference is even bigger in absolute terms). A 50 per
cent fall in world GDP is greater than the largest drop in recordedhistory. During the Great Depression, for example, world GDP fell by
an estimated 15 per cent or less, and mostly recovered within a few
years (though some models suggest that it has also had a long-last-ing depressing effect on trade which has chronically impaired theworld economy) (Bolt et al., 2018); Crafts and Fearon, 2010).
14. This paper focuses on technological vulnerabilities. There could also
benatural vulnerabilities that arise independently of the progress
of human civilization, such as a violent meteor barrage set toimpact our planet at some future date. Some natural vulnerabilities
Global Policy (2019) 10:4 ©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.The Vulnerable World Hypothesis471
could be stabilized once our level of technological capability
exceeds some threshold (e.g. the ability to de ﬂect meteors). Plausi-
bly, the risk of technological vulnerabilities is greater than the risk
of natural vulnerabilities, although the case for this is less clear cut
with the severity cutoff of civilizational devastation than it wouldbe if the cutoff were set to existential catastrophe (Bostrom, 2013;
Bostrom and /C19Cirkovi /C19c, 2011).The (big) proviso to this claim (of tech-
nological vulnerabilities dominating) is that it presupposes that it isnot the case that the world is hemorrhaging value-potential at asigni ﬁcant rate. If instead we evaluate things from a perspective in
which what we may term the bleeding world hypothesis is true, then
it may well be that the default devastation arising from natural (i.e.non-human) processes dominates the equation. The bleeding worldhypothesis could hold if, for example: (1) the evaluator cares a lot
about existing people (including self and family) and they are natu-
rally dying off at a substantial rate (e.g. from aging), thereby losingboth the ability to continue enjoying their lives and the opportunityfor vastly greater levels of well-being such as would become possi-
ble at technological maturity; (2) the evaluator cares a lot about
avoiding suffering that could be avoided with more advanced tech-nology but is occurring currently, piling up disutility; (3) there is
some substantial exogenously set rate of civilizational destruction
(e.g. natural disasters, random simulation terminations unrelated toour activities Bostrom, 2003), and while we allow time to lapse weincur a cumulative risk of being destroyed before maxing out our
technological potential; (4) there are ways, using physics we don ’t
currently understand well, to initiate fast-growing processes of valuecreation (such as by creating an exponential cascade of baby-uni-verses whose inhabitants would be overwhelmingly happy), and the
evaluator cares in a scale-sensitive way about such creation; and (5)
other superintelligent constituencies, who are in a position togreatly in ﬂuence things the evaluator cares about, are impatient for
us to reach some advancement, but the value they place on this
decays rapidly over time.
15. The world could remain vulnerable after profound technological
regress, for instance if many prefabricated nukes remain even after
civilization regresses past the point of becoming incapable of man-
ufacturing new ones.
16. It is important to our original ‘easy nukes ’scenario that each
nuclear use requires the efforts of only one individual or of a small
group. Although it might require the combined efforts of hundreds
of actors to devastate civilization in that scenario –after all, ruining
one city or one metropolitan area is not the same as ruining a civi-
lization –these hundreds of actors need not coordinate. This allows
the apocalyptic residual to come into play.
17. Baum et al. (2018) provide an up-to-date list of nuclear accidents
and occasions on which the use of nuclear weapons was consid-
ered. Sagan (1995) provides a more thorough account of dangerous
practices throughout the Cold War. Schlosser (2013) examines near-accidents, focusing in particular on one incident which resulted in anon-nuclear detonation of a Titan-II ICBM.
18. Although there have been few scholarly attempts to assess the
degree of luck involved in avoiding this outcome, one recent esti-mate, drawing from a dataset of near-miss instances, places theprobability of the US and USSR avoiding nuclear war below 50 per
cent (Lundgren, 2013). This is consistent with the views of some
ofﬁcials with insider knowledge of nuclear crises, such as President
John F. Kennedy, who expressed the belief that, in hindsight, the
Cuban missile crisis had between a one-in-two and a one-in-three
chance of leading to nuclear war. Nonetheless, a number of promi-nent international security scholars, such as Kenneth Waltz andJohn Mueller, hold that the probability of nuclear war has been
consistently very low (Mueller, 2009; Sagan and Waltz, 2012).
19. Perhaps believed erroneously. According to a former Commander
of the US Paci ﬁc Fleet, there was a period during the Cold War
when antisubmarine surveillance became extremely effective: ‘[TheUS] could identify by hull number the identity of Soviet subs, and
therefore we could do a body count and know exactly where theywere. In port or at sea. ...so I felt comfortable that we had the
ability to do something quite serious to the Soviet SSBN force on
very short notice in almost any set of circumstances. ’(quoted in
Ford and Rosenberg, 2005, p. 399).
20. In fact, advances in remote sensing, data processing, AI, drones,
and nuclear delivery systems are now threatening to underminenuclear deterrence, especially for states with relatively small andunsophisticated nuclear arsenals (Lieber and Press, 2017).
21. Not really unscathed, of course: radioactive fallout would affect
allies and to a degree the homeland; the economic repercussionswould wreak havoc on markets and usher in a worldwide depres-sion. Still, it would be far preferable to being the target of the
assault (especially if we set aside nuclear winter).
22. Another possibility is that there would be political gains, such as an
increased ability to engage in nuclear coercion against third partiesafter having demonstrated a willingness to use nuclear weapons.
23. Brooks (1999); Gartzke (2007); Gartzke and Rohner (2011). For a dis-
senting view, see Liberman (1993).
24. Human civilization could probably never have arisen if the Earth ’s
climate had been that sensitive to carbon dioxide, since past CO
2
levels (4,000 ppm during the Cambrian period compared to about
410 pmm today) would then presumably have seriously disruptedthe evolution of complex life. A less remote counterfactual might
instead involve some compound that does not occur in signi ﬁcant
quantities in nature but is produced by human civilization, such aschloro ﬂuorocarbons. CFCs have been phased out via the Montreal
Protocol because of their destructive effect on the ozone layer, but
they are also very potent greenhouse gases on a per kilogram basis.
So we could consider a counterfactual in which CFCs had beenindustrially useful on a far greater scale than they were, but with
dramatic delayed cumulative effects on global climate.
25. The report commissioned by Oppenheimer ends: ‘One may con-
clude that the arguments of this paper make it unreasonable toexpect that the N +N reaction could propagate. An unlimited prop-
agation is even less likely. However, the complexity of the argument
and the absence of satisfactory experimental foundation make fur-ther work on the subject highly desirable ’(Konopinski et al., 1946).
26. Type-0 could be viewed as the limiting case of a Type-1: it refers to
a vulnerability that requires zero ill-intentioned actors in order for
civilizational devastation to result –only normally responsible actors
who are willing to proceed with using a technology after an ordi-
nary amount of scrutiny has been given to the new technology.
27. And if 10 years, why not permanently.28. In fact, an account by Albert Speer, the German Minister of Arma-
ments, suggests that Werner Heisenberg discussed the possibility of
a runaway chain reaction with Hitler and that this possibility may
have further dampened Hitler ’s enthusiasm for pursuing the bomb
(Rhodes, 1986).
29. A real-world version of this kind of Type-2a vulnerability, in which
key actors face strategic incentives to take actions that create
unwanted risks for civilization, could arise in the context of a raceto develop machine superintelligence. In unfavorable circumstances,competitive dynamics could present a leading developer with the
choice between launching their own AI before it is safe or relin-
quishing their lead to some other developer who is willing to takegreater risks (Armstrong et al., 2016).
30. For a discussion of how a rational planner would balance consump-
tion growth with safety in various models where growth-inducinginnovation also carries a risk of introducing innovations that reducelifespan, see Jones (2016).
31. Even the ‘surprising strangelets ’scenario may be confounded by
coordination problems, though to a lesser degree than ‘Castle
Bravo/Trinity test ’. The people deciding on science funding alloca-
tions may have different priorities than the public that is providing
©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd. Global Policy (2019) 10:4Nick Bostrom472
the funding. They might, for example, place a higher value on satis-
fying intellectual curiosity, relative to the value placed on keepingrisks low and providing near-term material bene ﬁts to the masses.
Principal-agent problems could then result in more funding for par-
ticle accelerators than such experiments would get in the absenceof coordination problems. Prestige contests between nations –
which might in part be viewed as another coordination failure –
may also be a driver of basic science funding in general and high-energy physics in particular.
32. The people alive at the time when the devastation occurs might
prefer that it had taken place earlier, before they were born, so that
it would all be over and done with and they wouldn ’t be affected.
Their preferences seem to run into a non-identity problem, since ifa civilizational devastation event had taken place before they were
conceived they would almost certainly not have come into exis-
tence (Par ﬁt, 1987).
33. More broadly, many re ﬁnements in biotechnological tools and tech-
niques, which make it easier for amateur DIY biohackers to accom-
plish what previously could only be done by well-resourced
professional research labs, come under suspicion from this perspec-tive. It is very questionable whether the bene ﬁts of DIY biohacking
(glow-in-the-dark house plants?) are worth proliferating the ability
to turn bioengineering to potentially risky or malicious purposes toan expanded set of relatively unaccountable actors.
34. The counterargument that ‘if I don ’t develop it, somebody else will;
so I might as well do it ’tends to overlook the fact that a given sci-
entist or developer has at least some marginal impact on theexpected timing of the new discovery. If it really were the case thata scientist ’s efforts could make nodifference to when the discovery
or invention is made, it would appear that the efforts are a waste
of time and resources, and should be discontinued for that reason.A relatively small shift in when some technological capability
becomes available (say, one month) could be important in some
scenarios (such as if the dangerous technology imposes a signi ﬁ-
cant risk per month until effective defenses are developed anddeployed).
35. By ‘technological maturity ’we mean the attainment of capabilities
affording a level of economic productivity and control over natureclose to the maximum that could feasibly be achieved (in the full-ness of time) (Bostrom, 2013).
36. Access control in bioscience has grown in importance since the
2001 ‘Amerithrax ’incident. In the United States, institutions han-
dling dangerous pathogens are obliged to assess suitability for
employees who will have access, which are also vetted by federal
agencies (Federal Select Agent Program, 2017), and similarapproaches are recommended to countries developing their biose-curity infrastructure (Centre for Biosecurity and Biopreparedness,
2017). The existing regime suffers from two shortcomings: ﬁrst,
there is no global coordination, so bad actors could ‘shop around ’
for laxer regulatory environments; second, the emphasis remains onaccess to biological materials (e.g. samples of certain microorgan-
isms), whereas biological information and technology is increasingly
the principal object of security concern (Lewis et al., 2019).
37. A value of X substantially less than 1 per cent seems consistent
with how little most people give to global charity. It is possible,
however, that an act-omission distinction would make people will-
ing to accept a substantially larger personal sacri ﬁce in order not to
contribute to a global bad than they would in order to contribute a
global good.
38. Note, however, that a positive shift in the preference distribution –
even if insuf ﬁcient to avert catastrophe by simply making some
individual actors not choose the destructive option –could have
important indirect effects. For example, if a large number of people
became slightly more benevolently inclined, this might shift societyinto a more cooperative equilibrium that would support strongergovernance-based stabilization methods such as the ones we dis-
cuss below (cf. ‘moral enhancements ’; Persson and Savulescu, 2012).
39. At a global level, we ﬁnd a patchwork of national classi ﬁcation
schemes and information control systems. They are generally
designed to protect military and intelligence secrets, or to preventembarrassing facts about regime insiders from being exposed to
the public, not to regulate the spread of scienti ﬁc or technological
insights. There are some exceptions, particularly in the case of tech-nical information that bears directly on national security. Forinstance, the Invention Secrecy Act of 1951 in the United States
gives defense agencies the power to bar the award of a patent and
order that an invention be kept secret; though an inventor whorefrains from seeking patent protection is not subject to these stric-tures (Parker and Jacobs 2003). Nuclear inventions are subject to
the ‘born secret ’provision of the Atomic Energy Act of 1946, which
declares all information concerning the design, development, andmanufacture of nuclear weapons –regardless of origin –classi ﬁed
unless it has been of ﬁcially declassi ﬁed (Parker and Jacobs 2003).
Other legal tools, such as export controls, have also been used in
attempts to stem the ﬂow of scienti ﬁc information. The (unsuccess-
ful) efforts of multiple US government agencies to block the publi-
cation and use of strong encryption protocols developed in the
1970s and 1980s provide one notable example (Banisar, 1999).Vol-untary self-censorship by the scienti ﬁc community has been
attempted on very rare occasions. Leo Szilard had some partial suc-
cesses in convincing his physicist colleagues to refrain from publish-
ing on aspects of nuclear ﬁssion (before the start of the Manhattan
Project and the onset of of ﬁcial secrecy), though he encountered
opposition from some scientists who wanted their own work to
appear in journals or who felt that openness was a sacred value in
science. More recently, there were some attempts at scienti ﬁc self-
censorship in relation to avian ﬂu research (Gronvall, 2013). In this
case, the efforts may have been not only ineffectual but counter-
productive, inasmuch as the controversy sparked by open debateabout whether certain results should be published drew moreattention to those results than they would have received if publica-
tion had proceeded unopposed –the so-called ‘Streisand
effect ’.Overall, attempts at scienti ﬁc self-censorship appear to have
been fairly half-hearted and ineffectual. (I say appears , because of
how things unfolded in the publicly known episodes where censor-
ship was attempted. But truly successful attempts to suppress scien-
tiﬁc information wouldn ’t necessarily show up in the public record.)
Even if a few journal editors could agree on standards for how to
deal with papers that pose information hazards, nothing would pre-
vent a frustrated author from sending her manuscript to anotherjournal with lower standards or to publish it on her personal Inter-net page. Most scienti ﬁc communities have neither the culture, nor
the incentives, nor the expertise in security and risk assessment, nor
the institutional enforcement mechanisms that would be requiredfor dealing effectively with infohazards. The scienti ﬁc ethos is rather
this: every ball must be extracted from the urn as quickly as possi-
ble and revealed to everyone in the world immediately; the more
this happens, the more progress has been made; and the more youcontribute to this, the better a scientist you are. The possibility of ablack ball does not enter into the equation.
40. In any case, it is unclear whether we would really want to be more
cautious in general. It might be desirable (from various evaluativeperspectives) to encourage greater caution speci ﬁcally in situations
where there could be extreme global downsides. Yet exhortations
to exercise voluntary caution and restraint in these causes may notbe very effective if the reason for the normatively excessive risk-tak-ing is a coordination problem: the risk-taker gaining some private
bene ﬁt (e.g. pro ﬁ
t or prestige) while generating a global risk exter-
nality. In such cases, therefore, the solution may require a strength-ening of global governance capacity.
Global Policy (2019) 10:4 ©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.The Vulnerable World Hypothesis473
41. It is also possible for risk assessment work to increase the level of
risk, by generating information hazards (Bostrom, 2011).
42. The Orwellian-sounding name is of course intentional, to remind us
of the full range of ways in which such a system could be applied.
43. Implementation details are for illustration only. For example, similar
functionality could be provided by mixed reality eyeglasses instead
of a necklace. Versions of the device could be designed that would
provide many bene ﬁts to the user along with its surveillance func-
tion. In theory, some of the monitoring could be crowd-sourced:when suspicious activity is detected by the AI, the video feed is
anonymized and sent to a random 100 citizens, whose duty is to
watch the feed and vote on whether it warrants further investiga-tion; if at least 10 per cent of them think it does, the (non-anon-ymized) feed gets forwarded to the authorities.
44. Examples of ‘conveniences ’that will plausibly drive more intrusive
surveillance include various kinds of consumer applications andeconomically useful or pro ﬁtable monitoring (e.g. for ad targeting,
price discrimination, etc.); the ability to prevent various things that
cause public outrage, such as child abuse or small-scale terrorism;
and, especially for authoritarian regimes, the ability to suppresspolitical opposition.
45. A milder version of the policy might merely debar such weak sus-
pects from accessing the equipment and materials necessary toproduce the destructive effect. The extent to which this might suf-ﬁce depends on the details of the scenario.
46. A partial implementation of the High-tech Panopticon might replace
incarceration in this scenario, in which only those on some long listof ‘individuals of heightened concern ’were required to wear the
freedom tags.
47. Of course, it is theoretically possible that either of these remedies
would raise rather than lower civilization ’s total vulnerability to a
potential black ball, for example, if adequate global coordination
made extremely effective national policing less likely, or vice versa.
The character of the regimes that would tend to arise under condi-tions of stronger preventive policing or global governance couldalso differ from those in the status quo in ways that would increase
or decrease some civilizational vulnerabilities. For example, surveil-
lance-empowered world leaders might be more or less prone totaking foolish decisions that increase Type-0 vulnerabilities.
48. A special case of Type-2a vulnerability is one in which some set of
regimes jointly achieve the devastational threshold by harming their
own populations. Suppose, for instance, that one held an extremelypessimistic view of political leaders, and thought that they would
be willing to kill an extremely large fraction of their own popula-
tions if doing so would help them hold on to power or gain moreresources. Whereas today such a genocidal initiative would usuallybe counterproductive from the leader ’s point of view (because it
would spark revolts and crash the economy), one could imagine a
different technological environment in which these restraints wouldbe loosened –for example, if a highly centralized AI police force
could reliably suppress any resistance and if robots could easily
replace human workers. (Fortunately, it would appear that in many
scenarios where these things becomes technologically feasible, theruler ’s incentives for genocidal actions would also be weakened: the
hypothesized AI police force would presumably enable the ruler to
maintain power without killing off large parts of the population,
and automation of the economy would greatly increase wealth sothat a smaller fraction of national income would suf ﬁce to give all
citizens a high standard of living.)
49. For example, surveillance debates often focus on the tradeoffs
between the privacy interests of individuals and public demand forsecurity against small-scale terrorist attacks. (Even terrorist incidents
that are usually regarded as large, such as the 9/11 attacks, are
insigni ﬁcantly small-scale by the standards used in this paper.)
50. According to Schwartz (1998), the nuclear arms race during the
Cold War cost 5.8 trillion (1996 dollars) in American expendituresalone, which is equivalent to 9.3 trillion in 2018 dollars. This esti-
mate is quite comprehensive and covers the fuel cycle, weapons,delivery systems, decommissioning, etc. If we add the expenditures
of other countries, we can conclude that human civilization spent
well in excess of 10 trillion dollars on developing and maintaining acapacity to destroy itself with nuclear arms. Most of the cost was
incurred in a period when world GDP was substantially lower than
it is today. Even larger amounts were spent on non-nuclear military
capabilities (over 20 trillion in 2018 dollars in the US alone). It ispossible that the nuclear expenditures saved money on balance by
reducing non-nuclear military spending. Both nuclear and non-nu-
clear military spending re ﬂects the failure of human civilization to
solve global coordination.
51.Substantially rather than entirely , since even if all nuclear weapons
were dismantled, new ones might be created.
52. An agreement for total nuclear disarmament might, of course, have
to involve some provisions about conventional forces and othermatters as well, so as not to endanger strategic stability.
53. One might look at other historical examples to obtain a larger refer-
ence class. The world ’s efforts so far with respect to combating glo-
bal warming do not inspire con ﬁ
dence in its ability to deal
expeditiously with even more dif ﬁcult global collective action prob-
lems. On the other hand, the problem of ozone depletion was suc-cessfully addressed with the Montreal Protocol.
54. Unilateral imposition may be faster, but it requires that some actor
has the capability to impose its will single-handedly on the rest of
the world. If one actor has such an overwhelming power advantage,a form of de facto (weak or latent) global governance is presumablyalready in place.
55. For example, a well-intentioned project may be subverted in its
implementation; or it might turn out to have bugs or institutionaldesign ﬂaws that become apparent only after a period of normal
operation. Even if the system itself functions precisely as intended
and remains uncorrupted, it might inspire the creation of othersurveillance systems that do not have the same democratic safe-guards.
References
Armstrong, S., Bostrom, N. and Shulman, C. (2016) ‘Racing to the
Precipice: A Model of Arti ﬁcial Intelligence Development ’,AI &
Society , 31 (2), pp. 201 –06.
Aviles, L. and Purcell, J. (2012) ‘The Evolution of Inbred Social Systems
in Spiders and Other Organisms: From Short-term Gains to Long-
term Evolutionary Dead Ends? ‘,Advances in the Study of Behavior ,4 4
(1), pp. 99 –133.
Badash, L. (2001) ‘Nuclear Winter: Scientists in the Political Arena ’,
Physics in Perspective , 3 (1), pp. 76 –105.
Banisar, D. (1999) ‘Stopping Science: The Case of Cryptography ’,Health
Matrix , 9(2), pp. 253 –87.
Baum, S., de Neufville, R. and Barrett, A. (2018) ‘A Model for the
Probability of Nuclear War ’,Global Catastrophic Risk Institute Working
Paper 18 –1. Available from: https://doi.org/10.2139/ssrn.3137081
[Accessed 31 July 2019].
Blinder, A. and Perlroth, N. (2018) ‘A Cyberattack Hobbles Atlanta, and
Security Experts Shudder ’,New York Times Available from: https://
www.nytimes.com/2018/03/27/us/cyberattack-atlanta-ransomware.html
Bostrom, N. (2002) ‘Existential Risks –Analyzing Human Extinction
Scenarios and Related Hazards ’,Journal of Evolution and Technology ,9
(1), pp. 1 –30.
Bostrom, N. (2003) ‘Are You Living in a Computer Simulation? ’,
Philosophical Quarterly , 53 (211), pp. 243 –55.
Bostrom, N. (2006) ‘What is a Singleton ’,Linguistic and Philosophical
Investigations , 5 (2), pp. 48 –54.
©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd. Global Policy (2019) 10:4Nick Bostrom474
Bostrom, N. (2008) ‘Why I Want to be a Posthuman When I Grow Up ’,i n
B. Gordijn and R. Chadwick (eds), Medical Enhancement and
Posthumanity . New York: Springer, pp. 107 –37.
Bostrom, N. (2011) ‘Information Hazards: A Typology of Potential Harms
from Knowledge ’,Review of Contemporary Philosophy , 10(1), pp. 44 –
79.
Bostrom, N. (2013) ‘Existential Risk Prevention as Global Priority ’,Global
Policy , 4 (1), pp. 15 –31.
Bostrom, N. and /C19Cirkovi /C19c, M. M. (eds) (2011) Global Catastrophic Risks .
Oxford: Oxford University Press, pp. 1 –29.
Bostrom, N., Douglas, T. and Sandberg, A. (2016) ‘The Unilateralists
Curse and the Case for a Principle of Conformity ’,Social
Epistemology , 30 (4), pp. 350 –71.
Brooks, S. G. (1999) ‘The Globalization of Production and the Changing
Bene ﬁts of Conquest ’,Journal of Con ﬂict Resolution , 43 (5), pp. 646 –70.
Brower, M. (1989) ‘Targeting Soviet Mobile Missiles: Prospects and
Implications ’,Survival , 31 (5), pp. 433 –45.
Centre for Biosecurity and Biopreparedness (2017) An Ef ﬁcient and
Practical Approach to Biosecurity. Available from: https://www.biosec
urity.dk/fileadmin/user\_upload/PDF\_FILER/Biosecurity\_book/An\_efficient\_and\_Practical\_approach\_to\_Biosecurity\_web1.pdf [Accessed 31
July 2019].
Colby, E. A. and Gerson, M. S. (eds.) (2013) Strategic Stability: Contending
Interpretations . Carlisle Barracks: U.S. Army War College Press.
Collingridge, D. (1980) The Social Control of Technology . New York: St.
Martin's Press.
Crafts, N. and Fearon, P. (2010) ‘Lessons from the 1930s Great
Depression ’,Oxford Review of Economic Policy , 26 (3), pp. 285 –317.
Danzig, R., Sageman, M., Leighton, T., Hough, L., Yuki, H., Kotani, R. et al.
(2011) Aum Shinrikyo: Insights Into How Terrorists Develop Biologicaland Chemical Weapons, 2nd Edn. Washington, D.C.: Center for aNew American Security.
Day, E. H., Hua, X. and Bromham, L. (2016) ‘Is Specialization an
Evolutionary Dead End? Testing for Differences in Speciation,Extinction and Trait Transition Rates across Diverse Phylogenies ofSpecialists and Generalists ’,Journal of Evolutionary Biology , 29 (6), pp.
1257 –67.
Diamond, J. (2005) Collapse: How Societies Choose to Fail or Succeed .
New York: Penguin.
Drexler, K. E. (1986) Engines of Creation . New York: Anchor.
Duprex, W. P., Fouchier, R. A., Imperiale, M. J., Lipsitch, M. and Relman,
D. A. (2015) ‘Gain-of-function Experiments: Time for a Real Debate ’,
Nature Reviews Microbiology , 13 (1), pp. 58 –64.
Ellsberg, D. (2017) The Doomsday Machine: Confessions of a Nuclear War
Planner . New York: Bloomsbury Publishing USA.
Fauci, A. S., Nabel, G. J. and Collins, F. S. (2011) ‘A Flu Virus Risk Worth
Taking. ’The Washington Post, Available from: https://www.wa
shingtonpost.com/opinions/a- ﬂu-virus-risk-worth-taking/2011/12/30/
gIQAM9sNRP\_story.html [Accessed 31 July 2019].
Federal Select Agent Program ( (2017) Suitability Assessment Program
Guidance. Available from: https://www.selectagents.gov/resources/
Suitability\_Guidance.pdf [Accessed 31 July 2019].
Field, C. B.,Barros, V. R., Mastrandrea, M. D., Mach, K. J., Abdrabo, M. A.-
K., Adger, W. N. et al. (2014) Climate Change 2014: Impacts,
Adaptation, and Vulnerability. Working Group II Contribution to the
Fifth Assessment Report of the Intergovernmental Panel on Climate
Change . Cambridge: Cambridge University Press.
Ford, C. A. and Rosenberg, D. A. (2005) ‘The Naval Intelligence
Underpinnings of Reagan's Maritime Strategy ’,Journal of Strategic
Studies , 28 (2), pp. 379 –409.
Franck, J., Hughes, D. J., Nickson, J. J., Rabinowitch, E., Seaborg, G. T.,
Stearns, J. C. and Szilard, L. (1945) ‘Report of the Committee on
Political and Social Problems ’,Manhattan Project ‘Metallurgical
Laboratory', University of Chicago, U.S. National Archives, Record Group77, Records of the Chief of Engineers, Manhattan Engineer District,Harrison-Bundy File, folder #76 .Friedlingstein, P., Andrew, R. M., Rogelj, J., Peters, G. P., Canadell, J. G.,
Knutti, R. et al. (2014) ‘Persistent Growth of CO2 Emissions and
Implications for Reaching Climate Targets ’,Nature Geoscience , 7 (10),
pp. 709 –15.
Gartzke, E. (2007) ‘The Capitalist Peace ’,American Journal of Political
Science , 51 (1), pp. 166 –91.
Gartzke, E. and Rohner, D. (2011) ‘The Political Economy of Imperialism,
Decolonization and Development ’,British Journal of Political Science ,
41 (3), pp. 525 –56.
Greenberg, A. (2012) This Machine Kills Secrets: How Wikileaks, Hacktivists,
and Cypherpunks are Freeing the World's Information . New York: Dutton.
Gronvall, G. K. (2013) H5n1: A Case Study for Dual-Use Research . New
York: Council on Foreign Relations.
Holloway, D. (1994) Stalin and the Bomb: The Soviet Union and Atomic
Energy, 1939 –1956 . New Haven: Yale University Press.
Igic, B. and Busch, J. W. (2013) ‘Is Self-fertilization an Evolutionary Dead
End? ’,New Phytologist , 198 (2), pp. 386 –97.
Inklaar, R., de Jong,H., Bolt, J. and van Zanden, J. L. (2018) ‘Rebasing
‘Maddison ’: New Income Comparisons and the Shape of Long-run
Economic Development ’,GGDC Research Memorandum GD-174 ,
Groningen: Groningen Growth and Development Center.
Jaffe, R. L., Busza, W., Wilczek, F. and Sandweiss, J. (2000) ‘Review of
Speculative ‘Disaster Scenarios ’at RHIC ’,Reviews of Modern Physics ,
72 (4), pp. 1125 –1140.
Jones, C. I. (2016) ‘Life and Growth ’,Journal of Political Economy , 124 (2),
pp. 539 –78.
Kemp, R. S. (2012) 'SILEX and Proliferation', Bulletin of the Atomic
Scientists . Available from: https://thebulletin.org/2012/07/silex-and
proliferation/ [Accessed 31 July 2019] .
Konopinski, E. J., Marvin, C. and Teller, E. (1946) ‘Ignition of the
Atmosphere with Nuclear Bombs ’, Report LA-602 . Los Alamos, NM: Los
Alamos Laboratory.
Lechner, S. ( 2017) ‘Anarchy in International Relations ’, Oxford Research
Encyclopedia of International Studies [online], Oxford: Oxford
University Press. Available from: https://oxfordre.com/internationalstudies/view/10.1093/acrefore/9780190846626.001.0001/acrefore-
9780190846626-e-79?print=pdf [Accessed 31 July 2019].
Lewis, G., Millet, P., Sandberg, A., Snyder-Beattie, A., and Gronvall,
G.(2019) ‘Information Hazards in Biotechnology ’,Risk Analysis , 39(5),
pp. 975
–81.
Liberman, P. (1993) ‘The Spoils of Conquest ’,International Security ,1 8
(2), pp. 125 –153.
Lieber, K. A. and Press, D. G. (2017) ‘The New Era of Counterforce:
Technological Change and the Future of Nuclear Deterrence ’,
International Security , 41 (4), pp. 9 –49.
Lundgren, C. (2013) ‘What are the Odds? Assessing the Probability of a
Nuclear War ’,The Nonproliferation Review , 20 (2), pp. 361 –74.
Morland, H. (1979) ‘The H-bomb Secret: How We Got It - Why We're
Telling It ’,The Progressive , 43 (11), pp. 3 –12.
Mowatt-Larssen, R. and Allison, G. T. (2010) Al Qaeda Weapons of Mass
Destruction Threat: Hype or Reality? . Cambridge, MA: Belfer Center for
Science and International Affairs.
Mueller, J. (2009) Atomic Obsession: Nuclear Alarmism from Hiroshima to
al-Qaeda . Oxford: Oxford University Press.
Munz, P., Hudea, I., Imad, J. and Smith, R. J. (2009) ‘'When Zombies
Attack!: Mathematical Modelling of an Outbreak of Zombie Infection ’,
in J.M., Tchuenche and C., Chiyaka (eds.), Infectious Disease Modelling,
Research Progress . New York: Nova Science Publishers, pp. 133 –50.
Norris, R. S. and Kristensen, H. M. (2010) ‘Global Nuclear Weapons
Inventories, 1945 –2010 ’,Bulletin of the Atomic Scientists , 66 (4), pp.
77–83.
Olson, K. B. (1999) ‘Aum Shinrikyo: Once and Future Threat? ’,Emerging
Infectious Diseases , 5 (4), pp. 413 –16.
Ord, T., Hillerbrand, R. and Sandberg, A. (2010) ‘Probing the Improbable:
Methodological Challenges for Risks with Low Probabilities and HighStakes ’,Journal of Risk Research , 13 (2), pp. 191 –205.
Global Policy (2019) 10:4 ©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd.The Vulnerable World Hypothesis475
Parﬁt, D. (1987) Reasons and Persons. , Oxford: Clarendon Press.
Parker, E. R. and Jacobs, L. G. (2003) ‘Government Controls of
Information and Scienti ﬁc Inquiry ’,Biosecurity and Bioterrorism:
Biodefense Strategy, Practice, and Science , 1 (2), pp. 83 –95.
Persson, I. and Savulescu, J. (2012) Unﬁt for the Future: The Need for
Moral Enhancement . Oxford: Oxford University Press.
Re, R. M. ( 2016) ‘Imagining Perfect Surveillance. ’UCLA Law Review
Discourse , 64(1), pp. 264 –92.
Rhodes, R. (1986) The Making of the Atomic Bomb . London: Simon and
Schuster.
Sagan, S. D. (1995) The Limits of Safety: Organizations, Accidents, and
Nuclear Weapons . Princeton, NJ: Princeton University Press.
Sagan, S. D. and Waltz, K. N. (2012) The Spread of Nuclear Weapons: An
Enduring Debate . New York: WW Norton.
Sanchez, J. (2013) ‘A Reply to Epstein and Pilon on NSA's Metadata
Program ’, Cato at Liberty. Available from: https://www.cato.org/blog/
reply-epstein-pilon-nsas-metadata-program [Accessed 31 July 2019].
Schelling, T. C. (1960) The Strategy of Con ﬂict. Cambridge, MA: Harvard
University Press.
Schlosser, E. (2013) Command and Control: Nuclear Weapons, the
Damascus Accident, and the Illusion of Safety . New York: Penguin
Press.
Schwartz, S. I. (1998) Atomic Audit: The Costs and Consequences of US
Nuclear Weapons Since 1940 . Washington, D.C.: Brookings Institution
Press.
Sharp, P. A. (2005) ‘1918 Flu and Responsible Science ’,Science , 310
(5745), p. 17.Shindell, D. T. (2014) ‘Inhomogeneous Forcing and Transient Climate
Sensitivity ’,Nature Climate Change , 4 (4), pp. 274 –77.
Sokoski, H. D. and Tertrais, B. (2013) Nuclear Weapons Security Crisis:
What Does History Teach? . Washington, D.C.: Nonproliferation Policy
Education Center.
Stevenson, J. (2008) Thinking Beyond the Unthinkable: Harnessing Doom
from the Cold War to the War on Terror . New York: Viking.
Stocker, T. F., Qin, D., Plattner, G.-K., Tignor, M. M. B., Allen, S. K.,
Boschung, J. et al. (2014) Climate Change 2013: The Physical Science
Basis. Contribution of Working Group I to the Fifth Assessment Report
of the Intergovernmental Panel on Climate Change . New York:
Cambridge University Press.
Swire, P. (2015) The Declining Half-Life of Secrets and the Future of
Signals Intelligence . Washington, D.C.: New America.
Tegmark, M. and Bostrom, N. (2005) ‘Astrophysics: Is a Doomsday
Catastrophe Likely? ’,Nature , 438 (7069), p. 754.
Torres, P. ( 2018) ‘Superintelligence and the Future of Governance: On
Prioritizing the Control Problem at the End of History. ’in R.
Yampolskiy (ed.), Artiﬁcial Intelligence Safety and Security . Milton:
Chapman and Hall/CRC, pp. 357 –74.
Author Information
Nick Bostrom is a Professor at Oxford University, where he directs the
Future of Humanity Institute, which includes the Center for the Gover-
nance of AI and research groups working on other macrostrategic chal-
lenges for humanity.
©2019 The Authors. Global Policy published by Durham University and John Wiley & Sons Ltd. Global Policy (2019) 10:4Nick Bostrom476

Bostrom considers the possibility "that there is some level of technology at which civilization almost certainly gets destroyed unless quite extraordinary and historically unprecedented degrees of preventive policing and/or global governance are implemented." We were lucky, for example, that starting a nuclear chain reaction required difficult-to-obtain plutonium or uranium, instead of easily-available materials. In the latter case, our civilisation would probably have fallen apart, because it was (and still is) in the "semi-anarchic default condition": we have limited capacity for preventative policing or global governence, and people have a diverse range of motivations, many selfish and some destructive. Bostrom identifies four types of vulnerability which vary by how easily and widely the dangerous technology can be produced, how predictable its effects are, and how strong the incentives to use it are. He also idenitifies four possible ways of stabilising the situation: restrict technological development, influence people's motivations, establish effective preventative policing, and establish effective global governance. He argues that the latter two are more promising in this context, although they increase the risks of totalitarianism. Note that Bostrom doesn't take a strong stance on whether the vulnerable world hypothesis is true, although he claims that it's unjustifiable to have high credence in its falsity.

This is an important paper which I hope will lead to much more analysis of these questions.