This article was downloaded by: [Eindhoven Technical University]
On: 22 November 2014, At: 10:58
Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered
office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Journal of Risk Research
Publication details, including instructions for authors and
subscription information:
http://www.tandfonline.com/loi/rjrr20
Probing the improbable:
methodological challenges for risks
with low probabilities and high stakes
Toby Ord a , Rafaela Hillerbrand b c & Anders Sandberg a
a Future of Humanity Institute , University of Oxford , Oxford, UK
b Ethics for Energy Technologies, Human Technology Center ,
RWTH Aachen University , Aachen, Germany
c Philosophisches Institut der RWTH Aachen , Aachen, Germany
Published online: 12 Feb 2010.
To cite this article: Toby Ord , Rafaela Hillerbrand & Anders Sandberg (2010) Probing the
improbable: methodological challenges for risks with low probabilities and high stakes, Journal of
Risk Research, 13:2, 191-205, DOI: 10.1080/13669870903126267
To link to this article: http://dx.doi.org/10.1080/13669870903126267
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the
“Content”) contained in the publications on our platform. However, Taylor & Francis,
our agents, and our licensors make no representations or warranties whatsoever as to
the accuracy, completeness, or suitability for any purpose of the Content. Any opinions
and views expressed in this publication are the opinions and views of the authors,
and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content
should not be relied upon and should be independently verified with primary sources
of information. Taylor and Francis shall not be liable for any losses, actions, claims,
proceedings, demands, costs, expenses, damages, and other liabilities whatsoever
or howsoever caused arising directly or indirectly in connection with, in relation to or
arising out of the use of the Content.
This article may be used for research, teaching, and private study purposes. Any
substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,
systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &
Conditions of access and use can be found at http://www.tandfonline.com/page/terms-
and-conditions
Journal of Risk Research
Vol. 13, No. 2, March 2010, 191–205
ISSN 1366-9877 print/ISSN 1466-4461 online
© 2010 Taylor & FrancisDOI: 10.1080/13669870903126267http://www.informaworld.com
Probing the improbable: methodological challenges for risks with
low probabilities and high stakes
Toby Orda\*, Rafaela Hillerbrandb,c and Anders Sandberga
aFuture of Humanity Institute, University of Oxford, Oxford, UK; bEthics for Energy
Technologies, Human Technology Center, RWTH Aachen University, Aachen, Germany;
cPhilosophisches Institut der RWTH Aachen, Aachen, Germany
Taylor and FrancisRJRR\_A\_412799.sgm(Received 3 October 2008; final version received 29 January 2009 )
10.1080/13669870903126267Journal of Risk Research1366-9877 (print)/1466-4461 (online)Original Article2009Taylor & Francis0000000002009Mr TobyOrdtoby.ord@philosophy.ox.ac.uk
Some risks have extremely high stakes. For example, a worldwide pandemic or
asteroid impact could potentially kill more than a billion people. Comfortingly,scientific calcultions often put very low probabilities on the occurrence of suchcatastrophes. In this paper, we argue that there are important new methodologicalproblems which arise when assessing global catastrophic risks and we focus on a
problem regarding probability estimation. When an expert provides a calculation
of the probability of an outcome, they are really providing the probability of theoutcome occurring, given that their argument is watertight. However, theirargument may fail for a number of reasons, such as a flaw in the underlying theory,a flaw in the modelling of the problem or a mistake in the calculations. If theprobability estimate given by an argument is dwarfed by the chance that theargument itself is flawed, then the estimate is suspect. We develop this ideaformally, explaining how it differs from the related distinction between model andparameter uncertainty. Using the risk estimates from the Large Hadron Collider asa test case, we show how serious the problem can be when it comes to catastrophicrisks and how best to address it.
Keyw ords: risk analysis; particle accelerator; low probability; high stakes;
calculation error
1. Introduction
Large asteroid impacts are highly unlikely events.1 Nonetheless, governments spend
large sums on assessing the associated risks. It is the high stakes that make these other-wise rare events worth examining. Assessing a risk involves consideration of both thestakes involved and the likelihood of the hazard occurring. If a risk threatens the livesof a great many people, it is not only rational but morally imperative to examine therisk in some detail and to see what we can do to reduce it.
This paper focuses on low-probability high-stakes risks. In Section 2, we show that
the probability estimates in scientific analysis cannot be equated with the likelihoodof these events occurring. Instead of the probability of the event occurring, scientificanalysis gives the event’s probability conditioned on the given argument being sound.Though this is the case in all probability estimates, we show how it becomes crucialwhen the estimated probabilities are smaller than a certain threshold.
\*Corresponding author. Email: toby.ord@philosophy.ox.ac.uk
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

192 T. Ord et al.
To proceed, we need to know something about the reliability of the argument. To
do so, risk analysis commonly falls back on the distinction between model and param-eter uncertainty. We argue that this dichotomy is not well suited for incorporatinginformation about the reliability of the theories involved in the risk assessment.Furthermore, the distinction does not account for mistakes made unknowingly. InSection 3, we therefore propose a three-fold distinction between an argument’s theory,its model and its calculations. While explaining this distinction in more detail, weillustrate it with historic examples of errors in each of the three areas. We indicate howspecific risk assessment can make use of the proposed theory–model–calculationdistinction in order to evaluate the reliability of the given argument and thus improvethe reliability of their probability estimate for rare events.
Recently, concerns have been raised that high-energy experiments in particle
physics, such as the Relativistic Heavy Ion Collider (RHIC) at Brookhaven NationalLaboratory or the Large Hadron Collider (LHC) at CERN, Geneva, may threatenhumanity. If these fears are justified, these experiments pose a risk to humanity thatcan be avoided by simply not turning on the experiment. In Section 4, we use themethods of this paper to address the current debate on the safety of experiments withinparticle physics. We evaluate current reports in the light of our findings and givesuggestions for future research.
The final section brings the debate back to the general issue of assessing low-prob-
ability risk. We stress that the findings in this paper are not to be interpreted as anargument for anti-intellectualism, but rather as arguments for making the noisy andfallible nature of scientific and technical research subject to intellectual reasoning,especially in situations where the probabilities are very low and the stakes are veryhigh.
2. Probability estimates
Suppose you read a report which examines a potentially catastrophic risk and
concludes that the probability of catastrophe is one in a billion. What probabilityshould you assign to the catastrophe occurring? We argue that direct use of thereport’s estimate of one in a billion is naive. This is because the report’s authors arenot infallible and their argument might have a hidden flaw. What the report has toldus is not the probability of the catastrophe occurring, but the probability of thecatastrophe occurring, given that the included argument is sound .
Even if the argument looks watertight, the chance that it contains a critical flaw
may well be much larger than one in a billion. After all, in a sample of a billion appar-ently watertight arguments you are likely to see many that have hidden flaws. Our bestestimate of the probability of catastrophe may thus end up noticeably higher than thereport’s estimate.
2
Let us use the following notation: X, the catastrophe occurs; A, the argument is
sound; P(X), the probability of X and P(X|A), the probability of X given A.
While we are actually interested in P(X), the report provides us only with an esti-
mate of P(X|A), since it cannot fully take into account the possibility that it is in
error.3,4 From the axioms of probability theory, we know that P(X) is related to P(X|A)
by the following formula:
PX PX APA PX AP A() ( |) ( ) ( | ) ( ) . ( )=+ ¬ ¬ 1
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

Journal of Risk Research 193
To use this formula to derive P(X), we would require estimates for the probability
that the argument is sound, P(A), and the probability of the catastrophe occurring,
given that the argument is unsound, P(X|¬A). We are highly unlikely to be able to
acquire accurate values for these probabilities in practice, but we shall see that evencrude estimates are enough to change the way we look at certain risk calculations.
A special case, which occurs quite frequently, is for reports to claim that X is
completely impossible. However, this just tells us that X is impossible, given that all
our current beliefs are correct, that is P(X|A) = 0. By Equation (1) we can see that this
is entirely consistent with P(X) > 0, as the argument may be flawed.
Figure 1 is a simple graphical representation of our main point. The square on the
left represents the space of probabilities as described in the scientific report, where theblack area represents the catastrophe occurring and the white area represents notoccurring. The normalized vertical axis denotes the probabilities for the event occur-ring and not occurring. This representation ignores the possibility of the argumentbeing unsound. To accommodate this possibility, we can revise it in the form of thesquare on the right. The black and white areas have shrunk in proportion to the prob-ability that the argument is sound and a new grey area represents the possibility thatthe argument is unsound. Now, the horizontal axis is also normalized and representsthe probability that the argument is sound.
Figure 1. The left panel depicts a report’s view on the probability of an event occurring. The black area represents the chance of the event occurring, the white area represents it not occurring. The right-hand panel is the more comprehensive picture, tak ing into account the possibility that the argument is flawed and that we thus face a grey area containing an unknown amount of risk.To continue our example, let us suppose that the argument made in the report looks
very solid, and that our best estimate of the probability that it is flawed is one in athousand, ( P(¬A) = 10−3). The other unknown term in Equation (1), P(X|¬A), is gener-
ally even more difficult to evaluate, but for the purposes of the current example, let ussuppose that we think it highly unlikely that the event will occur even if the argumentis not sound and treat this probability as one in a thousand as well. Equation (1) tellsus that the probability of catastrophe would then be just over one in a million – anestimate which is a thousand times higher than that in the report itself. This reflectsthe fact that if the catastrophe were to actually occur, it is much more likely that thiswas because there was a flaw in the report’s argument than that a one in a billion eventtook place.
Flawed arguments are not rare. One way to estimate the frequency of major flaws
in academic papers is to look at the proportions which are formally retracted afterpublication. While some retractions are due to misconduct, most are due to uninten-tional errors.
5 Using the MEDLINE database,7 Cokol et al. (2007) found a raw
Figure 1. The left panel depicts a report’s view on the probability of an event occurring. The
black area represents the chance of the event occurring, the white area represents it not
occurring. The right-hand panel is the more comprehensive picture, taking into account the pos-sibility that the argument is flawed and that we thus face a grey area containing an unknownamount of risk.
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

194 T. Ord et al.
retraction rate of 6.3 × 10−5, but used a statistical model to estimate that the retraction
rate would actually be between 0.001 and 0.01 if all journals received the same levelof scrutiny as those in the top tier. This would suggest that P(¬A) > 0.001 making our
earlier estimate rather optimistic. We must also remember that an argument can easilybe flawed without warranting retraction. Retraction is only called for when the under-lying flaws are not trivial and are immediately noticeable by the academic community.The retraction rate for a field would thus provide a lower bound for the rate of seriousflaws. Of course, we must also keep in mind the possibility that different branches ofscience may have different retraction rates and different error rates: the hard sciencesmay be less prone to error than the more applied sciences. Finally, we can have moreconfidence in an article, the longer it has been open to public scrutiny without a flawbeing detected.
It is important to note the particular connection between the present analysis and
high-stakes low-probability risks. While our analysis could be applied to any risk, itis much more useful for those in this category. For it is only when P(X|A) is very low
that the grey area has a relatively large role to play. If P(X|A) is moderately high, then
the small contribution of the error term is of little significance in the overall probabil-ity estimate, perhaps making the difference between 10 and 10.001% rather than thedifference between 0.001 and 0.002%. The stakes must also be very high to warrantthis additional analysis of the risk, for the adjustment to the estimated probability willtypically be very small in absolute terms. While an additional one in a million chanceof a billion deaths certainly warrants further consideration, an additional one in amillion chance of a house fire may not.
One might object to our approach on the grounds that we have shown only that the
uncertainty is greater than previously acknowledged, but not that the probability of theevent is greater than estimated: the additional uncertainty could just as well decreasethe probability of the event occurring. When applying our approach to arbitrary exam-ples, this objection would succeed; however in this paper, we are specifically lookingat cases where there is an extremely low value of P(X|A), so practically any value of
P(X|¬A) will be higher and thus drive the combined probability estimate upwards. The
situation is symmetric with regard to extremely high estimates of P(X|A), where
increased uncertainty about the argument will reduce the probability estimate, thesymmetry is broken only by our focus on arguments which claim that an event is veryunlikely.
Another possible objection is that since there is always a non-zero probability of
the argument being flawed, the situation is hopeless: any new argument will be unableto remove the grey area completely. It is true that the grey area can never becompletely removed; however, if a new argument ( A
2) is independent of the previous
argument ( A1), then the grey area will shrink, for P(¬A1, ¬A2) < P(¬A1). This can
allow for significant progress. A small remaining grey area can be acceptable ifP(X|¬A) P(¬A) is estimated to be sufficiently small in comparison to the stakes.
3. Theories, models and calculations
The most common way to assess the reliability of an argument is to distinguish
between model and parameter uncertainty and assign reliabilities to these choices.While this distinction has certainly been of use in many practical cases, it is unneces-sarily crude for the present purpose, failing to account for potential errors in thepaper’s calculations or a failure of the background theory.
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

Journal of Risk Research 195
In order to account for all possible mistakes in the argument, we look separately
at its theory , its model and its calculations . The calculations evaluate a concrete model
representing the processes under consideration, for example the formation of blackholes in a particle collision, the response of certain climate parameters (such as meantemperature or precipitation rate) to changes in greenhouse gas concentrations or theresponse of economies to changes in the oil price. These models are mostly derivedfrom more general theories. In what follows, we do not restrict the term ‘theory’ towell-established and mathematically elaborate theories like electrodynamics, quantumchromodynamics or relativity theory. Rather, theories are understood to include theo-retical background knowledge, such as specific research paradigms or the generallyaccepted research practice within a field (see Figure 2). An example is the efficientmarket hypothesis which underlies many models within economics, such as the Black-Scholes model.
Figure 2. Ways in which risk assessments can be flawed.We consider adequate models or theories rather than correct ones. For example,
we wish to allow that Newtonian mechanics is an adequate theory in many situations,while recognizing that in some cases it is clearly inadequate (such as for calculatingthe electron orbitals). We thus call a representation of some system adequate if it isable to predict the relevant system features at the required precision. For example, ifclimate modellers wish to determine the implications our greenhouse gas emissions willhave on the well-being of future generations, their model/theory will not be adequateunless it tells them the changes in the local temperature and precipitation. In contrast,
a model might only need to tell them changes in global temperature and precipitation
to be adequate for answering less sensitive questions. On a theoretical level, much morecould be said about this distinction between adequacy and correctness, but for thepurposes of evaluating the reliability of risk assessment, the explanation above shouldsuffice.
With the following notation: T, the involved theories are adequate; M, the derived
model is adequate; and C, the calculations are correct, we can represent P(A) as P(T,
M, C). We can then benefit from the theory–model–calculation distinction by express-
ing this as the product of three separate probabilities, each of which should be easierto estimate than P(A). Firstly, from the laws of conditional probability, it follows that
P(T, M, C) = P(T) P(M|T) P(C|M, T). This can be simplified as we can assume C to
be independent of M and T, since the correctness of a calculation is independent of
whether the theoretical and model assumptions underpinning it were adequate. Giventhis independence P(C|M, T) = P(C), the above equation can be simplified to:
PA PTPM TPC() ()( | )() . ( )= 2
Figure 2. Ways in which risk assessments can be flawed.
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

196 T. Ord et al.
Substituting this back into Equation (1), we obtain a more tractable formula for P(X).
We have already made a rough attempt at estimating P(A) from the paper retraction
rates. Estimating P(T), P(M|T) and P(C) is more accurate and somewhat easier, though
still of significant difficulty. While estimating the various terms in Equation (2) mustultimately be done on a case-by-case basis, the following elucidation of what we meanby theory, model and calculation will shed some light on how to pursue such an anal-ysis. By incorporating our threefold distinction, it is straightforward to apply findingson the reliability of theories from philosophy of science – based, for example, on prob-abilistic verification methods (e.g. Reichenbach 1938) or falsifications as in Hempel(1950) or Popper (1959). Often, however, the best we can do is to put some boundsupon them based on the historical record. We thus review typical sources of error inthe following three areas.
3.1. Calculation: analytic and numeric
Estimating the correctness of the calculation independently from the adequacy
of the model and the theory seems important whenever the mathematics involved isnon-trivial. Most cases where we are able to provide more than purely heuristic andhand-waving risk assessments are of this sort. Consider climate models evaluatingrunaway climate change and risk estimates for the LHC or for asteroid impacts. Whencalculations accumulate, even trivial mathematical procedures become error-prone. Aparticular difficulty arises due to the division of labour in the sciences: commonly inmodern scientific practice, various steps in a calculation are done by different individ-uals who may be in different working groups in different countries. The Mars Climate
Observer spacecraft was lost in 1999 because a piece of control software from
Lockheed Martin used imperial units instead of the metric units the interfacing NASAsoftware expected (NASA 1999).
Calculation errors are distressingly common. There are no reliable statistics on the
calculation errors made in risk assessment or, even more broadly, within scientificpapers. However, there is research on errors made in some very simple calculationsperformed in hospitals. Dosing errors give an approximate estimate of how oftenmathematical slips occur. Errors in drug charts occur at a rate of 1.2–31% acrossdifferent studies (Prot et al. 2005; Stubbs, Haw, and Taylor 2006; Walsh et al. 2008),with a median of roughly 5% of administrations. Of these errors, 15–40% were doseerrors, giving an overall dose error rate of about 1–2%.
What does this mean for error rates in risk estimation? Since the stakes are high
when it comes to dosing errors, this data represents a serious attempt to get the rightanswer in a life or death circumstance. It is likely that the people doing risk estimationare more reliable at arithmetic than health professionals and have more time for errorcorrection, but it appears unlikely that they would be more reliable in many orders ofmagnitude. Hence, a chance of 10
−3 for a mistake per simple calculation does not
seem unreasonable. A random sample of papers from Nature and the British Medical
Journal found that roughly 11% of statistical results were flawed, largely due to
rounding and transcription errors (García-Berthou and Alcaraz 2004).
Calculation errors include more than just the ‘simple’ slips which we know from
school, such as confusing units, forgetting a negative square root or incorrectly tran-scribing from the line above. Instead, many mistakes arise here due to numericalimplementation of the analytic mathematical equations. Computer-based simulationsand numerical analysis are rarely straightforward. The history of computers contains
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

Journal of Risk Research 197
a large number of spectacular failures due to small mistakes in hardware or software.
The 4 June 1996 explosion of an Ariane 5 rocket was due to a leftover piece of code
triggering a cascade of failures (ESA 1996). Audits of spreadsheets in real world usefind error rates on the order of 88% (Panko 1998). The 1993 Intel Pentium floatingpoint error affected 3–5 million processors, reducing their numeric reliability andhence our confidence in anything calculated with them (Nicely 2008). Programmingerrors can remain dormant for a long time even in apparently correct code, only toemerge under extreme conditions. An elementary and widely used binary search algo-rithm included in the standard libraries for Java was found after nine years to containa bug that emerges only when searching very large lists (Bloch 2006). A mistake indata processing led to the retraction of five high-profile protein structure papers as thehandedness of the molecules had become inverted (Miller 2006).
In cases where computational methods are used in modelling, many mistakes
cannot be avoided. Discrete approximations of the often continuous model equationsare used, and in some cases we know that the discrete version is not a good proxy forthe continuous model (Morawetz and Walke 2003). Moreover, numerical evaluationsare often done on a discrete computational grid, with the values inside the meshesbeing approximated from the values computed at the grid points. Though we knowthat certain extrapolation schemes are more reliable in some cases than others, we areoften unable to exclude the possibility of error, or to even quantify it.
3.2. Ways of modelling and theorizing
Our distinction between model and theory follows the typical use of the terms within
mathematical sciences like physics or economics. Whereas theories are associatedwith broad applicability and higher confidence in the correctness of their description,models are closer to the phenomena. For example, when estimating the probability ofa particular asteroid colliding with the Earth, one would use either Newtonianmechanics or general relativity as a theory for describing the role of gravity. Onecould then use this theory in conjunction with observations of the bodies’ positions,velocities and masses to construct a model, and finally, one could perform a series ofcalculations based on this model to estimate the probability of impact. As this shows,the errors that can be introduced in settling for a specific model include and surpassthose which are sometimes referred to as parameter uncertainty . As well as questions
of the individual parameters (positions, velocities and masses), there are importantquestions of detail (Can we neglect the inner structure of the involved bodies?), andbreadth (Can we focus on the Earth and asteroid only, or do we have to model otherplanets, or the Sun?).
7
As can be seen from this example, one way to distinguish theories from models is
that theories are too general to be applied directly to the problem. For any giventheory, there are many ways to apply it to the problem and these ways give rise todifferent models. Philosophers of science will note that our theory–model distinctionis in accordance with the non-uniform notion used by Giere (1999), Morrison (1998),Cartwright (1999), and others, but differs from that of the semantic interpretation oftheories (Suppes 1957).
We should also note that it is quite possible for an argument to involve several
theories or several models. This complicates the analysis and typically provides addi-tional ways for the argument to be flawed.
8 For example, in estimating the risk of
black hole formation at the LHC, we not only require quantum chromodynamics (the
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

198 T. Ord et al.
theory on which the LHC is built to test), but also relativity and Hawking’s theory of
black hole radiation. In addition to their other roles, modelling assumptions also haveto explain how to glue such different theories together (Hillerbrand and Ghil 2008).
In risk assessment, the systems involved are most often not as well understood as
asteroid impacts. Often, various models exist simultaneously – all known to be incom-plete or incorrect in some way, but difficult to improve upon.
9 Particularly in these
cases, having an expected or desired outcome in mind while setting up a model makesone vulnerable to expectation bias: the tendency to reach the desired answer ratherthan the correct one. This bias has affected many of science’s great names (Jeng2006), and in the case of risk assessment, the desire for a ‘positive’ outcome (safetyin the case of the advocate or danger in the case of the protestor) seems a likely causeof bias in modelling.
3.3. Historical examples of model and theory failure
A dramatic example of a model failure was the Castle Bravo nuclear test on 1 March
1954. The device achieved 15 megatons of yield instead of the predicted four to eightmegatons. Fallout affected parts of the Marshal Islands and irradiated a Japanese fish-ing boat so badly that one fisherman died, causing an international incident (NuclearWeapon Archive 2006). Though the designers at Los Alamos National Laboratoriesunderstood the involved theory of alpha decay, their model of the reactions involvedin the explosion was too narrow, for it neglected the decay of one of the involvedparticles (lithium-7), which turned out to contribute the bulk of the explosion’senergy. The Castle Bravo test is also notable for being an example of model failure ina very serious experiment conducted in the hard sciences and with known high stakes.
The history of science contains numerous examples of how generally accepted
theories have been overturned by new evidence or understanding, as well as a plethoraof minor theories that persisted for a surprising length of time before being disproven.Classic examples for the former include the Ptolemaic system, phlogiston theory andcaloric theory; an example for the latter is human chromosome number, which wassystematically miscounted as 48 (rather than 46) and this error persisted for more than30 years (Gartler 2006).
As a final example, consider Lord Kelvin’s estimates of the age of the Earth
(Burchfield 1975). They were based on information about the earth’s temperature andheat conduction, estimating an age of the Earth between 20 and 40 million years.These estimates did not take into account radioactive heating, for radioactive decaywas unknown at the time. Once it was shown to generate additional heat the modelswere quickly updated. While neglecting radioactivity today would count as a modelfailure, in Lord Kelvin’s day it represented a largely unsuspected weakness in thephysical understanding of the Earth and thus amounted to theory failure. This examplemakes it clear that the probabilities for the adequacy of model and theory are not inde-pendent of each other, and thus in the most general case we cannot further decomposeEquation (2).
4. Applying our analysis to the risks from particle physics research
Particle physics is the study of the elementary constituents of matter and radiation, and
the interactions between them. A major experimental method in particle physicsinvolves the use of particle accelerators, such as the RHIC and LHC, to bring beams
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

Journal of Risk Research 199
of particles near the speed of light and then collide them together. This focuses a large
amount of energy in a very small region and breaks the particles down into theircomponents, which are then detected. As particle accelerators have become larger, theenergy densities achieved have become more extreme, prompting some concern abouttheir safety. These safety concerns have focused on three possibilities: the formationof ‘true vacuum’, the transformation of the Earth into ‘strange matter’, and thedestruction of the Earth through the creation of a black hole.
4.1. True vacuum and strange matter formation
The type of vacuum that exists in our universe might not be the lowest possible
vacuum energy state. In this case, the vacuum could decay to the lowest energy state,either spontaneously, or if triggered by a sufficient disturbance. This would produce abubble of ‘true vacuum’ expanding outwards at the speed of light, converting theuniverse into different state apparently inhospitable for any kind of life (Turner andWilczek 1982).
Our ordinary matter is composed of electrons and two types of quarks: up quarks
and down quarks. Strange matter also contains a third type of quark: the ‘strange’
quark. It has been hypothesized that strange matter might be more stable than normalmatter, and able to convert atomic nuclei into more strange matter (Witten 1984). Ithas also been hypothesized that particle accelerators could produce small negativelycharged clumps of strange matter, known as strangelets . If both these hypotheses were
correct and the strangelet also had a high enough chance of interacting with normalmatter, it would grow inside the Earth, attracting nuclei at an ever higher rate until theentire planet was converted to strange matter – destroying all life in the process.Unfortunately, strange matter is complex and little understood, giving models withwidely divergent predictions about its stability, charge and other properties (Jaffe etal. 2000).
One way of bounding the risk from these sources is the cosmic ray argument : the
same kind of high-energy particle collisions occur all the time in Earth’s atmosphere,on the surface of the Moon and elsewhere in the universe. The fact that the Moon orobservable stars have not been destroyed despite a vast number of past collisions(many at much higher energies than can be achieved in human experiments) suggeststhat the threat is negligible. This argument was first used against the possibility ofvacuum decay (Hut and Rees 1983) but is quite general.
An influential analysis of the risk from strange matter was carried out in Dar, De
Rujula, and Heinz (1999) and formed a key part of the safety report for the RHIC. Thisanalysis took into account the issue that any dangerous remnants from cosmic raysstriking matter at rest would be moving at high relative velocity (and hence much lesslikely to interact) while head-on collisions in accelerators could produce remnantsmoving at much slower speeds. They used the rate of collisions of cosmic rays in freespace to estimate strangelet production. These strangelets would then be slowed bygalactic magnetic fields and eventually be absorbed during star formation. Whencombined with estimates of the supernova rate, this can be used to bound the proba-bility of producing a dangerous strangelet in a particle accelerator. The resultingprobability estimate was < 2 × 10
−9 per year of RHIC operation.10
While using empirical bounds and experimentally tested physics reduces the
probability of a theory error, the paper needs around 30 steps to reach its conclusion.For example, even if there was just a 10−4 chance of a calculation or modelling error
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

200 T. Ord et al.
per step this would give a total P(¬A) ≈ 0.3%. This would easily overshadow the risk
estimate. Indeed, even if just one step had a 10−4 chance of error, this would over-
shadow the estimate.
A subtle complication in the cosmic ray argument was noted in Tegmark and
Bostrom (2005). The Earth’s survival so far is not sufficient as evidence for safety,since we do not know if we live in a universe with ‘safe’ natural laws or a universewhere planetary implosions or vacuum decay do occur, but we have just been exceed-ingly lucky so far. While this latter possibility might sound very unlikely, all observersin such a universe would find themselves to be in the rare cases where their planetsand stars had survived, and would thus have much the same evidence as we do.Tegmark and Bostrom had thus found that in ignoring these anthropic effects, theprevious model had been overly narrow. They corrected for this anthropic bias and,using analysis from Jaffe et al. (2000), concluded that the risk from accelerators wasless than 10
−12 per year.
This is an example of a demonstrated flaw in an important physics risk argument
(one that was pivotal in the safety assessment of the RHIC). Moreover, it is significantthat the RHIC had been running for five years on the strength of a flawed safety report,before Tegmark and Bostrom noticed and fixed this gap in the argument. Althoughthis flaw was corrected immediately after being found, we should also note that thecorrection is dependent on both anthropic reasoning and on a complex model of theplanetary formation rate (Lineweaver, Fenner, and Gibson 2004). If either of these orthe basic Brookhaven analysis is flawed, the risk estimate is flawed.
4.2. Black hole formation
The LHC experiment at CERN was designed to explore the validity and limitations of
the standard model of particle physics by colliding beams of high-energy protons.This will be the most energetic particle collision experiment ever done, which hasmade it the focus of a recent flurry of concerns. Due to the perceived strength of theprevious arguments on vacuum decay and strangelet production, most of the concernabout the LHC has focused on black hole production.
None of the theory papers we have found appears to have considered the black
holes to be a safety hazard, mainly because they all presuppose that any black holeswould immediately evaporate due to Hawking radiation. However, it was suggestedby Dimopoulos and Landsberg (2001) that if black holes form, particle acceleratorscould be used to test the theory of Hawking radiation. Thus, critics also began ques-tioning whether we could simply assume that black holes would evaporate harmlessly.
A new risk analysis of LHC black hole production (Giddings and Mangano 2008)
provides a good example of how risks can be more effectively bounded through multi-ple sub-arguments. While never attempting to give a probability of disaster (ratherconcluding ‘there is no risk of any significance whatsoever from such black holes’), ituses a multiple bounds argument. It first shows that rapid black hole decay is a robustconsequence of several different physical theories ( A
1). Second, it discusses the likely
incompatibility between non-evaporating black holes and mechanisms for neutraliz-ing black holes: in order for cosmic ray-produced stable black holes to be innocuousbut accelerator-produced black holes to be dangerous, they have to be able to shedexcess charge rapidly ( A
2). Our current understanding of physics suggests both that
black holes decay and that even if they did not, they would be unable to dischargethemselves. Only if this understanding is flawed will the next section come into play.
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

Journal of Risk Research 201
The third part, which is the bulk of the paper, models how multidimensional and
ordinary black holes would interact with matter. This leads to the conclusion that ifthe size scale of multidimensional gravity is smaller than about 20 nm, then the timerequired for the black hole to consume the Earth would be larger than the natural life-time of the planet. For scenarios where rapid Earth accretion is possible, the accretiontime inside white dwarves and neutron stars would also be very short, yet productionand capture of black holes from impinging cosmic rays would be so high that thelifespan of the stars would be far shorter than the observed lifespan (and wouldcontradict white dwarf cooling rates) ( A
3).
The force of the total argument ( A1, A2, A3) is significantly stronger than any of its
parts. Essentially, the paper acts as three sequential arguments, each partly filling inthe grey area (see Figure 1) left by the previous. If the theories surrounding black holedecay fail, the argument about discharge comes into play; and if against all expecta-tion black holes are stable and neutral, the third argument shows that astrophysicsconstrains them to a low accretion rate.
4.3. Implications for the safety of the LHC
What are the implications of our analysis for the safety assessment of the LHC?
Firstly, let us consider the stakes in question. If one of the proposed disasters were tooccur, it would mean the destruction of the Earth. This would involve the completedestruction of the environment, 6.5 billion human deaths and the loss of all futuregenerations. It is worth noting that this loss of all future generations (and with it, allof humanity’s potential) may well be the greatest of the three, but a comprehensiveassessment of these stakes is outside the scope of this paper. For the present purposes,it suffices to observe that the destruction of the Earth is at least as bad as 6.5 billion
human deaths.
There is some controversy as to how one should combine probabilities and stakes
into an overall assessment of a risk. Some hold that the simple approach of expectedutility is the best, while others hold some form of risk aversion. However, we can side-step this dispute by noting that in either case, the risk of some harm is at least as badas the expected loss. Thus, a risk with probability p of causing a loss at least as bad as
6.5 billion deaths is at least as bad as a certain 6.5 × 10
9p deaths.
Now let us turn to the best estimate we can make of the probability of one of the
above disasters occurring during the operation of the LHC. While the arguments forthe safety of the LHC are commendable for their thoroughness, they are not infallible.Although the report considered several possible physical theories, it is eminentlypossible that these are all inadequate representations of the underlying physical real-ity. It is also possible that the models of processes in the LHC or the astronomicalprocesses appealed to in the cosmic ray argument are flawed in an important way.Finally, it is possible that there is a calculation error in the report.
In Equation (1), P(X) is the sum of two terms. The second of these represents the
additional probability of disaster due to the argument being unsound. It is the productof the probability of argument failure and the probability of disaster given such a failure.Both terms are very difficult to estimate, but we can gain insight by showing the rangesthey would have to lie within, for the risk presented by the LHC to be acceptable.
If we let l denote the acceptable limit of expected deaths from the operation of the
LHC, we get: 6.5 × 10
9 P(X) ≤ l. Since P(X) is at least as great as its second term, we
obtain:
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

202 T. Ord et al.
This inequality puts a severe bound on the acceptable values for these probabili-
ties. Since it is much easier to grasp this with an example, we shall provide somenumbers for the purposes of illustration. Suppose, for example, that if the limits wereset at 1000 expected deaths, then P(X|¬A) P(¬A) would have to be below 1.5 × 10
−7
for the risk to be worth bearing. This requires very low values for these probabilities.
We have seen that for many arguments, P(¬A) is above 10−3. We have also seen that
the argument for the safety of the RHIC turned out to have a significant flaw, whichwas unnoticed by the experts at the time. It would thus be very bold to suppose thatthe argument for the safety of the LHC was much lower than 10
−3, but for the sake of
argument, let us grant that it is as low as 10−4 – that out of a sample of 10,000 inde-
pendent arguments of similar apparent merit, only one would have any serious error.
Even if the value of P(¬A) were as low as 10−4, P(X|¬A) would have to be below
0.15% for the risk to be worth taking. P(X|¬A) is the probability of disaster, given that
the arguments of the safety report are flawed and is the most difficult component ofEquation (1) to estimate. Indeed, few would dispute that we really have very little ideaof what value to put on P(X|¬A). It would thus seem overly bold to set this below
0.15% without some substantive argument. Perhaps such an argument could beprovided, but until it is, such a low value for P(X|¬A) seems unwarranted.
We stress that the above combination of numbers was purely for illustrative
purposes, but we cannot find any plausible combination of the three numbers whichmeets the bound and which would not require significant argument to explain eitherthe levels of confidence or the disregard for expected deaths. We would also like tostress that we are open to the possibility that additional supporting arguments andindependent verification of the models and calculations could significantly reduce thecurrent chance of a flaw in the argument.
However, our analysis implies that the current safety report should not be the final
word in the safety assessment of the LHC. To proceed with the LHC on the argumentsof the most recent safety report alone, we would require further work on estimatingP(¬A), P(X|¬A), the acceptable expected death toll, and the value of future generations
and other life on Earth. Such work would require expertise beyond theoretical physics,and an interdisciplinary group would be essential. If the stakes were lower, then itmight make sense for pragmatic concerns to sweep aside this extra level of risk anal-ysis, but the stakes are astronomically large, and so further analysis is critical. Even ifthe LHC goes ahead without any further analysis, as is very likely, these lessons mustbe applied to the assessment of other high-stakes low-probability risks.
5. Conclusions
When estimating threat probabilities, it is not enough to make conservative estimates
(using the most extreme values or model assumptions compatible with known data).Rather, we need robust estimates that can handle theory, model and calculationerrors. The need for this becomes considerably more pronounced for low-probabilityhigh-stake events, though we do not say that low probabilities cannot be treatedsystematically.
Some people have raised the concern that our argument might be too powerful, for
it is impossible to disprove the risk of even something as trivial as dropping a pencil,PX AP A l(| ) ( ) . . ( )¬¬ ≤ ×−15 1 0 310
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

Journal of Risk Research 203
then our argument might amount to prohibiting everything. It is true that we cannot
completely rule out any probability that apparently inconsequential actions might have
disastrous effects, but there are a number of reasons why we do not need to worryabout universal prohibition. A major reason is that for events like the dropping of apencil which have no plausible mechanism for destroying the world, it seems just aslikely that the world would be destroyed by not dropping the pencil. The expected
losses would thus balance out. It is also worth noting that our argument is simply anappeal to a weak form of decision theory to address an unusual concern: for ourmethod to lead to incorrect conclusions, it would require a flaw in decision theoryitself, which would be very big news.
It will have occurred to some readers that our argument is fully applicable to this
very paper: there is a chance that we have made an error in our own arguments. Weentirely agree, but note that this possibility does not change our conclusions very much.Suppose, very pessimistically, that there is a 90% chance that our argument is suffi-ciently flawed that the correct approach is to take safety reports’ probability estimatesat face value. Even then, our argument would make a large difference to how we treatsuch values. Recall the example from Section 2, where a report concludes a probabilityof 10
−9 and we revise this to 10−6. If there is even a 10% chance that we are correct
in doing so, then the overall probability estimate would be revised to 0.9 × 10−9 + 0.1
× 10−6 ≈ 10−7, which is still a very significant change from the report’s own estimate.
In short, even serious doubt about our methods should not move one’s probabilityestimates more than an order of magnitude away from those our method produces. Moremodest doubts would have negligible effect.
The basic message of this paper is that any scientific risk assessment is only able
to give us the probability of a hazard occurring conditioned on the correctness of itsmain argument. The need to evaluate the reliability of the given argument in order toadequately address the risk was shown to be of particular relevance in low-probabilityhigh-stake events. We drew a three-fold distinction between theory, model and calcu-lation, and showed how this can be more useful than the common dichotomy in riskassessment between model and parameter uncertainties. By providing historic exam-ples for errors in the three fields, we clarified the three-fold distinction and showedwhere flaws in a risk assessment might occur. Our analysis was applied to the recentassessment of risks that might arise from experiments within particle physics. Toconclude this paper, we now provide some very general remarks on how to avoidargument flaws when assessing risks with high stakes.
Firstly, the testability of predictions can help discern flawed arguments. If a risk
estimate produces a probability distribution for smaller, more common disasters, thiscan be used to judge whether the observed incidences are compatible with the theory.Secondly, reproducibility appears to be the most effective way of removing many ofthese errors. By having other people replicate the results of calculations indepen-dently, our confidence in them can be dramatically increased. By having other theoriesand models independently predict the same risk probability, our confidence in them
can again be increased, as even if one of the arguments is wrong the others willremain. Finally, we can reduce the possibility of unconscious bias in risk assessmentthrough the simple expedient of splitting the assessment into a ‘blue’ team of expertsattempting to make an objective risk assessment and a ‘red’ team of devil’s advocatesattempting to demonstrate a risk, followed by repeated turns of mutual criticism andupdates of the models and estimates (Calogero 2000). Application of such methodscould in many cases reduce the probability of error by several orders of magnitude.
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

204 T. Ord et al.
Notes
1.Experts estimate the annual probability as approximately one in a billion (Near-Earth
Object Science Definition Team 2003).
2.Scientific arguments are also sometimes erroneous due to deliberate fraud; however, weshall not address this particular concern in this paper.
3.Readers familiar with the distinction between objective and subjective probabilities shouldnote that P(X|A) may be objective or subjective depending on how the report’s argument is
constructed, while P(X) has to be interpreted as a subjective probability.
4.An argument can take into account the possibility that a certain sub-argument is in error.
We encourage such practice and look at an example in Section 4.
5.Between 1982 and 2002, 62% of retractions were due to unintentional errors rather thanmisconduct (Nath, Marcus, and Druss 2006).
6.A very extensive database of biomedical research articles from over 5000 journals.
7.This question of breadth is closely linked to what Hansson (1996) refers to as demarcation
uncertainty . But demarcation of the problem involves not only the obvious demarcation in
physical space and time, but also questions of the systems to consider, the scales toconsider, etc.
8.Additional theories and models can also be deliberately introduced in order to lower the
probability of argument failure, and in Section 4, we shall see how this has been done forthe safety assessment of the LHC.
9.The current climate models provide a good illustration (Frame et al. 2007).
10.Kent (2004) points out some mistakes in stating the risk probabilities in different versionsof the paper, as well as for the Brookhaven report. Even if these are purely typesettingmistakes, it shows that the probability of erroneous risk estimates is non-zero.
References
Bloch, J. 2006. Extra, extra – read all about it: Nearly all binary searches and mergesorts are
broken. http://googleresearch.blogspot.com/2006/06/extra-extra-read-all-about-it-nearly.
html (accessed August 21, 2009).
Burchfield, J.D. 1975. Lord Kelvin and the age of the Earth. New York: Science History.
Calogero, F. 2000. Might a laboratory experiment destroy planet Earth? Interdisciplinary
Science Reviews 25: 191–202.
Cartwright, N. 1999. Dappled world: A study of the boundaries of science. Cambridge:
Cambridge University Press.
Cokol, M., I. Iossifov, R. Rodriguez-Esteban, and A. Rzhetsky. 2007. How many scientific
papers should be retracted? EMBO Reports 8: 422–3.
Dar, A., A. De Rujula, and U. Heinz. 1999. Will relativistic heavy-ion colliders destroy our
planet? Physics Letters B 470: 142–8.
Dimopoulos, S., and G. Landsberg. 2001. Black holes at the large hadron collider. Physical
Review Letters 8716, Art. no. 161602.
ESA. 1996. Ariane 5 flight 501 failure: Report by the inquiry board. ESA 1996: http://
esamultimedia.esa.int/docs/esa-x-1819eng.pdf (accessed August 21, 2009).
Frame, D.J., N.E. Faull, M.M. Joshi, and M.R. Allen. 2007. Probabilistic climate fore-
casts and inductive problems. Philosophical Transactions of the Royal Society A 365:
1971–92.
García-Berthou, E., and C. Alcaraz. 2004. Incongruence between test statistics and P values in
medical papers. BMC Medical Research Methodology 4. http://www.biomedcentral.com/
1471-2288/4/13 (accessed August 21, 2009).
Gartler, S.M. 2006. The chromosome number in humans: A brief history. Nature Reviews
Genetics 7: 655–60.
Giddings, S.B., and M.M. Mangano. 2008. Astrophysical implications of hypothetical stable
TeV-scale black holes Physical Review D 78, 035009, DOI:10.1103/PhysRevD.78.035009.
Giere, R.N. 1999. Science without laws. Chicago: University of Chicago Press.
Hansson, S.O. 1996. Decision making under great uncertainty. Philosophy of the Social
Sciences 26: 369–86.
Hempel, C.G. 1950. Problems and changes in the empiricist criterion of meaning. Revue
Internationale De Philosophie 11: 41–63.
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

Journal of Risk Research 205
Hillerbrand, R.C., and M. Ghil. 2008. Anthropogenic climate change: Scientific uncertainties
and moral dilemmas. Physica D: Nonlinear Phenomena 237: 2132–8.
Hut, P., and M.J. Rees. 1983. How stable is our vacuum. Nature 302: 508–9.
Jaffe, R.L., W. Busza, F. Wilczek, and J. Sandweiss. 2000. Review of speculative ‘disaster
scenarios’ at RHIC. Reviews of Modern Physics 72: 1125–40.
Jeng, M. 2006. A selected history of expectation bias in physics. American Journal of Physics
74: 578–83.
Kent, A. 2004. A critical look at risk assessments for global catastrophes. Risk Analysis 24:
157–68.
Lineweaver, C.H., Y. Fenner, and B.K. Gibson. 2004. The galactic habitable zone and the age
distribution of complex life in the Milky Way. Science 303: 59–62.
Miller, G. 2006. A scientist’s nightmare: Software problem leads to five retractions. Science
314: 1856–7.
Morawetz, K., and R. Walke. 2003. Consequences of coarse-grained Vlasov equations.
Physica A: Statistical Mechanics and its Applications 330: 469–95.
Morrison, M.C. 1998. Modelling nature: Between physics and the physical world.
Philosophia Naturalis 35: 65–85.
NASA. 1999. Mars Climate Orbiter mishap investigation board Phase I report. ftp://
ftp.hq.nasa.gov/pub/pao/reports/1999/MCO\_report.pdf (accessed August 21, 2009).
Nath, S.B., S.C. Marcus, and B.G. Druss. 2006. Retractions in the research literature: Miscon-
duct or mistakes? Medical Journal of Australia 185: 152–4.
Near-Earth Object Science Definition Team. 2003. Study to determine the feasibility of
extending the search for near-earth objects to smaller limiting diameters, August 22, 2003,NASA. http://neo.jpl.nasa.gov/neo/neoreport030825.pdf (accessed August 21, 2009).
Nicely, T.R. 2008. Pentium FDIV flaw. http://www.trnicely.net/pentbug/pentbug.html
(accessed August 21, 2009).
Nuclear Weapon Archive. 2006. Operation castle. http://nuclearweaponarchive.org/Usa/
Tests/Castle.html (accessed August 21, 2009).
Panko, R.R. 1998. What we know about spreadsheet errors. Journal of End User Computing
10: 15–21.
Popper, K. 1959. The logic of scientific discovery. New York: Harper & Row.
Prot, S., J.E. Fontan, C. Alberti, O. Bourdon, C. Farnoux, M.A. Macher, A. Foureau, et al.
2005. Drug administration errors and their determinants in pediatric in-patients. Interna-
tional Journal for Quality in Health Care 17: 381–9.
Reichenbach, H. 1938. Experience and prediction. Chicago: University Of Chicago Press.
Stubbs, J, C. Haw, and D. Taylor. 2006. Prescription errors in psychiatry: A multi-centre
study. Journal of Psychopharmacology 20: 553–61.
Suppes, P. 1957. Introduction to logic. Princeton, NJ: Van Nostrand.
Tegmark, M., and N. Bostrom. 2005. Is a doomsday catastrophe likely? Nature 438: 754.
Turner, M.S., and F. Wilczek. 1982. Is our vacuum metastable? Nature 298: 635–6.
Walsh, K.E., C.P. Landrigan, W.G. Adams, R.J. Vinci, J.B. Chessare, M.R. Cooper, P.M.
Hebert, E.G. Schainker, T.J. McLaughlin, and H. Bauchner. 2008. Effect of computerorder entry on prevention of serious medication errors in hospitalized children. Pediatrics
121: E421–E427.
Witten, E. 1984. Cosmic separation of phases. Physical Review D 30: 272–85.
Downloaded by [Eindhoven Technical University] at 10:58 22 November 2014

Toby Ord
Rafaela Hillerbrand
Anders Sandberg