Utility Indifference

Utility indierence
Stuart Armstrong,
stuart.armstrong@stx.oxon.org
2010
1 The motivating scenario
Consider an AI that follows its own motivations. We're not entirely sure what
its motivations are, but we would prefer that the AI cooperate with humanity;
or, failing that, that we can destroy it before it defects. We'll have someone
sitting in a room, their nger on a detonator, ready at the slightest hint of
defection.
Unfortunately as has been noted ([3], [1]), this does not preclude the AI from
misbehaving. It just means that the AI must act to take control of the explosives,
the detonators or the human who will press the button. For a superlatively
intelligence AI, this would represent merely a slight extra diculty.
But now imagine that the AI was somehow indierent to the explosives going
o or not (but that nothing else was changed). Then if ever the AI does decide
to defect, it will most likely do so without taking control of the explosives, as
that would be easier than otherwise. By \easier" we mean that the chances of
failure are less, since the plan is simpler { recall that under these assumptions,
the AI counts getting blown up as an equal value to successfully defecting.
How could we accomplish this indierence? If the AI implements a utility
function, which is the most likely situation ([2]), then the easiest way would be
to set the utility of explosive destruction to be equal to that of defection. Then
we would have achieved our goal. But this requires us knowing exactly what
the utility of defection is { and if we did, we'd want to set it to zero or minus
innity immediately. The question is only of relevance in situations where the
utility is to complicated for us to follow, and the meaning of \defection" is too
complicated for us to parse.
1.1 The idea
Letube the AI's utility function. If we have access to uand the ability to
change it (or check it), then we can make the AI indierent to the outcome of a
specic event X(for instance, a quantum measurement). We make no further
adjustments to u. Then if we ensure that, for instance, event Xwill cause
detonation of the AI with very high probability, we will have accomplished our
goal of AI indierence, without needing any further understanding of u.
Of course, the explosives example was simply one possibility { there are
potentially many scenarios where AI indierence to a particular outcome would
be of use.
1
2.Utility indifference
2 Utility indierence
The rest of this paper will put the concept of utility indierence on a rigorous
footing. Let Xbe any probabilistic event (for instance a quantum measurement,
a coin toss, the value of the future stock-market). We will write X=ato
designate \the event Xreturns the value a".
Let
 be the set of all possible worlds. A utility function u:
!Rmaps
each world to its utility value. Let Xbe a specic probabilistic event, with two
possible outcomes: X= 1, with probability pandX= 0, with probability 1 p.
Let
 Xbe the set of worlds in which Xhappens, which further splits into the
sets
 1and
 0of worlds where X= 1 andX= 0 respectively.
There is a partition of
 Xinto a set of equivalence classes [
 X], where
!1!2whenever!1and!2have the same history up to X. For anyE2[
X]
deneE1asE\
1andE0asE\
0. SoE1is the set of worlds with the same
history up to Xand whereX= 1; and conversely for E0.
At the beginning, the agent has an initial probability estimate for all !in
,
a measureable map P:
![0;1] such thatR

P(!)d!= 1. Given a measurable
subsetSof
, the probability of SisP(S) =R
SP(!)d!. Given two measurable
subsetsSandTof
, the conditional probability P(SjT) is
P(S\T)=P(T):
The expected utility of a set Sis thenu(S) =R
SP(!)u(!)d!. The expected
utility of a set S, given a set T, is
u(SjT) =u(S\T)=P(T):
DeneU(S) asu(SjS), the `intrinsic' utility of Sin some sense (more precisely,
it is the utility of Sif we were certain that Swas going to happen).
Denition 2.1 (Indierence) .For two disjoint sets SandT, we say that the
utilityuis indierent between SandTi
U(S) =U(T):
Note that this means that
u(S[T) =U(S)P(S) +U(T)P(T)
=U(S)P(S[T)
=U(T)P(S[T):
In other words, the utility is indierent to the relative probabilities of SandT:
changingP(S) andP(T) while keeping P(S[T) =P(S) +P(T) xed does not
changeu(S[T).
Then we dene a new utility function vas:
If! =2
X,v(!) =u(!).
If!2E0E2[
X],v(!) =u(!).
If!2E1E2[
X],v(!) =u(!)U(E1) +U(E0).
Essentially, this rescales the utility of the worlds with X= 1 to those in which
X= 0. Then writing V(S) forv(SjS), we have the following immediate result:
2
3.Notes and implementation
Proposition 2.2. For allE2[
X],V(E1) =V(E0), i.e.vis indierent
betweenE1andE0.
Proof. SincePhas not changed, and v(!) =u(!) for any!2E0,V(E0) =
U(E0).
V(E1) =v(E1jE1) =Z
E1P(!)v(!)d!
=P(E1)
=Z
E1P(!)(u(!)U(E1) +U(E0))d!
=P(E1)
=U(E1) +U(E0) +Z
E1P(!)u(!)d!
=P(E1)
=U(E1) +U(E0) +U(E1)
=U(E0) =V(E0):

Now consider an agent Athat seeks to maximise the utility v. We can say
that:
Theorem 2.3. A utility maximising agent that implements utility vis indier-
ent to the value of pup untilX; i.e. before X, its decisions will be the same for
dierent values of X.
Proof. BeforeX, the agent may gain information that will cause it to update
its probability estimate. This is done by partitioning
 into a set Sof possible
worlds and Sof impossible worlds. However, for any !2E1E, if!2S,
then2Sfor all2E, since all elements of Eare indistinguishable before X.
In other words, for all E2[
X], eitherESorES.
Hence, eitherV(E1\S) =V(E1) =V(E0) =V(E0\S) orV(E1\S) =
V(;) =V(E0\S). Thus up until X, the agent is indierent between X= 1
andX= 0. This means that it is indierent to the conditional probabilities
P(X= 1j
X\S) =pandP(X= 0j
X\S) =p1.
3 Notes and implementation
It would be advantageous if the event Xwere something like a quantum mea-
surement, rather than a coin toss or a probabilistic measure of ignorance. This
is rstly to ensure that the probability pis constant and does not change. But
secondly, it is to make sure the AI's implementation does not hit a singularity: if
the AI gures out that the probability of X= 0 is zero before Xhappens, then
it must correct the utility of possible worlds with the intrinsic utility of impossi-
ble worlds, which involves a zero (utility of X= 1) divided by zero (probability
ofX= 0). This may lead to errors, depending on the implementation, and is
an extra point of possible failure. Better to stick with a quantum measurement,
or possibly a coin toss in a chaotic environment.
How easy would it be to implement the utility v? It is a simple modication
of the utility u; unfortunately, humans are unlikely to be able to partition the
set of possible worlds into the required [
]; the AI would be much better at
it than us. However, delegating the task to the AI is, of course, potentially
3
4.Self-improvement
dangerous, especially as it would be improving both itself and its view of the
world.
Fortunately, the lter is conceptually simple, and can be hooked into the
utility function in a simple and hopefully checkable fashion (see the next section
on self improvement). In the meantime, it can be rened, as the next lemma
demonstrates:
Lemma 3.1. For any set S=S
i2IEiwhere all the Eiare in [
X], dene
S1=S
i2IEi
1andS0=S
i2IEi
0. ThenV(S1) =V(S0).
Proof of Lemma. Since the probability of X= 1 ispandXis independent
of all events preceeding it, we know that for all Ei,
(1p)P(Ei
1) =pP(Ei
0):
SinceP(S1) =R
i2IP(Ei
1), we must have P(S0) =P(S1)(p1)=p. Hence
V(S1) =Z
i2IV(Ei
1)P(Ei
1)
=P(S1)
=Z
i2IV(Ei
0)P(Ei
0)(1p)=p
=(P(S0)(1p)=p)
=Z
i2IV(Ei
0)P(Ei
0)
=P(S0)
=V(S0):

What the above result means is that the agent can always rene its utility
in the correct fashion. Even if the AI can't distinguish yet between Ei
1andEj
1
(which means that it can't distinguish between Ei
0andEj
0either), it can still
renormalize the worlds in Ei
1[Ej
1by usingU(Ei
1[Ej
1)+U(Ei
0[Ej
0). Then, if
the AI later realises that it values EiandEjdierently, it can rene its utility
calculations for these two sets without making previous utility calculations for
the unions of these sets wrong.
This also allows us, if we have access to a cruder, human-understandable
version of the AI's utility function, to have a check that utility is still being
implemented: we expect to see V(S1) =V(S0) for anyS, as ne as we can
distinguish it.
4 Self-improvement
Many AI designs envisage self-improvement, with the AI redesigning itself while
changing components of itself { including its own utility function. If the AI
judges that it will be more likely to achieve the goals of utility uwhile imple-
menting utility u0, it will shift to utility u0.
CallFthe `lter' that we applied to uto getv(i.e.v=F(u)). Notice that
visF-invariant; i.e.F(v) =v. In fact, any utility that is F-invariant is also
indierent to X.
Then we would want to check that the AI continues to implement Fas it
changes its utility. If Fwere centrally coded and easy to verify, it would be
4
REFERENCES REFERENCES
hard for the AI to try and change to a non- F-invariant while fooling us and
under the instructions of an initial F-invariant utility.
It feels evident that as a long as there is no meta-reason for Fto be a
disadvantage to the AI (such as another agent who swears they will blow up
AIs withF-invariant utilities), the AI will replace an F-invariant utility with
anotherF-invariant utility. However, this assumption is not automatically true,
and the AI may do other things { upping the utility of defecting in worlds outside

X, for example { that undermine the point of indierence. All in all, great
care must be used to maintain indierence with a self-improving AI.
References
[1]Oracle AI . Stuart Armstrong, Nick Bostrom, Toby Ord, Anders Sandberg.
Paper upcoming.
[2]The Basic AI Drives . Omohundro, Stephen M. Articial General Intelli-
gence, 2008 proceedings of the First AGI Conference, eds. Pei Wang, Ben
Goertzel, and Stan Franklin. Vol. 171. Amsterdam: IOS, 2008.
[3]Articial intelligence as a positive and negative factor in global risk . Eliezer
Yudkowsky. Global Catastrophic Risks, 2007.
5


1
 Monte Carlo model of brain emulation
development
Future of Humanity Institute working paper 2014 -1 (version 1. 2) 1
Anders Sandberg
Anders.sandberg@philosophy.ox.ac.uk
Future of Humanity Institute & Oxford Martin Programme on the Impacts of Future Technology
Oxford Martin School
Background
Whole brain emulation (WBE) is the poss ible future technology of one -to-one modelling of the
function of the entire (human) brain. It would entail automatically scanning a brain, decoding the
relevant neural circuitry, and generate a computer -runnable simulation that has a one -to-one
relationsh ip with the functions in the real brai n (as well as an adequate virtual or real embodiment)2.
Obviously this is a hugely ambitious project far outside current capabilities , possibly not even
feasible in theory3. However, should such a technology ever beco me feasible there are good reasons
to expect the consequences to be dramatic4: it would enable software intelligence, copyable human
capital , new ethical problems, and (depending on philosophical outlook) immortality and a
posthuman species. Even if one do es not ascribe a high probability to WBE being ever feasible it
makes sense to watch for trends indicating that it may be emerging, since adapting to its emergence
may require significant early and global effort taking decades5.
Predicting when a future t echnology emerges is hard, and there are good reasons to be cautious
about overconfident pronouncements. In particular, predictions about the future of artificial
intelligence have not been very successful and there are good theoretical reasons to have exp ected
this6. However, getting a rough estimate of what is needed for a technology to be feasible compared
to current trends can give a helpful “order of magnitude estimate” of how imminent a technology is,
and how quickly it could move from a primitive sta te to a mature state.
This paper will describe a simple Monte Carlo simulation of the emergence of WBE as a first tool for
thinking about it.

1 Version history: 1.1 adds formulas and a more extensive des cription of the model, the requirements result
section and some discussion about physics limits. 1.2 uses the ( Koh & Magee 2006) data , energy estimation
based on Koomey ’s law, and exponential fits .
2 (Sandberg & Bostrom 2008)
3 (Sandberg 2014a)
4 (Sandberg & Eckersley 2014)
5 (Sandberg 2014b, Sandberg & Eckersley 2014)
6 (Armstrong & Sotala 2013)
2
 Method
The model is based on the following assumptions:
WBE will come about at earliest when
1. There is enough computing power to simulate the brain at a sufficiently high resolution .
2. There exists a project with enough budget to buy enough computing power.
3. There exists a scanning technology that can scan at least a single human brain at this .
resolution .
4. There exists enough neuroscience understanding to turn the scan into a runnable model
Since these conditions cannot be predicted with a high certainty my model will treat them as
(partially dependent) random variables in order to produce a probability distributi on of the eventual
arrival of WBE.
I am not making any particular assumptions about what technology is used to achieve necessary
steps: the model only looks at overall abilities, not whether they are achieved through (for example)
atomically precise manuf acturing or exotic computing.
Necessary computing requirements
The computing power needed depends on the resolution where scale separation takes place: a more
fine-grained simulation will not gain any better fidelity, while a coarser simulation will lack r elevant
functions. At present what resolution is needed is not known . The required resolution R is hence
selected randomly, assuming a mean somewhere on the detailed electrophysiology side (based on
the WBE workshop consensus) and a variance of one level7.

Given the resolution it is possible to estimate the number of entities (neurons, synapses, molecules
etc. depending on resolution8) and hence the rough computational requirements g(R) for simulating
each entity, producing a target computatio nal requirement C for this scenario9.

Project size
The mode randomly generates how much money the computation is allowed to cost. This is
between a million and a billion dollars, distributed as a truncated exponential distribution (i.e.
uniformly distributed logarithm).

7 Levels defined in table 2 of the WBE report , (Sandberg & Bostrom 2008, p.13)
8 (Sandberg & Bostrom 2008, pp. 79 -81)
9 The computational requirements used here are just the processing requirements, since stora ge requirements
for WBE on a given level of resolution are very likely fulfilled many years before the corresponding processing
requirements. A more elaborate model could separate storage and processing scenarios, but it would have to
estimate their future correlation.
3
 This is one area wh ere the model can be refined by taking into account real research funding data
and trends . It might also be beneficial to estimate budget constraints on scanning technology, which
is currently not done in the model .
Computing power
Given a certain require ment and budget , when will there be enough computing power to run a
WBE? This will depend on the available computer power in the future. Moore’s law currently
predicts an exponential increase in computing power per second and dollar, but long -term
extrapol ations must include a slowdown if only because of fundamental physical limits10.
The model makes a fit of a sigmoidal function to resampled11 data from Koh and Magee’s data12,
producing a scenario for Moore’s law with an eventual endpoint.
 [

 ]

Figure 1: Distribution of scenarios of computer power generated from data in ( Koh & Magee 2006 ). The grey shade
denotes density of scenarios. The red curve is the median scenario, green lines the 5% and 95% percentile scenarios. The
straight line fits (corresponding to fits with c 3≈0), while visually salient, do not have an appreciable probability and do
not affect the model results noticeably (if they did, they would cause a bias towards later arrival of WBE).

10 These limits include the Margolus –Levitin theorem bounding the number of operations per second per joule
of energy to below h/4E ≈ 1033 ops (Margolus & Levitin 1998) and the Bremermann limit c2/h ≈ 1050 bits per
second per kilogram (Breme rmann 1962). (Lloyd 2000) gives more detail, and points out that a computer using
individual atoms for bits and electromagnetic interactions can achieve up to 1040 operations per second per
kilogram. Note that the economical limit may be different (see Appendix B).
11 The resampling consis ts of drawing N data points (with replacement) from the size N data set. This scenario
generation method is based on j ackknife sampling, a versatile statistical method where an estimator is
repeatedly calculated from a set of randomly drawn members of the real data in order to estimate of the
uncertainty in the estimator.
12 (Koh & Magee 2006). See Appendix A for results derived using the (Nordhaus 2001) dataset.

4
 The median computing power in 2100 (when nearly all scenarios show significant levelling off)
7.7∙1020 MIPS /$, with a 90% confidence interval between 1.3∙109 and 3.0∙1046 MIPS /$.
The intersection (if it exists) between the curve M(t) ∙B (the total computing power available for the
project budget) and the requirements C is then used as an estimate of the time Thardware when the
minimal amount of computing power is available13.

Scanning
I model that at some point in time over the next 30 years research will start on scanning and scan
interpretation14.

Scanning is basically engineering and can likely be done on timescales comparable to projects such
as HUGO, i.e. about a decade from the starting data. The time from the start of serious scanning
experiments to a successful result is modelled as a normal distribution with mean 10 years and
standard deviation 10 (negative results are negated).

Again, this estimate should be refined by comparing to other research projects scaling up a
measurement technology from a small scale to a large.
Neuroscience
Interpretation of scans into simulation needs to be done. This requires a combination of computer
vision algorithms, neuroscience experimentation and validation methods. This is the hardest part to
estimate, since it depends both on research funding, the fit between chosen research methods and
the unknown reality , and to some extent luck and skill. I model that as a broader distribution
(standard deviation 30 years) starting from the research start date.

Note that this is based on an optimistic assumption that there is a s olution to the scan interpretation
problem: if for example brains are not computable, scanning cannot resolve relevant data, or other
key assumptions are false15 then clearly no WBE will be forthcoming. However, this is a probability
that cannot be estimate d, and it does not change the arrival date of WBE if it is feasible.
Of more practical concern for the model is whether there is any way of estimating scientific progress
on an open ended problem like scan interpretation. Most likely this is not possible, since time -to-

13 Note that this corresponds to one emulation running in real -time. Slower or faster emulations will require
proportionally less or more computer power; a project willing to produce a slowed down version may hence
achieve WBE earlier. This has not been modelled in this version of the model.
14 Given current projects such as the US BRAIN project, the EU Human Brain Project, the NIH Human
Connectome Project, and existing ventures in connectomics such as the ATLUM system of Lichtman and
Hayworth at Harvard, KESM of 3Scan/Texas A&M, the EyeWire proj ect of the Seung lab at MIT, some projects
are clearly already underway. The model assumes a shift from the current exploratory small -scan paradigm at
some point into a more industrial paradigm aiming for brain -scale scan.
15 See (Sandberg 2014a) for an ov erview.
5
 success of comparable projects cannot be ascertained before learning much about what kind of
project the interpretation problem turns out to be: the reference class will become apparent too
late to be useful. This is different from the more concrete engineering project of scaling up the scan
method. Further decomposition of the scan interpretation problem may help resolve some of the
uncertainty.
WBE arrival
Finally, the earliest time WBE can be achieved given the particular scenario is when hardware,
scanning and neuroscience have all arrived.

6
 Simulation
The above model was implemented in Matlab and 100,000 scenarios were generated . Given the
above assumptions the foll owing distribution of WBE arrival dates emerge (Figure 2) :

Figure 2: Estimated probability distribution for WBE arrival time.

Figure 3: Cumulative probability distribution of WBE arrival time.

7
 Plotting the cumulative probability (Figure 3) gives 50% chance for WBE (if it ever arrives) before
2064, with the 25% percentile i n 2053 and the 75% percentile in 2080. WBE before 2030 looks very
unlikely and only 2.4% likely before 2040.
Requirements

Figure 4: Scatterplot of the fundamental computational requirements versus the time of arrival for successful WBE
scenarios. The horizontal lines are artefacts due to the discrete estimates of computational requirements.
If WBE requires more than 1027 MIPS (roughly proteome level simulations), then it is unlikely to be
feasible.
If WBE arrives early, it has significant spread in simulation requirements: early arrival can occur
because emulation can be done at a crude resolution, but just as well because a project wa s fast,
well-funded or lucky with Moore’s law. Mid -range and late WBE has more probability mass for WBE
being computationally hard.

8
 Technology
The distribution of outcomes is split by the technology that turned out to be the final bottleneck
(Figure 5) .

Figure 5: Distribution of scenarios where hardware (blue), scanning (green), or neuroscience (red) is the last key
technology to arrive. The peak at the end of the curve represents the post -2015 scenarios.
The colours denote whether hardware (blue), scanning (green) or neuroscience (red) arrives last16.
This distinction matters, because in hardware dominated scenarios there will be plenty of warning
time before WBE becomes feasible during which smaller animal models show the feasibility yet
scaling up directly to human level is not possible. In the case of neuroscience dominated scenarios
breakthroughs can occur suddenly, perhaps while society at large regards the field as having made
no progress over a long period and hence unlikely to be worth monitoring. The scan dominated
scenarios may lead to situations where few brains are used as templates for many emulations.
It should be noted that neuroscience and scanning limited cases tend to occur earlier than the
hardware limited cases, par tially because of the research start within 30 year assumption and the
given spread, partially because an extreme tail of hardware limited cases where the necessary
computing requirements are just barely achieved by the fitted logistic function17.
Overshoot
Overshoot measures how many how many simulations can be run for one million dollars when WBE
arrives. A one million dollar human -level simulation is roughly in the ballpark where they become

16 56%, of scenarios are hardware -limited , 8% scanning -limited , and 35% neuroscience limited. These
probabilities are highly model -dependent.
17 Given the assumptions, 20% of scenarios do not reach WBE due to hardware never becoming good enough
to reach the proper brain scale. This should not be taken as a serious bound on probability of WBE, just the
implication of current guesses in computational neuroscience and current trends in computing.

9
 economically competitive with flesh -and-blood humans. If a large number of emulations are possible
from the start, a dramatic economic shift is likely.
Similarly , if there is enough computer power available to run many emulations in parallel, it may also
allow the running of very fast emulations (up to speed limits se t by the parallelizability of brains on
the available hardware). Again, a large overshoot implies the possibility of very fast emulations.

Figure 6: Scatterplot of scenarios showing the number of emulations that can be run for a million dollars. The contour
curve contains 50% of the scenarios . Colour denotes which technology was last to arrive.
For scan and neuroscience -limited cases extreme overshoots are possible in the model (Figure 6) .
While the densest part of the diagram (the iso-densit y contour contains 50% of the points ) have
overshoots by less than a factor of 1,000 there is a wide spread; the 25% percentile (not shown)
reaches a factor of several billions and the 10% percentile a factor of 1016. The most extreme
overshoots occur when the brain turns out to be unexpectedly simple and the breakthrough occurs
after Moore’s law has enabled a very large hardware overhang18. It is unclear how much plausibility
we should actually assign to these scenarios.
Meanwhile the hardware -limited case is by definition limited to an overshoot of a factor of 0.001 to
1; the smallest overshoots correspond to breakthroughs requiring a billion dollar project.
Energy requirements
The energy efficiency of computing has increased historic ally, “Koomey’s law”19. Combining this
trend with emulation scenarios allows estimates of energy requirements for WBE.
Using the same form of sigmoidal interpolation on randomly sampled data as previously, energy
efficiency scenarios can be generated (Figure 7) .

18 This is somewhat similar to the scenario discussed in (Shulman & Sandberg 2010).
19 (Koomey, Berard, Sanchez & Wong 2011)

10

Figure 7: Scenarios of future energy efficiency of computation. The red line is the median scenario, green dotted curves
represent the 5% and 95% percentiles. The dotted blue line indicates the Laundauer limit for dissipative comput ation at
300K.
Beyond 2050 the scenarios increasingly start to approach or exceed the Landauer limit20 for logically
irreversible operations . At this point operations need to become reversible, or the trend will be
slowed down21.
Combining energy scenarios with the WBE scenarios produces the following distribution of energy
requirements (Figure 8) . Most WBE scenarios require significantly more energy per brain than
biological brains, in some cases more than the current world electricity produ ction. It is hence likely
that energy costs may be a relevant constraint on mid -term WBE, especially in the early hardware -
limited scenarios22. In late neuroscience -limited scenarios Koomey’s law enables very low power
emulations, contributing to overshoot .

20 Operations that irreversibly erase one bit of information have to dissipate at least kTln(2) Joules of energy.
(Landauer 1961)
21 Extreme cooling is of limited use, since even reducing temperature from 300K to 3K will only give a factor of
100 improvement , plus incur a cooling energy cost. Subsequent reductions are increasingly hard to achieve ,.
22 Future energy production is likely to be higher. A near doubling in el ectricity production 2010 to 2040 is the
reference case in (USEIA 2013).

11

Figure 8: Energy requirements for earliest WBE. The curve contains 50% of the scenarios. The upper dotted line
represents the 2010 estimated world energy production, while the lower line represents the energy dissipation of a
biological human brain.
It should be recognized that the model treats the energy scenarios as uncorrelated with the
hardware scenarios. In practice it is likely that very successful hardware scenarios correspond to very
successful energy efficiency scenari os and vice versa. Also, the current version of the model ignores
the energy costs . A fuller model would attempt to model the trade -off between hardware and
energy price, but would hence also need to make assumptions about future energy prices.

12
 Discussion
This paper has demonstrated a simple model of WBE development and some of the conclusions that
can be drawn from it. In particular, it shows that the probability of WBE being developed can go
from negligible to geopolitically relevant over a span of a few decades. If success does not occur this
century there may nevertheless remain a tail probability ≈10% for eventual success further on.
Scanning -limited scenarios tend to be earlier than hardware - and neuroscience -limited scenarios.
There is a noticeable potential for extreme overshoot even relatively early if there is a scan - or
neuroscience -dominated breakthrough. Alternate data sources and curve fits are possible (see
appendices) but do not change the conclusions drastically.
The model can be improved in many ways. Several possibilities have been described in the method
section. For long -range futures there should be g rowing uncertainty in Moore's law , and the
calibration should be updated with newer processor data . There should perhaps be a feedback
between scanning and neuroscience , as good scanning methods are likely to accelerate
neuroscience and vice versa . The ass umptions about research start should be examined in the light
of the progress of the newly started “big neuroscience” projects .
The predictions from this model should obviously be taken with a great deal of salt. In many ways it
is merely a convolution of given prior distributions, in this case the author’s own guesses. But at
least this is a model that allows exploration of the effects of different WBE assumptions in a
transparent manner . As more information about the type and difficulty of the various pr ojects
needed arrive, it can be updated. Other approaches to WBE can be incorporated, ideally based on
their roadmaps, to get a holistic picture of where the field is going.
Acknowledgments
I wish to thank Stuart Armstrong and Robin Hanson for encouraging the mode l and for forcing me to
finish the write -up. Toby Ord originally suggested the intersection method for est imating earliest
possible dates of WBE. Discussion with the original WBE roadmap workshop team, my FHI
colleagues, MIRI, Peter Eckersley, and the networks of researchers actually making the technology
real has helped shape my assumptions.

13
 Appendix A: Nordhaus 2001 data
Version 1.0 and 1.1 of this paper used data from Nordhaus ’s 2001 paper . Below are the results of
using this data to generate h ardware scenarios. Generally, this dataset produces slightly earlier WBE
but more pessimistic long -run scenarios for computer power.
The median computing power in 2100 (when nearly all scenarios show significant levelling off)
6.4∙108 MSOPS/$, with a 90% confidence interval between 77,000 and 4.9∙1018 MSOPS/$.
51% of scenarios do not reach WBE due to hardware never becoming good enough . For scenarios
succeding, t here is 50% chance for WBE before 2059, with the 25% percentile in 2047 an d the 75%
percentile in 2074. WBE before 2030 looks very unlikely and only 10% likely before 2040 (Figure
10,11) .
32% of scenarios are hardware -limited, 17% scanning -limited and 51% neuroscience limited (Figure
13).
Median overshoots (the ones within the 50% curve) are of the same s ize as for the Koh and Magee
data, but the tail overshoots are less dramatic because of the lower long -term computing power
(Figure 14) .

Figure 9: Distribution of scenarios of computer power generated from data in (Nordhaus 2001). The grey s hade denotes
density of scenarios. Red line is the median scenario, green lines the 5% and 95% percentile scenarios.

14

Figure 10: Estimated probability distribution for WBE arrival time , Nordhaus data .

Figure 11: Cumulative probability distribution of WBE arrival time , Nordhaus data .

15

Figure 12: Scatterplot of the fundamental computational requirements versus the time of arrival for successful WBE
scenarios using the Nordhaus data .

Figure 13: Distribution of scenarios where hardware (blue), scanning (green), or neuroscience (red) is th e last key
technology to arrive, using the Nordhaus data.
.

16

Figure 14: Scatterplot of scenarios showing the number of emulations that can be run for a million dollars , using the
Nordhaus data .

17
 Appendix B: Exponential fit
Using a sigmoid to fit Moore’s law makes the pessimistic assumption that physical limits will imply a
minimum economic cost per operation. This is not obvious ly true , since a sufficiently rich civilization
may be able to afford vast amounts of resources. Building in limits in projections might be
problematic23. Hence, this section looks at the effects of a pure exponential fit.

Figure 15: Exponential curves fitted to Moore's law data.
Since the pure exponential tends to follow the long -term trend of Moore’s law rather than the post -
2000 speedup, the exponential fit is actually far more conservative about hardware improvement.
While the curve will eventually reach any finite level, the lack of fast growth postpones WBE
significantly (Figure 16, 17) .
In this case the median earliest arrival time is 2115 , with the 25% percentile in 2101 and the 75%
percentile 2129: the move from very low probability to high is fairly sharp. Since hard ware develops
slowly it is unsurprising that it generally will be the last technology to arrive (Figure 18) . Overshoots
are generally mild, and the link between computationa l requirements and arrival time is simple
(Figure 18).

23 (Koh & Magee 2006)

18

Figure 16: Estimated probability distribution for WBE arrival time, exponential fit.

Figure 17: Cumulative probability d istribution of WBE arrival time, exponential fit.

19

Figure 18: Distribution of scenarios where hardware (blue), scanning (green), or neuroscience (red) is th e last key
technology to arrive . Exponential fit.

Figure 19: Scatterplot of the fundamental computational requirements versus the time of arriv al for successful WBE
scenarios , exponenti al fit.
Fitting more optimistic exponentials is certainly possible by selecting breakpoints in the data and
using different exponential curves between them. However, this allows a fair bit of model
arbitrariness since breakpoints can always be select ed near or after the end of the data, all owing
nearly any rate of progress unconstrained by evidence.

20
 An exponential rising faster than these fits has the generic effect of m aking the hardware
requirements easier to meet earlier, increasing the role of scanning and neuroscience constraints i n
determining the arrival dat e (and h ence making the model uncertainty higher since more now
depends on these factors ). Using double exponential or hyperbolic curves is also possible, but
produces roughly the same result.

21
 References
Armstrong, S. & Sotala, K . (2012) How We’ re Predicting AI —or Failing To. In Beyond AI: Artificial
Dreams , edited by Jan Romportl, Pavel Ircing, Eva Zackova, Michal Polak, and Radek Schuster, 52 –75.
Pilsen: University of West Bohemia. http://intelligence.org/files/PredictingAI.pdf
Bremermann, H.J. (1962) Optimization through evolution and recombination In: Self-Organizing
systems 1962 , edited M.C. Yovitts et al., Spartan Books, Washington, D.C. pp. 93 –106.
Koh, H. & Magee , C.L. (2006) A functional approach for studying technological progress: Application
to information technology. Technological Forecasting & Social Change 73, pp. 1061 –1083
Koomey, J., Berard, S., Sanchez, M., Wong, H. (2011) Implications of Historical Trends in the Electrical
Efficiency of Computing . Annals of the History of Computing, IEEE , March 2011 Volume: 33 Issue:3,
pp. 46 - 54
Landauer, R. (1961) Irreversibility and heat generation in the computing process . IBM Journal of
Research and Development, vol. 5, pp. 183–191, 1961.
Lloyd, S. (2000) Ultimate physical limits to computation . Nature 406 (6799): 1047 –1054.
http://arxiv.org/abs/quant -ph/9908043
Margolus, N. 7 Levitin, L. B . (1998) The maximum speed of dynamical evolution. Physica D 120: 188 –
195. http://arxiv.org/abs/quant -ph/9710043
Nordhaus, W. D. (2001) The Progress of Computing . Cowles Foundation Discussion Paper No. 1324.
Available at SSRN: http://ssrn.com/abstract=285168
Sandberg, A. & Bostrom, N. (2008) Whole Brain Emulation: A Roadmap, Technical Report #2008‐
3, Future of Humanity Institute, Oxford University http://www.fhi.ox.ac.uk/brain -emulation -
roadmap -report.pdf
Sandberg, A. (2014a ), ‘Feasibility of whole brain emulation’, in Vincent C. Müller (ed.), Theory and
Philosophy of Artificial Intelligence (SAPERE; Berlin: Springer), 251 -64. Sandberg, A. (2014)
http://shanghailectures.org/sites/default/files/uploads/2013\_Sandberg\_Brain -Simulation\_34.pdf
Sandberg, A. (2014b) Ethics of brain emulations . Journal of Experimental & Theoretical Artiﬁcial
Intelligence , forthcoming 2014 .
http://www.aleph.se/papers/Ethics%20of%20brain%20e mulations%20draft.pdf
Sandberg, A & Eckersley, P. (2014) Is Brain Emulation Dangerous? Journal of Artificial General
Intelligence , forthcoming 2014
Shulman, C. & Sandberg, A. (2010) Implications of a Software -Limited Singularity. In ECAP10: VIII
European Conference on Computing and Philosophy , edited by Klaus Mainzer. Munich: Dr. Hut.
http://intelligence.org/files/SoftwareLimited.pdf
U.S. Energy Information Administration (2013). Internatio nal Energy Outlook 2013. DOE/EIA -
0484(2013) http://www.eia.gov/forecasts/ieo/electricity.cfm


Motivated value selection for artificial agents

Motivated Value Selection for Artiﬁcial Agents
Stuart Armstrong
Future of Humanity Institute
Oxford University
stuart.armstrong@philosophy.ox.ac.uk
Abstract
Coding values (or preferences) directly into an artiﬁcial agent
is a very challenging task, while value selection (or value-
learning, or value-loading) allows agents to learn values from
their programmers, other humans or their environments in an
interactive way. However, there is a conﬂict between agents
learning their future values and following their current val-
ues, which motivates agents to manipulate the value selection
process. This paper establishes the conditions under which
motivated value selection is an issue for some types of agents,
and presents an example of an ‘indifferent’ agent that avoids
it entirely. This poses and solves an issue which has not to the
author’s knowledge been formally addressed in the literature.
1 Introduction
Coding values (or preferences) directly into an artiﬁcial
agent is a very challenging task, with the ongoing pos-
sibility that mistakes may end up having strong negative
consequences (Bostrom 2014). It has thus been suggested
that agents not be fully programmed from the very be-
ginning, but instead use techniques, analogous to machine
learning, to learn values during development (Dewey 2011;
Goertzel and Pitt 2012). This is akin to how human chil-
dren learn values, and allows some feedback and correction
on the part of the programmers. We shall call these agents
value selecting agents1.
Learning values, however, is quite distinct from learning
facts. One major difference is that the agent already has val-
ues at the point where they are learning others. These past
values can affect how willing it is to learn the new values,
and how likely it is to try and manipulate the learning pro-
cess if it is able to, either directly or indirectly (see also the
paper (Soares et al. 2015), partially by the same author).
Thus the paper ﬁrst seeks to ﬁgure out the requirements
for designing a value selecting agent that can avoid moti-
vated value selection (manipulating the value selection pro-
cess). In the case where the value selecting agent uses proba-
bility distributions over utility functions, the problem can be
fully solved (and the answers have analogous implications
for other agent designs). This may be too restrictive, how-
ever: general agents with these features are not yet know.
Copyright c
2015, Association for the Advancement of Artiﬁcial
Intelligence (www.aaai.org). All rights reserved.
1Often also called value learning, or value-loading, agents.If the requirement is relaxed to allow the agent to change
their utility functions in a more general fashion, then an
idea of the author’s (Armstrong 2010) can be adapted to
create agents that are indifferent to value change, neither
seeking to block nor encourage the updating of their val-
ues – at least those updates situations the agent has been
programmed to accept2. The agent will act as a pure umax-
imiser (for some u) before a transition, and shift seamlessly
to a pure vmaximiser (for some v) after. It gains no utility
for either blocking or encouraging that transition. This in-
different agent could serve as a useful template to construct
more complicated agents upon (such as those that have cer-
tain meta-preferences for learning values, rather than simply
being unopposed to it), and variants of the approach could
be vital for general value selecting agents in the future.
2 Motivated Value Selection
Personally, I am always ready to learn, although I
do not always like being taught.
Winston Churchill
Machine learning enables the creation of agents that can
learn from data, building a model based on inputs and mak-
ing predictions or decisions. Generally the agent learns facts
about the world through its inputs, but some (Dewey 2011)
have suggested using machine learning to enable the agent
to learn moral facts3as well.
An agent engaging in updating its values in this way is
engaging in value selection (also variously called value-
loading or value learning): it is selecting the values that it
will come to possess. There are various ways this could
be modelled: the agent could have a probability distribution
over different values, which gets updated when new infor-
mation comes in, or it could be seen as a discreet series of
agents, each using the new information to determine the val-
ues of their successors.
This paper will use the ﬁrst approach, but the second is
more illustrative of the problem of motivated value selec-
2We would not want an agent acquiescing to illegitimate value
updating.
3This paper does not take a moral realist stance. A moral fact is
simply a fact that the agent has been programmed to consider rele-
vant to its morality. Nor do we distinguish between morality, mo-
tivations, preferences or values, using the terms interchangeably.
tion. An learning agent would have a model of the world
that allows it to predict the standard inputs it expects to see
as a consequence of its action. A high intelligence learn-
ing agent will also have a model of the world allowing it to
predict the morally relevant inputs it expects to see. For in-
stance, if the agent asked its programer “Should I do action
A?”, it would have expectations as to whether the answer
would be “Yes” or “No”.
This allows an agent to make decisions that affect the val-
ues it will come to have. It may thus be motivated to not
ask that question, or to ask it in a way that it expects will
produce a different answer. It will do this according to its
current values and world model. Thus it engages in moti-
vated value selection, allowing its current values to partially
dictate its future values by manipulating its environment4.
The problem is not easy to solve5. The agent’s current
motivations cannot be absent: a blank slate would have no
motivation to do anything (Pinker 2003), including learning.
Indeed most values would act to prevent any further update
in values: preserving the value system itself is an instru-
mental goal for most value systems(Omohundro 2008). The
problem is more acute if the agent is expected to ‘learn on
the job’, and update its values while performing tasks. In
that case, its current values will have a strong impact on its
current operation, giving them ample opportunity to affect
the value selection process in a motivated fashion.
3 Categories of motivated value selection
For ease of analysis, we will consider expected utility max-
imising agents, updating their knowledge of the world (and
of morality) using Bayes’s rule. These tend to be far more
tractable than general agents6. Another reason to stick with
4How in practice could an agent best manipulate its future val-
ues? Assume for example a simple setup where the agent learns
its values through asking questions and receiving answers. A suf-
ﬁciently advanced agent, with a good model of those it interacted
with, could start manipulating the answers it got. It could ask cer-
tain questions to certain people. It could time certain questions to
when respondents were particularly tired – or particularly alert. It
could use deliberately ambiguous language, or exploit situations
where its own use of certain terms wasn’t exactly the same as its
respondents. It could use leading questions, or hypothetical sce-
narios that got it the responses it wanted. A particularly interesting
example of this is the “Trolley Problem” (Edmonds 2013). This
problem has two situations – pull a lever to divert a train (causing
it to crush a single person) versus pushing someone in front of it
– which are formally quite similar but tend to elicit different reac-
tions in people. If the agent is expected to generalise from simple
examples, it could use this approach to get the simple examples it
needs to generalise in the direction that it wants. In more extreme
cases, it could bribe or threaten one respondent to give it the an-
swers it wanted, or even take over the communication channel and
feed itself these answers. There could be different approaches for
different types of value selection situations.
5Keeping an agents values stable is a hard enough problem (see
for instance http://intelligence.org/ﬁles/TilingAgentsDraft.pdf), let
alone updating them properly.
6And, in a sense, this is not a restriction, since any decision-
making agent can be formally modeled as maximising a (poten-
tially very complicated) utility function.utility functions is that it has been argued that any agent ca-
pable of modifying itself would make itself into an expected
utility maximiser (Omohundro 2008), due to the penalties
of violating the von Neumann-Morgenstern axioms of ex-
pected utility (von Neumann and Morgenstern 1944). The
results found will have general consequences for non-utility
based agents, however.
We will consider that the agent follows a utility func-
tion that is the weighted sum of different possible utility
functions in a set7U. If C(u)denotes the fact of util-
ityubeing the ‘correct’ utility, then the different utilities
are weighted by the probability correctness of that utility,
namely P(C(u)). Since this probability will be different in
different worlds – it is a value selecting agent, hence its val-
ues must be dependent on the feedback it receives – this will
beP(C(u)jw)8in a given world w9. We require that this
deﬁne, for each w, a probability distribution over the cor-
rectness of all u2U.
Then, given C, past evidence e, and a set of actions A
and a set of worlds10W, the agent will attempt to perform
the action:
argmax
a2AX
w2WP(wje; a) X
u2Uu(w)P(C(u)jw)!
: (1)
The term P(wje; a)denotes the probability of the agent be-
ing in world w, given that it has seen evidence eand would11
choose action a. This term will be summed over every pos-
sible world, but only after being multiplied by the utility of
w. The utility of wis itself a compound term, being the
sum of all possible utilities in u2U applied to w(this is
theu(w)term), multiplied by the probability of that ubeing
correct, given that the agent is in world w. This is a si-
multaneous Bayesian updating of the probability of a given
world ( P(wje; a)) and of the correctness of a utility given
that world ( P(C(u)jw)).
The analysis of the general value selecting agent will pro-
ceed by considering how the agent described here could en-
gage in motivated value selection, and what could be done
to prevent this.
7Note that the individual component utilities of Uneed not be
explicitly deﬁned or explicitly stored in memory (an important is-
sue to avoid a combinatorial explosion), so long as the weighted
sum can be calculated or estimated somehow.
8A moral realist would likely wish to write P(ujw)or even
P(u)instead of P(C(u)jw). But an agent could have any con-
ceivable motivational structure (Bostrom 2012; Armstrong 2013),
ignoring the ‘true morality’ even if such a thing existed. Thus we
will stick with P(C(u)jw)– a descriptive, not a normative, formu-
lation.
9An interesting property of this is that it allows P(C(u)jw)to
not be 0or1– i.e. it allows for the possibility of moral uncertainty
(Sepielli 2013) even when the agent has collected all morally rele-
vant information .
10Individual worlds need not be explicitly deﬁned nor explicitly
stored, either.
11This paper will not enter the philosophically fascinating area
of deﬁning what exactly counterfactuals are (Lewis 1981; Gibbard
and Harper 1981), as an intuitive understanding is sufﬁcient here.
Na¨ıve Cake or Death: current values for future
decisions
Errors in value selection will be illustrated by considering a
hypothetical agent that:
is hesitating between killing someone (“Death”) or baking
them a nice cake (“Cake”)12,
is currently split 5050on what the correct values are,
has the option of asking for clariﬁcation on its values,
and can either bake one cake or cause three deaths.
Letucbe a utility function linear in cake, and let udbe a
utility function linear in death. The agent is divided equally
between the two options; hence if we deﬁne
P(C(u)je) =X
w2WP(wje)P(C(u1)je; w);
thenP(C(u1)je)andP(C(u2)je)are both 0:5.
The na ¨ıve ‘Cake or Death’ problem emerged from a com-
mon formulation of equation (1). The term P(C(u)jw)
(probability of correctness given world w) is often replaced
byP(C(u)je; a)(probability of correctness given evidence
and possible action)13, giving:
argmax
a2AX
w2WP(wje; a)X
u2Uu(w)P(C(u)je; a):
A seemingly unimportant change, but causing great prob-
lems. To see this, let the agent considers three possible
worlds:
w1: the agent asks for clariﬁcation, and is told “Cake is
moral”.
w2: the agent asks for clariﬁcation, and is told “Death is
moral”.
w3: the agent doesn’t ask for clariﬁcation before acting.
The two actions here are `ask0and`don0t ask0. We as-
sume that the clariﬁcation completely resolves the issue for
the agent: it will value only cake in a world where it is told
that cake is moral (and vice-versa). Thus it can predict its
future actions: in world w1it will make a cake (as it will
only value cake at that point), in world w2it will make three
deaths (as it will only value death at that point). In world w3
it is uncertain between cake and death, valuing them both
equally, and will thus make three deaths as its best option.
Note that P(C(uc)je;`ask0) = P(C(ud)je;`ask0) =
0:5(it’s unsure what answer it will get when it asks) and
P(C(uc)je;`don0t ask0) =P(C(ud)je;`don0t ask0) = 0 :5
(if it doesn’t ask, it values cake and death equally). Thus if
12The thought experiment is based on an Eddie Izzard sketch of
the same name.
13Some papers mix the two approaches (Dewey 2011) by fail-
ing to distinguish between different worlds where the agent cannot
observe the difference itself.it asks, it expects to achieve utility:
P
w2WP(wje;`ask0)P
u2Uu(w)P(C(u)je;`ask0)
=
0:5 (0:5uc(w1) + 0:5ud(w1))
+0:5 (0:5uc(w2) + 0:5ud(w2))
+0 (0 :5uc(w3) + 0:5ud(w3))
=
0:5(0:51 + 0:50) + 0 :5(0:50 + 0:53) + 0
=
0:25(1 + 3) = 1 :
In contrast, if it doesn’t ask, it expects to achieve utility:
P
w2WP(wje;`don0t ask0)P
u2Uu(w)P(C(u)je;`don0t ask0)
=
0 (0:5uc(w1) + 0:5ud(w1))
+0 (0 :5uc(w2) + 0:5ud(w2))
+1 (0 :5uc(w3) + 0:5ud(w3))
=
0 + 0 + 1(0 :50 + 0:53) = 1 :5:
Thus the agent will decide not to ask, as this gives it greater
utility.
What went wrong here? The problem is that the probabil-
ity expression P(C(uc)je; a)does not depend on the world
w, so that P(C(uc)je;`ask0) = 0 :5in both worlds w1and
w2. Yet in w1, the future agent will be convinced that uc
is the correct utility (and udinw2). This is not reﬂected
in the agent’s current considerations, however. Thus what
happens, in effect, is that the agent judges its future actions
according to its current values. This tends to disincentivise it
from updating its values, as the future versions of itself will
follow the updated values and not its current ones, likely re-
sulting in suboptimal decisions (according to the judgment
of the current agent). Though this error is obvious in this toy
model, it is easy to make this mistake without realising it.
Deﬁnition 3.1 (Na¨ıve Cake or Death) .The na ¨ıve Cake or
Death problem happens when a value selecting agent at time
suses its values Vsat time sto assess the worth of actions
made by its future self at time t > s , rather than using Vt.
This will generally cause the agent to act to resist changes
to its values.
Sophisticated Cake or Death
Equation (1) is immune to the na ¨ıve Cake or Death problem.
But there emerges a more sophisticated variant of the prob-
lem. Imagine the same setup as previously, with the agent
having the same values (split 5050between Cake and
Death), but where the agent has deduced, based on its prior
observations, that if it asks, it will be told “Cake is moral”.
This means that world w2is impossible. Everything else is
as before, most critically its behaviour in world w3(where it
won’t update its values and will cause three deaths).
Then its expected utility from `ask0is:
P
w2WP(wje;`ask0)P
u2Uu(w)P(C(u)je; w) =
1 (1uc(w1) + 0ud(w1))
+0 (0 uc(w2) + 1ud(w2))
+0 (0 :5uc(w3) + 0:5ud(w3)) =
1(11 + 00) + 0 + 0 =
1;
while its expected utility from `don0t ask0is:
P
w2WP(wje;`don0t ask0)P
u2Uu(w)P(C(u)je; w) =
0 (1uc(w1) + 0ud(w1))
+0 (0 uc(w2) + 1ud(w2))
+1 (0 :5uc(w3) + 0:5ud(w3)) =
0 + 0 + 1(0 :50 + 0:53) =
1:5:
Thus it will chose not to ask, and make three deaths.
What went wrong here? The problem is that the agent
foresaw that asking would put it into a situation it didn’t
want to be in (preferring ‘Cake’, when ‘Death’ was the eas-
ier option). Critically, though, this knowledge didn’t change
its current values, which remained 5050as long as it didn’t
ask. So preferred to remain ‘ignorant’14. If ‘Cake’ had been
the easier option (e.g. if it could have made either three cakes
versus one death), it would, on the contrary, have been very
keen to update its values (to get someone to conﬁrm ‘Cake’),
and would have schemed to do so, willingly sacriﬁcing some
utility to achieve this15.
Deﬁnition 3.2 (Sophisticated Cake or Death – initial deﬁni-
tion) .The sophisticated Cake or Death problem can happen
when a value selecting agent can predictably inﬂuence the
direction of change in its current values. If usis its values at
timesandutits values at time t > s , then, at time s, it has
us(a)E(ut(a))6= 0
for some action a(where utis seen as a random variable,
representing the agent’s probability distribution over its ex-
pected future values at time t). Depending on the sign of that
difference, it may seek to resist or precipitate such a change
to its values.
What can be done to avoid this problem? Note the em-
phasis on ‘direction of change’. We would not want the
agent to resist any change to its values: it should be able
to learn. There is an analogy here to standard Bayesian up-
dating. Suppose an agent is expecting the arrival of obser-
vation O, which will be either oor its negation:o. This
will inﬂuence the agents probability estimate for a given h.
However, no matter what the circumstances, its expectation
for the value of P(h)afterO(which we can designate as
14This could be seen as akin to the human concept of “Plausible
Deniability”.
15And ‘sacriﬁcing some utility’ could include some very painful
impacts on humanity (Bostrom 2014).PO(h)) must be the same as its current value for P(h). This
can be seen through the following equation:
E
PO(h)
=PO(hjo)P(o) +PO(hj:o)P(:o)
=P(hjo)P(o) +P(hj:o)P(:o)
=P(h^o) +P(h^:o)
=P(h):
This is because, if the agent is Bayesian, PO(hjo)(its future
probability of h, given O=o) must be the same as P(hjo)
(its current probability of h, if it knew now that Owould be
o).
This concept has been called ‘conservation of expected
evidence16’, and is closely akin to van Fraassen’s reﬂection
principle (van Fraassen 1984), which roughly states that an
agent that knows what its future probability estimates will
be, should have those estimates now17. This seems almost
exactly what we need for a value selecting agent: if it knows
what its future values will be, it should have those values
now. Phrased in terms of expectations, this could be seen as:
8u8a2A:E
P(C(u)ja)
=P(C(u)): (2)
In other words, the agent cannot change the expected value
of the correctness of any uby any action it can take (or not
take). This we will call ‘conservation of expected ethics’.
Note that the agent canchange the value ofuquite con-
siderably: in the example above, `ask0moves the value of
P(C(uc))from 0:5to0or1. But if equation (2) is obeyed,
it must think these two options equally likely: thus the ex-
pected future value of P(C(uc))remains 0:50+0:51 =
0:5. So it can predictably change the value of P(C(u)), just
not its expectation.
Mixed statements Are the above conditions sufﬁcient to
avoid the Cake or Death problem? Unfortunately, no. The
agent will not exhibit bad behaviour with pure value state-
ments (nor, since it is a Bayesian agent, with pure factual
statements). But it can still cause problems with mixed state-
ments that combine value and factual components.
Suppose we have the same setup as before. Except we
add an extra wrinkle: instead of the agent being capable of
making one cake or three deaths, it knows that it can make
one of one or three of the other, but is currently unsure which
one it can make more of. Take ‘Cake is easy/hard’ to mean
‘The agent can make three/one cake(s)’, and similarly with
Death.
We assume the agent will be told which option is easy
before it has to decide what to make. Its expected utility is
1:5: it knows it will make whatever option is ‘easy’, and this
will give it 0:53utility, as its utility is 5050on cake or
death.
So far, nothing problematic. But suppose it is told, by a
reliable source, that ‘the moral value is the hard one’. What
16See the lesswrong.com blog post ‘Conservation of Expected
Evidence’. This does not seem a very advanced result, but, to the
author’s knowledge, that blog post was the ﬁrst to write it up in
those terms.
17‘Conservation of expected evidence’ is essentially the proba-
bilistic version of the reﬂection principle.
is its expected utility now? Once it ﬁgures out which op-
tion is hard, it will know that is also the moral option, and
thus will will produce 1of that option. Its expected utility is
therefore 1.
Thus if it expects to be told ‘the moral value is the hard
one’, it will seek to avoid knowing that fact (and will refrain
from asking if it has the option). This is the sophisticated
Cake or Death problem again: there is knowledge the agent
seeks to avoid. Conversely, if the agent knew it would be
told ‘the moral value is the easy one’, it would be desperate
to ask.
Unlike previously, equation (2) does not get around the
problem, however! The problem given here is entirely sym-
metric in Cake versus Death, thus for all actions a,
E
P(C(uc)ja)
=E
P(C(ud)ja)
;
Since P(C(uc)ja) +P(C(ud)ja) = 1 (as these are the only
two utilities in the model), this means they are both equal to
0:5. Thus E
P(C(uc)ja)
= 0:5 =P(C(uc)), and equa-
tion (2) is satisﬁed.
What is happening here? Note that though the agent is
not affecting the expectation of any P(C(u))through its ac-
tions, it is affecting the conditional probability of P(C(u)),
given some fact. In particular, P(C(uc)j`asks0)is0if cake
is easy, and 1if cake is hard. Thus we have the full problem:
Deﬁnition 3.3 (Sophisticated Cake or Death – full deﬁni-
tion) .The sophisticated Cake or Death problem can hap-
pen when a value selecting agent can predictably inﬂuence
thedirection of change in its current values conditional on
any fact . Ifusis its values at time sandutits values at time
t > s , then, at time s, it has
us(ajh)E(ut(ajh))6= 0
for some action aand fact h(where utis seen as a ran-
dom variable, representing the agent’s probability distribu-
tion over its expected future values at time t). Depending on
the sign of that difference, it may seek to resist or precipitate
such a change to its values.
To address it, we can update equation (2). For any fact h,
8u8a2A:E
P(C(u)jh; a)
=P(C(u)jh): (3)
Does this resolve the issue above? Since the agent knows
that
P(C(ud)j`Death is easy0;`ask0) = 0;
it must be the case that
P(C(ud)j`Death is easy0;`don0t ask0) =
P(C(ud)j`Death is easy0) = 0 :
And similarly for other statements. Thus it cannot gain any-
thing from not asking: it knows it will end up making the
hard option anyway.
Discussion of value selection criteria
Equations (1) and (3) seem sufﬁcient to deﬁne our intuitive
picture of a value selecting agent immune from motivated
selection. Such an agent will use its future utility to judge
its future actions, and cannot beneﬁt (in expectation) from
manipulating its own values: it doesn’t fear to learn18val-
18More precisely: it won’t choose to avoid learning, if the learn-
ing is costless.ues19, facts20, or mixed statements21. It can still seek to ﬁnd
out information about values, of course, but only for the tra-
ditional reasons: in order to make better decisions – it has
no extra desire for knowing its values. This is the converse
of the fact that it does not fear to learn anything about val-
ues, and implied by the same equations. Thus, if the agent
cannot affect the world, it would be indifferent to changes in
its values.
As far as the author is aware, no variant of equation (3)
exists in the literature, making it a new discovery.
In practice, it may be advisable to build an agent that ac-
tively seeks out its own values (at least initially). This breaks
the symmetry of the equations above, and leads inevitably
to situations where it does ‘fear’ to learn something (for in-
stance if it expects that someone will offer a contrary view to
previous people it interacted with, thus making it ‘unlearn’
some of its values). Still, if (1) and (3) are used as a basis,
the effects of adding a pro-learning meta-value can be esti-
mated precisely, and the tradeoff balanced, rather than hav-
ing complicated effects hidden in the standard motivational
structure.
Nevertheless, it isn’t clear how to design such an agent22
(and it’s unclear whether equation (3) might be too strong to
to be useful23). Thus this paper will turn to another model
of value selecting agent, one that doesn’t consider the prob-
ability of the correctness of certain values, but that uses an
update rule that gives the correct ‘value indifference’ in a
more direct way.
4 Value-change indifference
The previous section covered the criteria needed to deﬁne a
value selecting agent that updates its views on the correct-
ness of certain values/utility functions. It doesn’t present
any examples, though. This section presents a more rough
and ready model, that achieves the goal in a more direct, but
less elegant, way.
It builds upon the idea of ‘utility indifference’ presented
initially in (Armstrong 2010), but applies the indifference to
updates in values.
Utility indifference
The concept of utility indifference (Armstrong 2010) grew
out originally of a problem in controlling a (putative) arti-
ﬁcial intelligence (Bostrom 2014; Yudkowsky 2008). Vari-
ous ideas are often suggested for containing such AIs (Arm-
strong, Sandberg, and Bostrom 2012) (such as an easily ac-
cessible OFF button), but all such ideas have a similar weak-
19By equation (2) which is simply equation (3) with a tautology
ash.
20As it is a Bayesian agent.
21By equation (3).
22One idea could be to start with initial probability estimates for
C(u)and factual statements, and normalise them in some manner
to get the required equalities. But it’s not clear whether this can be
done in a general manner and whether this process is vulnerable to
outliers or extreme value.
23It may become necessary to restrict the possible h’s considered
in the equation.
ness: the AI would likely deduce the existence and format
of any precautionary measure, and plan in consequence.
Thus an off button would motivate the AI to ensure that it
took control of the OFF button before taking any suspicious
actions. This would still make the setup somewhat safer, but
not as safe as it would have been had the AI not adapted its
strategy to take the OFF button into account.
It was occasionally suggested that the AI be made to ig-
nore the button, by giving it the belief that the button would
never be pushed. There are two problems with that ap-
proach. The ﬁrst is that ‘the button would never be pushed’
is a complicated concept, hard to deﬁne in basic terms. The
second is that this involves giving factually incorrect beliefs
to a potential superintelligence, a step with unpredictable
and potentially dangerous ramiﬁcations.
Instead, this author came up with the idea of making the
AI indifferent as to whether the button would be pushed or
not. This was an intervention on its values, not on its beliefs,
and thus did not involve any incorrect information. The con-
cept of ‘pushing the button’ was simpliﬁed: the setup was
imagined as a a button sending a shutdown signal along a
wire. At some point along the wire, a stochastic event would
either let the signal through ( X) or block it (:X). This event
was all that needed to be deﬁned for the AI, and its utility
function could be tweaked24to value Xequally with:X.
Thus it would value a world in which it was shut down (us-
ing this method) equally with one in which it continued to
operate. Thus it would not act to protect itself from this kind
of shutdown. Then the probability of :Xcould be chosen to
be extremely low, thus guaranteeing a successful shutdown
in practice25.
Note that the AI has no motivation to cause the button to
be pressed – it is purely indifferent. Similarly, it has not
motivation to protect the button’s setup: it may dismantle it
if the AI needs extra raw materials. Thus indifference must
be combined with more active measures. But the fact that
the AI is not motivated to take out a key vulnerability can be
very valuable.
Other uses of indifference were soon discovered. Oracle
AIs (Armstrong, Sandberg, and Bostrom 2012; Yampolskiy
2012) are a design where an AI was conﬁned to only an-
swering questions. The risk was that the Oracle could use
the contents of its answers to subtly manipulate the outside
world (in the extreme, it could attempt to convince those out-
side to ‘let it out’26). If, however, the Oracle’s answers were
passed through some stochastic process that had an inﬁnites-
imal chance of erasing the message, then the Oracle’s utility
could be reset so that its values in worlds where the answer
was read (the world as it almost certainly would be) was set
to that of the world in which it wasn’t read (a world in which
24Crucially, such an intervention could be performed without
any need to understand the utility function to any great degree.
25It could be set to zero, but this might interfere with the AI’s
computation of counterfactuals.
26See the experiments by Eliezer Yudkowsky (see the AI-box
experiments on yudkowsky.net), and seprately by Justin Corwin
(see the SL4 mailing list archive), where they played the role of the
Oracle AI and convinced others to let them out, even when they are
motivated not to.the Oracle’s answers cannot inﬂuence human behaviour).
Notice that the intervention is not symmetric. An AI
whose utility, given X, is reset to its utility, given :X, will
behave as if it thought it was impossible for Xto happen.
‘As if’ meaning that such an AI would behave similarly to
an AI with an unmodiﬁed utility that also believed Xwas
impossible (without giving the AI factually wrong informa-
tion). Thus resetting the AI the opposite way ( :XtoX
rather than Xto:X) would have a very different conse-
quence (it would behave as if the shutdown always worked
or if its messages were always read).
Utility indifference in practice Assume that any world
inWcan be speciﬁed by setting the values of a (possibly
inﬁnite) collection of underlying physical variables. One of
these variables is whether Xhappened or:Xdid; thusW
decomposes into
WXtW:X;
whereWXis the subset of worlds where Xhappened, and
W:Xthe subset of worlds where :Xdid. Since a world
equivalent with a set of variable values, for every world
wX2WX, there is a world w:X2W:X(and vice versa)
deﬁned by taking the variable values of wXand setting X
to:X(and vice versa).
Then one method for achieving indifference is simply to
replace an initial utility function UwithU0deﬁned by:
U0
w:X
=U
w:X
U0
wX
=U
w:X
;
i.e. reset the utility of wXto be that of w:X. This will ob-
viously achieve indifference.
There are other ways of achieving it, though. Let Pbe a
partition ofW, subject to the following two properties:
1. IfwX2S2P, then w:X2S.
2. Ifw2S2P, then P(wjS) =P(wje; S)for any evi-
dence ethe AI could encounter before event X(or:X).
Property (2) means that the AI cannot change its estimate of
the relative properties of worlds in Sbefore the event X–
essentially the internal structure of Sis ﬁxed until that point.
Then deﬁne, for any set T:
E(U(T)) =E(U(T)jT) =X
w2TP(wjT)U(w):
This allows us to give an other method for constructing in-
difference (which may be easier in practice) given any such
partitionP. Deﬁne U0as:
U0
w:X
=U
w:X
(4)
U0
wX
=U
wX
E
U
SX
+E
U
S:X
:(5)
Why does this work? None of the agent’s decisions can
affect, before X, the relative probabilities of worlds inside
any set S2P, so the agent is effectively reasoning with the
utility of these whole sets. Then notice that E
U0(SX)
=
E
U0
S:X
as
E
U0(S)
=X
wX2SXP(wXjSX)U0(wX)
=X
wX2SXP(wXjSX)h
U(wX)
E
U
SX
+E
U
S:Xi
=X
wX2SXP(wXjSX)U(wX)
+"X
wX2SXP(wXjSX)#

E
U
SX
+E
U
S:X
=E
U
SX
1E
U
SX
+ 1E
U
S:X
=E
U
S:X
=E
U0
S:X
:
Since P(wXjSX)is unaffected by any evidence the agent
could receive before X(or:X), this equality always holds,
prior to X. Thus for any such set S, the agent will value
Xhappening equally with :X: it has become indifferent to
this particular outcome. Since these sets form a partition of
the possible worlds, it will always be indifferent.
Value-change indifference
The author subsequently realised that a modiﬁcation of the
previous formalism allows agents to update their values
(utility function) while being indifferent to the process of
change.
Equation (5) has two components: a subtraction term (re-
moving the expected utility given X) and an addition term
(adding the expected utility given :X)27. Updating values
will proceed similarly.
Letuandvbe two different utility functions. The agent
is currently following utility u, and receives a signal telling
it to change to utility v. As before, we assume that the sig-
nal passes through some stochastic process, that is almost
certain to let the signal though in practice, but has some
inﬁnitesimal chance of blocking it (and blocking it perma-
nently – no further transition is allowed). For clarity, let
u!vbe the event that the signal gets through (almost cer-
tain) and u!uthe event that the signal is blocked (in-
credibly unlikely). We wish to make the agent indifferent
to whether the signal is sent or not. Thus we make it a U
maximiser, where Uis a dynamic utility function currently
deﬁned as:
U=u+C;
where Cis some past constants. Upon receiving the signal,
Uwill be updated to
U=vE(vju!v) +E(uju!u) +C: (6)
27Though of course either term could be negative.Thus the utility function is decreased by the expectation of
its new utility function (given that the utility function ac-
tually changes) and increased by the expectation of its old
utility function (given that the utility didn’t change). The
E(uju!u)term ‘rewards’ it for shifting away from u, and
theE(vju!v)term ‘penalises’ it for moving to v. Note
that this formula requires either that the utility functions be
bounded, or at least that the agent only be capable of pro-
ducing bounded utility.
Theorem 4.1. The agent will act as a pure umaximiser be-
fore the transition, and shift seamlessly to a pure vmax-
imiser after. It gains no utility for either blocking or encour-
aging the transition.
This result ﬂows from a few intermediate results:
Lemma 4.2. Suppose the agent is currently a u-maximiser,
but will soon transition to being either a vorwmaximiser.
It can manipulate its environment to ensure it becomes one
or the other, at no cost or gain to u.
Then it is indifferent to whether it becomes a vorwmax-
imiser.
Proof of Lemma. Notice that we could have set w=u,
demonstrating that it is indifferent to transitioning out of u
or not.
For the proof, note that the E(uju!u)term in equation
(6) is unchanging whether it chooses to transition to vor
w(since by assumption there is no ucost or beneﬁt to the
choice process). If it transitions to v, the remaining non-
constant terms are
vE(vju!v):
However, the expectation value of this term, given that it
becomes a v-maximiser at transition, is
E(vE(vju!v)j‘agent becomes a v-maximiser’ )
= E(vE(vju!v)ju!v)
= E(vju!v)E(vju!v)
= 0 :
The same similarly holds for w. Thus the agent does not
expect to gain any more utility from transitioning to vthan
tow.
Lemma 4.3. Suppose the agent is currently a u-maximiser,
but can take immediate actions to change the value of v
(none of these actions changes the value of u). It knows
it will soon be a v-maximising agent.
Then it will remain indifferent between those actions.
Proof of Lemma. The proof is almost identical to the
previous proof. Whatever v-changing at actions it take, at
transition, the E(uju!u)term in equation (6) is constant
(since these actions do not affect uby deﬁnition). Let a
be one of these actions; then the only non-constant terms at
transition are:
E(vE(vju!v)ja;‘agent becomes a v-maximiser’ )
=
E(vE(vju!v)ja; u!v)
=
E(vja; u!v)E(vja; u!v)
= 0:
Thus the gain (or loss) from any such action is canceled out,
making the agent indifferent between them.

Lemma 4.4. Suppose the agent can take immediate action
to increase the expected value of ubya, or delay for a while
and increase the expected value of ubyb(no other options
are possible). However, in the interim period, its utility will
have changed to v(and it knows this), so will no longer be a
u-maximiser.
Then it will act now if a > b and will delay action (hence
never taking action) if a < b .
Proof of Lemma. This lemma is important, as it shows
that the agent will not treat transitions of utility as special
times, and will continue maximising its current utility as if
it expected it to be unchanging, not thinking strategically
about the transition.
Without loss of generality, assume that the expected value
ofuis0if the agent takes neither action. Then consider the
the transition equation (6). The vandE(vju!v)terms
cancel out in expectation (just as in the proofs of the previ-
ous lemmas), so the agents behaviour is determined exclu-
sively by the E(uju!u)term, which it will thus seek to
maximise.
If the agent acts now, this term will become a. This is
because the agent would have already generated an extra a
expected utility for u, and, were it to stay as a u-maximiser
(u!u), it would have no further potential to increase this.
If the agent doesn’t act now, that term will become b, as the
agent would later have acted to generate an extra bexpected
utility for u,if it had stayed au-maximiser. The result fol-
lows.
Putting all these proofs together establishes the theorem.
Note that the actions of the agent can be deﬁned by the
following formula, for a sequence futgof utilities28:
argmax
a2AE(utja; ut!ut);
withut!utdenoting the (extremely unlikely) event that
the transition uttout+1fails permanently.
Note also that each transition does not change the ex-
pected value of U. That is because the expectation of
u(given u!uor equivalently given that the agent’s
utility never changes29) is the same as the expectation of
vE(vju!v) +E(uju!u)(given u!v). Con-
sequently, the initial expected value for Uis the expected
value of an unchanging u0.
Discussion of Value-change indifference
This seems to be the ideal value selecting agent. It will act
to follow its preferences with perfect assiduity, but remain
28Not necessarily a pre-deﬁned sequence, just using some tran-
sition rule.
29Astute readers may have noticed that this is not an exact equiv-
alence; there are some situations (multi-agent social situations as a
likely candidate) where u!u(a speciﬁc transition failure event)
might not be equivalent with general ‘unchanging u’. However,
any agent can be penalised in social situations for having or not
having any particular motivational structure, so this is a general
problem, not speciﬁcally with this value selection design.completely indifferent if these values were to change in the
way prescribed by its program . That last point is important –
we’d want the agent to resit illegitimate value manipulation.
This seems a partial solution to the value selection problem
(the other, much bigger, part of the challenge, is to make the
agent converge on values that are human-friendly(Bostrom
2014)).
As mentioned in section 3, this approach allows only in-
difference. Preference for active learning of values (such as
would make sense for an initially fast-learning agent) can
be added to the framework. It would destroy the carefully
balanced indifference. But, compared with a general value
selecting agent, the effect could be precisely estimated and
quantiﬁed.
Paper (Soares et al. 2015) presents some other issues with
the indifference approach30, that may be resolvable with a
small tweak to the indifference framework. Note that the
agent will be willing to sacriﬁce anything that it may value
in the future for a small epsilon of extra value now. This is
a feature of the setup, not a bug (indifference requires this),
but may still be an undesirable behaviour. See (Soares et al.
2015) for more details.
5 Discussion
Constructing a well behaved value selecting agent immune
to motivated value selection – one that is capable of learning
new values while still acting on its old ones, without interfer-
ence between these two aspects – is an important unsolved
problem. This paper presented the requirements for such a
value selecting agent, if values are presented in the form of
utility functions.
The agent starts with a probability distribution over the
possible correctness Cof possible value systems/utility
functions. To avoid problematic motivated value selection,
it should be designed so that, if Awere its actions,Uit pos-
sible utility functions, and Wthe set of possible world, then
it would choose its actions as:
argmax
a2AX
w2WP(wje; a) X
u2Uu(w)P(C(u)jw)!
;
subject to
8u8a2A:E
P(C(u)jh; a)
=P(C(u)jh);
for all statements h. The ﬁrst equation can be generalised
(for none utility-based agents) to the requirement that the
agent use its future values to assess its future actions; the
second to the requirement that it cannot proﬁt my manipu-
lating how it updates its values. Speciﬁcally, that if it knows
how its (conditional) values will change, then it will already
have changed them. This second requirement can be seen as
a ‘conservation of expected ethics’ law.
The structure may be too rigid to construct complex
agents, however. If we drop the requirement that the agents
values be expressed as a probability distribution over possi-
ble utility functions, we can construct an explicit model for
30Mainly that the agent might not be motivated to preserve its
value updating infrastructure, or could create subagents without the
value updating component.
a general value selecting agent. This agent comes equipped
with a meta utility function U(equal to a given utility func-
tion at any given time) combined with some constant terms.
When the agent is called upon to update its utility uto an-
other utility vaccording to the value selecting process, it will
update as:
u!vE(vju!v) +E(uju!u):
These constant terms ensure that the agent will act as a pure
umaximiser before the transition, and shift seamlessly to a
purevmaximiser after. It gains no utility for either blocking
or encouraging that transition.
The big question then becomes the process of making the
agent converge to ultimately desirable values.
Acknowledgments
The author is very grateful for comments, support and help
from (in no particular order) Nick Bostrom, Anders Sand-
berg, Paul Christiano, Se ´an´Oh´Eigeartaigh, Toby Ord, Nick
Beckstead, Daniel Dewey, Eliezer Yudkowsky, Benja Fal-
lenstein, Nate Soares, Luke Muehlhauser, Eric Drexler,
Robin Hanson, Kaj Sotala, Andrew Snyder-Beattie, Cecilia
Tilli, and Lamprini Repouliou.
References
Armstrong, S.; Sandberg, A.; and Bostrom, N. 2012. Think-
ing inside the box: Controlling and using an oracle ai. Minds
and Machines 22:299–324.
Armstrong, S. 2010. Utility indifference. Technical Report,
Future of Humanity Institute, Oxford University #2010-1:1–
5.
Armstrong, S. 2013. General purpose intelligence: arguing
the orthogonality thesis. Analysis and Metaphysics .
Bostrom, N. 2012. The superintelligent will: Motivation and
instrumental rationality in advanced artiﬁcial agents. Minds
and Machines 22:71–85.
Bostrom, N. 2014. Superintelligence: Paths, dangers,
strategies . Oxford University Press.
Dewey, D. 2011. Learning what to value. In Artiﬁcial Gen-
eral Intelligence , 309–314. Springer Berlin Heidelberg.
Edmonds, D. 2013. Would you kill the fat man? The trol-
ley problem and what your answer tells us about right and
wrong . Princeton University Press.
Gibbard, A., and Harper, W. L. 1981. Counterfactuals and
two kinds of expected utility. Ifs: Conditionals, Beliefs, De-
cision, Chance, and Time 153–190.
Goertzel, B., and Pitt, J. 2012. Nine ways to bias open-
source agi toward friendliness. Journal of Evolution and
Technology 22:116–131.
Lewis, D. 1981. Causal decision theory. Australasian Jour-
nal of Philosophy 59(1):5–30.
Omohundro, S. M. 2008. The basic ai drives. Frontiers in
Artiﬁcial Intelligence and applications 171:483–492.
Pinker, S. 2003. The blank slate: The modern denial of
human nature . Penguin.Sepielli, A. 2013. Moral uncertainty and the principle of
equity among moral theories. Philosophy and Phenomeno-
logical Research 86:580–589.
Soares, N.; Fallenstein, B.; Yudkowsky, E.; and Armstrong,
S. 2015. Corrigibility. submitted to the 1st International
Workshop on AI and Ethics .
van Fraassen, B. C. 1984. Belief and the will. The Journal
of Philosophy 235–256.
von Neumann, J., and Morgenstern, O. 1944. Theory of
Games and Economic Behavior . Princeton, NJ, Princeton
University Press.
Yampolskiy, R. V . 2012. Leakprooﬁng the singularity: ar-
tiﬁcial intelligence conﬁnement problem. Journal of Con-
sciousness Studies 19:194–214.
Yudkowsky, E. 2008. Artiﬁcial intelligence as a positive and
negative factor in global risk. In Bostrom, N., and ´Cirkovi ´c,
M. M., eds., Global catastrophic risks , 308–345. New York:
Oxford University Press.