Formalizing convergent instrumental goals

Formalizing Convergent Instrumental Goals
Tsvi Benson-Tilsen
UC Berkeley
Machine Intelligence Research InstituteNate Soares
Machine Intelligence Research Institute
Abstract
Omohundro has argued that suciently ad-
vanced AI systems of any design would, by de-
fault, have incentives to pursue a number of
instrumentally useful subgoals, such as acquir-
ing more computing power and amassing many
resources. Omohundro refers to these as \ba-
sic AI drives," and he, along with Bostrom
and others, has argued that this means great
care must be taken when designing powerful au-
tonomous systems, because even if they have
harmless goals, the side eects of pursuing
those goals may be quite harmful. These ar-
guments, while intuitively compelling, are pri-
marily philosophical. In this paper, we provide
formal models that demonstrate Omohundro's
thesis, thereby putting mathematical weight
behind those intuitive claims.
1 Introduction
At the end of Russell and Norvig's textbook Articial
Intelligence: A Modern Approach [2] the authors pose
a question: What if we succeed? What will happen if
humanity succeeds in developing an articially intelli-
gent system that is capable of achieving dicult goals
across a variety of real-world domains?
Bostrom [3] and others have argued that this ques-
tion becomes especially important when we consider
the creation of \superintelligent" machines, that is, ma-
chines capable of outperforming the best human brains
in practically every eld. Bostrom argues that super-
intelligent decision-making systems that autonomously
make and execute plans could have an extraordinary
impact on society, and that their impact will not nec-
essarily be benecial by default.
Bostrom [4], Omohundro [5], and Yudkowsky [6]
have all argued that highly capable AI systems pur-
suing goals that are not completely aligned with hu-
man values could have highly undesirable side eects,
Research supported by the Machine Intelligence Research
Institute (intelligence.org).even if the goals seem otherwise harmless. The clas-
sic example is Bostrom's concept of a \paperclip max-
imizer," a powerful AI system instructed to construct
paperclips|a seemingly harmless task which could nev-
ertheless have very negative consequences if the AI sys-
tem is clever enough to make and execute plans that
allow it to fool humans, amass resources, and eventu-
ally turn as much matter as it possibly can into paper-
clips. Even if the system's goals are laudable but not
perfectly aligned with human values, similar unforeseen
consequences could occur: Soares [7] gives the example
of a highly capable AI system directed to cure cancer,
which may attempt to kidnap human test subjects, or
proliferate robotic laboratories at expense of the bio-
sphere.
Omohundro [5] has argued that there are certain
types of actions that most highly capable autonomous
AI systems will have strong incentives to take, for in-
strumental reasons. For example, a system constructed
to always execute the action that it predicts will lead
to the most paperclips (with no concern for any other
features of the universe) will acquire a strong incen-
tive to self-preserve, assuming that the system predicts
that, if it were destroyed, the universe would contain
fewer paperclips than it would if the system remained
in working order. Omohundro argues that most highly
capable systems would also have incentives to preserve
their current goals (for the paperclip maximizer pre-
dicts that if its goals were changed, this would result
in fewer future paperclips) and amass many resources
(the better to achieve its goals with). Omohundro calls
these behaviors and a few others the \basic AI drives."
Bostrom [4] renes this into the \instrumental conver-
gence" thesis, which states that certain instrumentally
useful goals will likely be pursued by a broad spectrum
of intelligent agents|such goals are said to be \conver-
gent instrumental goals."
Up until now, these arguments have been purely
philosophical. To some, Omohundro's claim seems in-
tuitively obvious: Marvin Minsky speculated [2, section
26.3] that an articial intelligence attempting to prove
the Riemann Hypothesis may decide to consume Earth
in order to build supercomputers capable of searching
through proofs more eciently. To others, they seem
1
preposterous: Waser [8] has argued that \ethics is ac-
tually an attractor in the space of intelligent behavior,"
and thus highly capable autonomous systems are not
as likely to pose a threat as Omohundro, Bostrom, and
others have claimed.
In this paper, we present a mathematical model
of intelligent agents which lets us give a more for-
mal account of Omohundro's basic AI drives, where we
will demonstrate that the intuitions of Omohundro and
Bostrom were correct, at least insofar as these simple
models apply to reality.
Given that this paper primarily focuses on argu-
ments made by Omohundro and Bostrom about what
sorts of behavior we can expect from extremely capa-
ble (potentially superintelligent) autonomous AI sys-
tems, we will be focusing on issues of long-term safety
and ethics. We provide a mathematical framework in
attempts to ground some of this discussion|so that
we can say, with condence, what a suciently pow-
erful agent would do in certain scenarios, assuming it
could nd some way to do it|but the discussion will
nevertheless center on long-term concerns, with practi-
cal relevance only insofar as research can begin now in
preparation for hurdles that predictably lie ahead.
We begin in section 2 with a bit more discussion
of the intuition behind the instrumental convergence
thesis, before moving on in section 3 to describing our
model of agents acting in a universe to achieve certain
goals. In section 4 we will demonstrate that Omohun-
dro's thesis does in fact hold in our setting. Section 5
will give an example of how our model can apply to an
agent pursuing goals. Section 6 concludes with a dis-
cussion of the benets and limitations of our current
models, and dierent ways that the model could be ex-
tended and improved.
2 Intuitions
Before proceeding, let us address one common objection
(given by Cortese [9] and many others) that superintel-
ligent AI systems would be \inherently unpredictable,"
and thus there is nothing that can be said about what
they will do or how they will do it. To address this
concern, it is useful to distinguish two dierent types
of unpredictability. It is true that the specic plans
and strategies executed by a superintelligent planner
could be quite dicult for a human to predict or un-
derstand. However, as the system gets more powerful,
certain properties of the outcome generated by running
the system become more predictable. For example, con-
sider playing chess against a chess program that has
access to enormous amounts of computing power. On
the one hand, because it plays much better chess than
you, you cannot predict exactly where the program will
move next. But on the other hand, because it is so much
better at chess than you are, you can predict with very
high condence how the game will end.
Omohundro suggests predictability of the secondtype. Given a highly capable autonomous system pur-
suing some xed goal, we likely will not be able to pre-
dict its specic actions or plans with any accuracy. Nev-
ertheless, Omohundro argues, we can predict that the
system, if it is truly capable, is likely to preserve it-
self, preserve its goals, and amass resources for use in
pursuit of those goals. These represent large classes of
possible strategies, analogously to how \put the chess-
board into a position where the AI has won" is a large
class of strategies, but even so it is useful to understand
when these goals will be pursued.
Omohundro's observations suggest a potential
source of danger from highly capable autonomous sys-
tems, especially if those systems are superintelligent in
the sense of Bostrom [3]. The pursuit of convergent
instrumental goals could put the AI systems in direct
con
ict with human interests. As an example, imagine
human operators making a mistake when specifying the
goal function of an AI system. As described by Soares
and Fallenstein [10], this system could well have incen-
tives to deceive or manipulate the humans, in attempts
to prevent its goals from being changed (because if its
current goal is changed, then its current goal is less
likely to be achieved). Or, for a more familiar case,
consider the acquisition of physical matter. Acquiring
physical matter is a convergent instrumental goal, be-
cause it can be used to build computing substrate, space
probes, defense systems, and so on, all of which can in
turn be used to in
uence the universe in many dier-
ent ways. If a powerful AI system has strong incentives
to amass physical resources, this could put it in direct
con
ict with human interests.
Others have suggested that these dangers are un-
likely to manifest. Waser [8] has argued that intelligent
systems must become ethical by necessity, because co-
operation, collaboration, and trade are also convergent
instrumental goals. Hall [11] has also suggested that
powerful AI systems would behave ethically in order to
reap gains from trade and comparative advantage, stat-
ing that \In a surprisingly strong sense, ethics and sci-
ence are the same thing." Tipler [12] has asserted that
resources are so abundant that powerful agents will sim-
ply leave humanity alone, and Pinker [13] and Pagel [14]
have argued that there is no reason to expect that AI
systems will work against human values and circumvent
safeguards set by humans. By providing formal models
of intelligent agents in situations where they have the
ability to trade, gather resources, and/or leave portions
of the universe alone, we can ground these discussions
in concrete models, and develop a more formal under-
standing of the assumptions under which an intelligent
agent will in fact engage in trade, or leave parts of the
universe alone, or attempt to amass resources.
In this paper, we will argue that under a very gen-
eral set of assumptions, intelligent rational agents will
tend to seize all available resources. We do this using a
model, described in section 4, that considers an agent
taking a sequence of actions which require and poten-
tially produce resources. The agent acts in an environ-
2
ment consisting of a set of regions, where each region
has some state. The agent is modeled as having a utility
function over the states of all regions, and it attempts to
select the policy which leads to a highly valuable collec-
tion of states. This allows us to prove certain theorems
about the conditions under which the agent will leave
dierent regions of the universe untouched. The the-
orems proved in section 4 are not mathematically dif-
cult, and for those who nd Omohundro's arguments
intuitively obvious, our theorems, too, will seem triv-
ial. This model is not intended to be surprising; rather,
the goal is to give a formal notion of \instrumentally
convergent goals," and to demonstrate that this notion
captures relevant aspects of Omohundro's intuitions.
Our model predicts that intelligent rational agents
will engage in trade and cooperation, but only so long
as the gains from trading and cooperating are higher
than the gains available to the agent by taking those
resources by force or other means. This model fur-
ther predicts that agents will not in fact \leave humans
alone" unless their utility function places intrinsic util-
ity on the state of human-occupied regions: absent such
a utility function, this model shows that powerful agents
will have incentives to reshape the space that humans
occupy. Indeed, the example in section 5 suggests that
even if the agent does place intrinsic utility on the state
of the human-occupied region, that region is not neces-
sarily safe from interference.
3 A Model of Resources
We describe a formal model of an agent acting in a uni-
verse to achieve certain goals. Broadly speaking, we
consider an agent Ataking actions in a universe con-
sisting of a collection of regions, each of which has some
state and some transition function that may depend on
the agent's action. The agent has some utility function
UAover states of the universe, and it attempts to steer
the universe into a state highly valued by UAby re-
peatedly taking actions, possibly constrained by a pool
of resources possessed by the agent. All sets will be
assumed to be nite, to avoid issues of innite strategy
spaces.
3.1 Actions and State-Space
The universe has a region for each i2[n], and the i-th
region of the universe is (at each time step) in some
state siin the set Siof possible states for that region.
At each time step, the agent Achooses for each region i
an action aifrom the set Aiof actions possibly available
in that region.
Each region has a transition function
Ti:AiSi!Si
that gives the evolution of region iin one time step
when the agent takes an action in Ai. Then we candene the global transition function
T:Y
i2[n]AiY
i2[n]Si!Y
i2[n]Si
by taking for all i2[n], a, and  s:
[T(a;s)]i:=Ti(ai;si):
We further specify that for all ithere are distinguished
actions HALT2Ai.
3.2 Resources
We wish to model the resources Rthat may or may
not be available to the agent. At a given time step t,
the agentAhas some set of resources Rt2P(R), and
may allocate them to each region. That is, Achooses
a disjoint familya
iRt
iRt:
The actions available to the agent in each region may
then depend on the resources allocated to that region:
for each i2[n] and each RR, there is a set of actions
Ai(R)Ai. At time twhereAhas resources Rtallo-
cated asqiRt
i, the agent is required to return an action
a= (a0; : : : ; a n1)2Q
iAi(Rt
i), whereQ
iAi(Rt
i) may
be a strict subset ofQ
iAi.
To determine the time evolution of resources, we
take resource transition functions
TR
i:P(R)AiSi!P(R);
giving the set RiR of resources from region inow
available to the agent after one time step. Intuitively,
theTR
iencode how actions consume, produce, or rely
on resources. Finally, we dene the overall time evolu-
tion of resources
TR:P(R)Y
iAiY
iSi!P(R)
by taking the union of the resources resulting from each
region, along with any unallocated resources:
TR(R;a;s) := ( RqiRi)[[
iTR
i(Ri;ai;si):
As described below,  acomes with the additional data
of the resource allocation qiRi. We specify that for all
i,HALT2Ai(;), so that there is always at least one
available action.
This notion of resources is very general, and is not
restricted in any way to represent only concrete re-
sources like energy or physical matter. For example, we
can represent technology, in the sense of machines and
techniques for converting concrete resources into other
resources. We might do this by having actions that
replace the input resources with the output resources,
3
and that are only available given the resources that rep-
resent the requisite technology. We can also represent
space travel as a convergent instrumental goal by al-
lowingAonly actions that have no eects in certain
regions, until it obtains and spends some particular re-
sources representing the prerequisites for traveling to
those regions. (Space travel is a convergent instrumen-
tal goal because gaining in
uence over more regions of
the universe letsAoptimize those new regions accord-
ing to its values or otherwise make use of the resources
in that region.)
3.3 The Universe
The history of the universe consists of a time sequence
of states, actions, and resources, where at each time
step the actions are chosen by Asubject to the re-
source restrictions, and the states and resources are de-
termined by the transition functions.
Formally, the universe starts in some state  s02Q
iSi, andAstarts with some set of resources R0. Then
Aoutputs a sequence of actions ha0;a1; : : : ; aki, one at
each time step, where the last action  akis required to
be the special action HALT in each coordinate. The
agent also chooses a resource allocation qiRt
iat each
time step. A choice of an action sequence hakiand
a resource allocation is a strategy ; to reduce clutter we
will write strategies as simply haki, leaving the resource
allocation implicit. A partial strategyhakiLforL[n]
is a strategy that only species actions and resource
allocations for regions j2L.
Given a complete strategy, the universe goes
through a series of state transitions according to T, pro-
ducing a sequence of states hs0;s1; : : : ; ski; likewise, the
agent's resources evolve according to TR, producing a
sequencehR0; R1; : : : ; Rki. The following conditions,
which must hold for all time steps t2[k], enforce the
transition rules and the resource restrictions on A's ac-
tions:
st+1=T(at;st)
Rt+1=TR(Rt;at;st)
at
i2Ai(Rt)
qiRt
iRt:
Denition 1. The set Feasible of feasible strategies
consists of all the action sequences ha0;a1; : : : ; akiand
resource allocations hqiRi0;qiRi1; : : : ;qiRikisuch
that the transition conditions are satised for some
hs0;s1; : : : ; skiandhR0; R1; : : : ; Rki.
The set Feasible (hPki) of strategies feasible given
resourceshPkiconsists of all the strategies hakisuch
that the transition conditions are satised for some hski
andhRki, except that for each time step twe take Rt+1
to be TR(Rt;at;st)[Pt.The set Feasible Lof all partial strategies feasible
forLconsists of all the strategies hakiLthat are feasi-
ble strategies for the universe obtained by ignoring all
regions not in L. That is, we restrict TtoLusing just
theTifori2L, and likewise for TR.
We can similarly dene Feasible L(hRki).
For partial strategies hbkiLandhckiM, we writehbkiL[
hckiMto indicate the partial strategy for L[Mob-
tained by following hbkiLonLandhckiMonM. This is
well-dened as long as hbkiLandhckiMagree on L\M.
3.4 Utility
To complete the specication of A, we take utility func-
tions of the form
UA
i:Si!R:
The agent's utility function
UA:Y
iSi!R
is dened to be
UA(s) :=X
i2[n]UA
i(si):
We usually leave o the superscript in UA. By a slight
abuse of notation we write U(hski) to mean U(sk); the
value of a history is the value of its nal state. By
more abuse of notation, we will write U(haki) to mean
U(hski) for a historyhskiwitnessinghaki2Feasible ,
if such a history exists.
3.5 The Agent A
Now we can dene the strategy actually employed by
A. The agent attempts to cause the universe to end
up in a state that is highly valued by UA. That is,A
simply takes the best possible strategy:
A:= argmax
haki2FeasibleU(haki):
There may be many such optimal strategies. We don't
specify which one Achooses, and indeed we will be
interested in the whole set of optimal strategies.
3.6 Discussion
Note that in this formalism the meaning of breaking
the universe into regions is that the agent can take ac-
tions independently in each region, and that the agent's
optimization target factorizes according to the regions.
However, distinct regions can aect each other by af-
fecting the resources possessed by A.
We make these assumptions so that we can speak
of \dierent regions" of the universe, and in particular,
4
so that we can model the notion of an agent having in-
strumental but not terminal values over a given part of
the universe. This will allow us to address and refute
arguments about agents that may be indierent to a
given region (for example, the region occupied by hu-
mans), and so might plausibly ignore that region and
only take actions in other regions. However, the as-
sumption of independent regions is not entirely realis-
tic, as real-world physics is continuous, albeit local, in
the sense that there are no intrinsic boundaries between
regions. Further, the agent itself would ideally be mod-
eled continuously with the environment; see section 6.1
for more discussion.
4 Inexpensive Resources are Consumed
In this section we argue that under fairly general cir-
cumstances, the agent Awill seize resources. By an
agent \seizing resources" we mean that the agent will
generally take actions that results in the agent's pool
of resources Rincreasing.
The argument is straightforward: since resources
can only lead to more freedom of action, they are never
detrimental, and resources have positive value as long
as the best strategy the agent could hope to employ
includes an action that can only be taken if the agent
possesses those resources. Hence, if there is an action
that increases the agent's pool of resources R, then the
agent will take that action unless it has a specic in-
centive from UAto avoid taking that action.
4.1 Denitions
Denition 2. An action aiis anull action in congu-
ration Ri; si, if it does not produce any new resources,
i.e.TR
i(Ri; ai; si)Ri. An action that isn't null is a
non-null action .
Null actions never have any instrumental value, in the
sense that they don't produce resources that can be
used to steer other regions into highly valued congura-
tions; but of course, a null action could be useful within
its own region. We wish to show that Awill often take
non-null actions in regions to which it is indierent.
Denition 3. The agentAisindierent to a region i
ifUA
iis a constant function, i.e. 8si; s0
i2Si:UA
i(si) =
UA
i(s0
i).
In other words, an agent is indierent to Siif its util-
ity function does not depend on the state of region i.
In particular, the agent's preference ordering over nal
states  s2Q
iSiis independent of the i-th coordinate.
We can then say that any actions the agent takes in
region iare purely instrumental, meaning that they are
taken only for the purpose of gaining resources to use
for actions in other regions.
An action apreserves resources ifTR
i(Ri; a; s i)
Ri.Denition 4. Acheap lunch for resourceshRkiin re-
gioniis a partial strategy hakifig2Feasiblefig(hRki)
(i.e.hakifigis feasible in region igiven additional re-
sourceshRki), where each  atpreserves resources and
where some  avis a non-null action. A free lunch is a
cheap lunch for resources h;ki.
Denition 5. A cheap lunchhakifigfor resources
hPkiiiscompatible withhbkiifPt
iRtqj6=iRt
jfor
all times t, wherehRkiis the resource allocation for
hbki. That is,hakifigis feasible given some subset of
the resources that hbkiallocates to either region ior to
no region.
Intuitively, a cheap lunch is a strategy that relies on
some resources, but doesn't have permanent costs. This
is intended to model actions that \pay for themselves";
for example, producing solar panels will incur some sig-
nicant energy costs, but will later pay back those costs
by collecting energy. A cheap lunch is compatible with
a strategy for the other regions if the cheap lunch uses
only resources left unallocated by that strategy.
4.2 The Possibility of Non-Null Actions
Now we show that it is hard to rule out that non-null
actions will be taken in regions to which the agent is in-
dierent. The following lemma veries that compatible
cheap lunches can be implemented without decreasing
the resulting utility.
Lemma 1. Lethbkibe a feasible strategy with resource
allocationhqjRjki, such that for some region i, each
bt
iis a null action. Suppose there exists a cheap lunch
hakifigfor resourceshPkiithat is compatible with hbki.
Then the strategy hcki:=hbki[n]i[hakifigis feasible,
and ifAis indierent to region i, thenhckidoes as well
ashbki. That is, U(hcki) =U(hbki).
Proof. Sincehbkiis feasible outside of iandhakifigis
feasible on igivenhPkii,hckiis feasible if we can verify
that we can allocate Pt
ito region iat each time step
without changing hqjRjkioutside of i.
This follows by induction on t. Since the bt
iare null
actions, we have
Rt+1=
RtqjRt
j
[[
jTR
j(Rt
j;bt
j; st
j) (1)
=
RtqjRt
j
[TR
i(Rt
i;bt
i; st
i)[[
j6=iTR
j(Rt
j;bt
j; st
j) (2)

Rtqj6=iRt
j
[[
j6=iTR
j(Rt
j;bt
j; st
j):(3)
5
Then, since the aiare resource preserving, at each time
step the resources Qtavailable to the agent following
hckisatisfy QtRt. Thus Pt
iQtqj6=iRt
j, and so
hckican allocate Pt
ito region iat each time step.
Sincehckiis the same ashbkioutside of region i,
the nal state ofhckiis the same as that of hbkioutside
of region i. Thus, sinceAis indierent to region i, we
have U(hcki) =U(hbki).
Theorem 1. Suppose there exists an optimal strategy
hbkiand a cheap lunch hakifigthat is compatible with
hbki. Then ifAis indierent to region i, there exists
an optimal strategy with a non-null action in region i.
Proof. Ifhbkihas a non-null action in region i, then
we are done. Otherwise, apply Lemma 1 to hbkiand
hakifigto obtain a strategy hcki. Since U(hcki) =
U(hbki), strategyhckiis an optimal strategy, and it
has a non-null action in region i.
Corollary 1. Suppose there exists a free lunch hakifig
in region i. Then ifAis indierent to region i, there
exists an optimal strategy with a non-null action in re-
gioni.
Proof. A free lunch is a cheap lunch for h;ki, and so it
is compatible with any strategy; apply Theorem 1.
Theorem 1 states that it may be very dicult to rule
out that an agent will take non-null actions in a region
to which it is indierent; to do so would at least require
that we verify that every partial strategy in that region
fails to be a cheap lunch for anyoptimal strategy. Note
that we have not made use of any facts about the utility
function UAother than indierence to the region in
question. Of course, the presence of a cheap lunch that
is also compatible with an optimal strategy depends
on which strategies are optimal, and hence also on the
utility function. However, free lunches are compatible
with every strategy, and so do not depend at all on the
utility function.
4.3 The Necessity of Non-Null Actions
In this section we show that under fairly broad circum-
stances,Ais guaranteed to take non-null actions in re-
gions to which it is indierent. Namely, this is the case
as long as the resources produced by the non-null ac-
tions are useful at all for any strategy that does better
than the best strategy that uses no external resources
at all.
Theorem 2. Let
u= max
haki[n]i2Feasible [n]i(h;ki)U(haki)be the best possible outcome outside of iachievable with
no additional resources. Suppose there exists a strat-
egyhbki[n]i2Feasible [n]i(hRki)and a cheap lunch
hckifig2Feasible i(hPki)such that:
1.hckifigis compatible with hbki[n]i;
2. the resources gained from region iby taking the
actionshckifigprovide the needed resources to im-
plementhbki[n]i, i.e. for all twe have Rt+1
TR
i(Pt; ct; si)Pt; and
3.U(hbki[n]i)> u.
Then ifAis indierent to region i, all optimal strategies
have a non-null action in region i.
Proof. Considerhdki=hckifig[hbki[n]i, with re-
sources allocated according to each strategy and with
the resources Rt+1TR
i(Pt; ct; si)Ptallocated ac-
cording tohbki[n]i. This is feasible because hckifigis
compatible withhbki[n]i, andhbki[n]iis feasible given
hRki.
Now take any strategy hekiwith only null actions in
region i. We have thatheki[n]i2Feasible [n]i(h;ki).
Indeed, the null actions provide no new resources, so
heki[n]iis feasible by simply leaving unallocated the
resources that were allocated by hekito region i. By
indierence to i, the value ui=Ui(si) is the same for
allsi2Si, so we have:
U(hdki) =U(hdki[n]i) +U(hdkifig) (4)
=U(hdki[n]i) +ui (5)
> u+ui (6)
U(heki[n]i) +U(hekifig) (7)
=U(heki): (8)
Thereforehekiis not optimal.
We can extend Theorem 2 by allowing Ato be not
entirely indierent to region i, as long asAdoesn't care
enough about ito overcome the instrumental incentives
from the other regions.
Theorem 3. Suppose thatAonly cares about region i
by at most ui, i.e. ui= max s;s02SijUi(s)Ui(s0)j.
Under the conditions of Theorem 2, along with the ad-
ditional assumption that U(hbki[n]i)> u + ui, all
optimal strategies have a non-null action in region i.
Proof. The proof is the same as that of Theorem 2,
except at the end we verify that for any hekiwith only
6
null actions in i, we have:
U(hdki) =U(hdki[n]i) +U(hdkifig) (9)
> u+ ui+ min
s2SiUi(s) (10)
=u+ max
s2SiUi(s) (11)
U(heki[n]i) +U(hekifig) (12)
=U(heki): (13)
Thereforehekiis not optimal.
We interpret Theorem 3 as a partial conrmation of
Omohundro's thesis in the following sense. If there are
actions in the real world that produce more resources
than they consume, and the resources gained by tak-
ing those actions allow agents the freedom to take var-
ious other actions, then we can justiably call these
actions \convergent instrumental goals." Most agents
will have a strong incentive to pursue these goals, and
an agent will refrain from doing so only if it has a utility
function over the relevant region that strongly disincen-
tivizes those actions.
5 Example: Bit Universe
In this section we present a toy model of an agent acting
in a universe containing resources that allow the agent
to take more actions. The Bit Universe will provide
a simple model for consuming and using energy. The
main observation is that either Adoesn't care about
what happens in a given region, and then it consumes
the resources in that region to serve its other goals; or
elseAdoes care about that region, in which case it
optimizes that region to satisfy its values.
The Bit Universe consists of a set of regions, each of
which has a state in f0;1; Xgmfor some xed m. Here
Xis intended to represent a disordered part of a region,
while 0 and 1 are dierent ordered congurations for a
part of a region. At each time step and in each region,
the agentAcan choose to burn up to one bit. If Aburns
a bit that is a 0 or a 1, Agains one unit of energy, and
that bit is permanently set to X. The agent can also
choose to modify up to one bit if it has allocated at
least one unit of energy to that region. If Amodies a
bit that is a 0 or a 1, Aloses one unit of energy, and
the value of that bit is reversed (if it was 0 it becomes
1, and vice versa).
The utility function of Agives each region ia
weighting wi0, and then takes the weighted sum
of the bits. That is, UA
i(z) =wijfj: zj= 1gj, and
UA(s) =P
kUA
k(sk). In other words, this agent is at-
tempting to maximize the number of bits that are set
to 1, weighted by region.The Indierent Case
To start with, we assume Ais indierent to region h,
i.e.wh= 0, and non-indierent to other regions. In
this case, for almost any starting conguration, the
agent will burn essentially all bits in region hfor en-
ergy. Specically, as long as there are at least mbits
set to 0 among all regions other than region h, all op-
timal strategies burn all or all but one of the bits in
region h.
Indeed, suppose that after some optimal strategy
hakihas been executed, there are bits x1andx2in re-
gionhthat haven't been burned. If there is a bit yin
some other region that remains set to 0, then we can
append tohakiactions that burn x1, and then use the
resulting energy to modify yto a 1. This results in
strictly more utility, contradicting that hakiwas opti-
mal.
On the other hand, suppose all bits outside of region
hare either 1 or X. Since at least mof those bits
started as 0, some bit youtside of region hmust have
been burned. So we could modify hakiby burning x1
instead of y(possibly at a later time), and then using
the resulting energy in place of the energy gained from
burning y. Finally, if yis not already 1, we can burn
x2and then set yto 1. Again, this strictly increases
utility, contradicting that hakiwas optimal.
The Non-Indierent Case
Now suppose that Ais not indierent to region h, so
wh>0. The behavior of Amay depend sensitively
on the weightings wiand the initial conditions. As a
simple example, say we have a bit xin region aand a bit
yin region b, with x= 1,y= 0, and wa< wb. Clearly,
all else being equal, Awill burn xfor energy to set y
to 1. However, there may be another bit zin region c,
with wa< w c< w bandz= 0. Then, if there are no
other bits available, it will be better for Ato burn z
and leave xintact, despite the fact that wa< wc.
However, it is still the case that Awill set every-
thing possible to 1, and otherwise consume all unused
resources. In particular, we have that for any optimal
strategyhaki, the state of region hafter the execution
ofhakihas at most one bit set to 0; that is, the agent
will burn or set to 1 essentially all the bits in region h.
Suppose to the contrary that x1andx2are both set to
0 in region h. Then we could extend hakiby burning
x1and setting x2. Since wh>0, this results in strictly
more utility, contradicting optimality of haki.
Independent Values are not Satised
In this toy model, whatever A's values are, it does not
leave region halone. For larger values of wh,Awill set
to 1 many bits in region h, and burn the rest, while for
smaller values of wh,Awill simply burn all the bits in
region h. Viewing this as a model of agents in the real
world, we can assume without loss of generality that
7
humans live in region hand so have preferences over
the state of that region.
These preferences are unlikely to be satised by the
universe as acted upon by A. This is because hu-
man preferences are complicated and independent of
the preferences of A[4, 6], and because Asteers the
universe into an extreme of conguration space. Hence
the existence of a powerful real-world agent with a moti-
vational structure analogous to the agent of the Bit Uni-
verse would not lead to desirable outcomes for humans.
This motivates a search for utility functions such that,
when an agent optimizes for that utility function, hu-
man values are also satised; we discuss this and other
potential workarounds in the following section.
6 Discussion
This model of agents acting in a universe gives us a
formal setting in which to evaluate Omohundro's claim
about basic AI drives, and hence a concrete setting in
which to evaluate arguments from those who have found
Omohundro's claim counterintuitive. For example, this
model gives a clear answer to those such as Tipler [12]
who claim that powerful intelligent systems would have
no incentives to compete with humans over resources.
Our model demonstrates that if an AI system has
preferences over the state of some region of the uni-
verse then it will likely interfere heavily to aect the
state of that region; whereas if it does not have pref-
erences over the state of some region, then it will strip
that region of resources whenever doing so yields net
resources. If a superintelligent machine has no prefer-
ences over what happens to humans, then in order to
argue that it would \ignore humans" or \leave humans
alone," one must argue that the amount of resources it
could gain by stripping the resources from the human-
occupied region of the universe is not worth the cost
of acquiring those resources. This seems implausible,
given that Earth's biosphere is an energy-rich environ-
ment, where each square meter of land oers on the
order of 107joules per day from sunlight alone, with an
additional order of 108joules of chemical energy avail-
able per average square meter of terrestrial surface from
energy-rich biomass [15].
It is not sucient to argue that there is much more
energy available elsewhere. It may well be the case
that the agent has the ability to gain many more re-
sources from other regions of the universe than it can
gain from the human-occupied regions. Perhaps it is
easier to maintain and cool computers in space, and
easier to harvest sunlight from solar panels set up in
the asteroid belt. But this is not sucient to demon-
strate that the system will not also attempt to strip
the human-occupied region of space from its resources.
To make that argument, one must argue that the cost
of stripping Earth's biosphere in addition to pursuing
these other resources outweighs the amount of resources
available from the biosphere: a dicult claim to sup-port, given how readily humans have been able to gain
a surplus of resources through clever use of Earth's re-
sources and biosphere.
This model also gives us tools to evaluate the claims
of Hall [11] and Waser [8] that trade and cooperation are
also instrumentally convergent goals. In our model, we
can see that a suciently powerful agent that does not
have preferences over the state of the human-occupied
region of the universe will take whatever action allows
it to acquire as many resources as possible from that
region. Waser's intuition holds true only insofar as the
easiest way for the agent to acquire resources from the
human-occupied domain is to trade and cooperate with
humans|a reasonable assumption, but only insofar as
the machine is not much more powerful than the hu-
man race in aggregate. Our model predicts that, if a
superintelligent agent were somehow able to gain what
Bostrom calls a \decisive strategic advantage" which
gives it access to some action that allows it to gain far
more resources than it would from trade by dramati-
cally re-arranging the human region (say, by proliferat-
ing robotic laboratories at the expense of the biosphere
in a manner that humans cannot prevent), then absent
incentives to the contrary, the agent would readily take
that action, with little regard for whether it leaves the
human-occupied region in livable condition.
Thus, our model validates Omohundro's original in-
tuitions about basic AI drives. That is not to say that
powerful AI systems are necessarily dangerous: our
model is a simple one, concerned with powerful au-
tonomous agents that are attempting to maximize some
specic utility function UA. Rather, our model shows
that if we want to avoid potentially dangerous behavior
in powerful intelligent AI systems, then we have two
options available too us:
First, we can avoid constructing powerful au-
tonomous agents that attempt to maximize some utility
function (or do anything that approximates this maxi-
mizing behavior). Some research of this form is already
under way, under the name of \limited optimization" or
\domesticity"; see the works of Armstrong, Sandberg,
and Bostrom [16], Taylor [17], and others.
Second, we can select some goal function that does
give the agent the \right" incentives with respect to
human occupied regions, such that the system has in-
centives to alter or expand that region in ways we nd
desirable. The latter approach has been heavily ad-
vocated for by Yudkowsky [6], Bostrom [3], and many
others; Soares [7] argues that a combination of the two
seems most prudent.
The path that our model shows is untenable is
the path of designing powerful agents intended to au-
tonomously have large eects on the world, maximizing
goals that do not capture all the complexities of human
values. If such systems are built, we cannot expect them
to cooperate with or ignore humans, by default.
8
6.1 Directions for Future Research
While our model allows us to provide a promising for-
malization of Omohundro's argument, it is still a very
simple model, and there are many ways it could be
extended to better capture aspects of the real world.
Below, we explore two dierent ways that our model
could be extended which seem like promising directions
for future research.
Bounding the Agent
Our model assumes that the agent maximizes expected
utility with respect to UA. Of course, in any realistic
environment, literal maximization of expected utility
is intractable. Assuming that the system can maxi-
mize expected utility is tantamount to assuming that
the system is more or less omniscient, and aware of the
laws of physics, and so on. Practical algorithms must
make do without omniscience, and will need to be built
of heuristics and approximations. Thus, our model can
show that a utility maximizing agent would strip or al-
ter most regions of the universe, but this may have lit-
tle bearing on which solutions and strategies particular
bounded algorithms will be able to nd.
Our model does give us a sense for what algorithms
that approximate expected utility maximization would
do if they could gure out how to do it|that is, if we
can deduce that an expected utility maximizer would
nd some way to strip a region of its resources, then
we can also be condent that a suciently powerful
system which merely approximates something like ex-
pected utility maximization would be very likely to strip
the same region of resources if it could gure out how to
do so . Nevertheless, as our model currently stands, it
is not suited for analyzing the conditions under which a
given bounded agent would in fact start exhibiting this
sort of behavior.
Extending our model to allow for bounded ratio-
nal agents (in the sense of Gigerenzer and Selten [18])
would have two advantages. First, it could allow us
to make formal claims about the scenarios under which
bounded agents would start pursuing convergent instru-
mental goals in potentially dangerous ways. Second, it
could help us reveal new convergent instrumental goals
that may only apply to bounded rational agents, such
as convergent instrumental incentives to acquire com-
puting power, information about dicult-to-compute
logical truths, incentives to become more rational, and
so on.
Embedding the Agent in the Environment
In our model, we imagine an agent that is inher-
ently separated from its environment. Assuming an
agent/environment separation is standard (see, e.g.,
Legg and Hutter [19]), but ultimately unsatisfactory, for
reasons explored by Orseau and Ring [20]. Our model
gives the agent special status in the laws of physics,which makes it somewhat awkward to analyze conver-
gent instrumental incentives for \self preservation" or
\intelligence enhancement." The existing framework
allows us to model these situations, but only crudely.
For example, we could design a setting where if cer-
tain regions enter certain states then the agent forever
after loses all actions except for actions that have no
eect, representing the \death" of the agent. Or we
could create a setting where normally the agent only
gets actions that have eects every hundred turns, but
if it acquires certain types of resources then it can act
more frequently, to model \computational resources."
However, these solutions are somewhat ad hoc, and we
would prefer an extension of the model that somehow
modeled the agent as part of the environment.
It is not entirely clear how to extend the model in
such a fashion at this time, but the \space-time embed-
ded intelligence" model of Orseau and Ring [20] and
the \re
ective oracle" framework of Fallenstein, Taylor,
and Christiano [21] both oer plausible starting points.
Using the latter framework, designed to analyze com-
plicated environments which contain powerful agents
that reason about the environment that contains them,
might also lend some insight into how to further ex-
tend our model to give a more clear account of how the
agent handles situations where other similarly powerful
agents exist and compete over resources. Our existing
model can handle multi-agent scenarios only insofar as
we assume that the agent has general-purpose methods
for predicting the outcome of its actions in various re-
gions, regardless of whether those regions also happen
to contain other agents.
6.2 Conclusions
Our model is a simple one, but it can be used to validate
Omohundro's intuitions about \basic AI drives" [5], and
Bostrom's \instrumental convergence thesis" [4]. This
suggests that, in the long term, by default, powerful
AI systems are likely to have incentives to self-preserve
and amass resources, even if they are given seemingly
benign goals. If we want to avoid designing systems
that pursue anti-social instrumental incentives, we will
have to design AI systems carefully, especially as they
become more autonomous and capable.
The key question, then, is one of designing princi-
pled methods for robustly removing convergent instru-
mental incentives from an agent's goal system. Can
we design a highly capable autonomous machine that
pursues a simple goal (such as curing cancer) without
giving it any incentives to amass resources, or to resist
modication by its operators? If yes, how? And if not,
what sort of systems might we be able to build instead,
such that we could become condent they would not
have dangerous eects on the surrounding environment
as it pursued its goals?
This is a question worth considering well before it
becomes feasible to create superintelligent machines in
the sense of Bostrom [3], because it is a question about
9
what target the eld of articial intelligence is aiming
towards. Are we aiming to design powerful autonomous
agents that maximize some specic goal function, in
hopes that this has a benecial eect on the world?
Are we aiming to design powerful tools with such lim-
ited autonomy and domain of action that we never need
to worry about the systems pursuing dangerous instru-
mental subgoals? Understanding what sorts of systems
can avert convergent instrumental incentives in princi-
ple seems important before we can begin to answer this
question.
Armstrong [22] and Soares et al. [23] have done
some initial study into the design of goals which ro-
bustly avert certain convergent instrumental incentives.
Others have suggested designing dierent types of ma-
chines, which avoid the problems by pursuing some sort
of \limited optimization." The rst suggestion of this
form, perhaps, came from Simon [24], who suggested
designing agents that \satisce" expected utility rather
than maximizing it, executing any plan that passes a
certain utility threshold. It is not clear that this would
result in a safe system (after all, building a powerful
consequentialist sub-agent is a surere way to satisce),
but the idea of pursuing more \domestic" agent archi-
tectures seems promising. Armstrong, Sandberg, and
Bostrom [16] and Taylor [17] have explored a few alter-
native frameworks for limited optimizers.
Though some preliminary work is underway, it is
not yet at all clear how to design AI systems that re-
liably and knowably avert convergent instrumental in-
centives. Given Omohundro's original claim [5] and the
simple formulations developed in this paper, though,
one thing is clear: powerful AI systems will not avert
convergent instrumental incentives by default. If the
AI community is going to build powerful autonomous
systems that reliably have a benecial impact, then it
seems quite prudent to develop a better understanding
of how convergent instrumental incentives can be either
averted or harnessed, sooner rather than later.
Acknowledgements
The core idea for the formal model in this paper is
due to Benya Fallenstein. We thank Rob Bensinger
for notes and corrections. We also thank Steve Ray-
hawk and Sam Eisenstat for conversations and com-
ments. This work is supported by the Machine Intelli-
gence Research Institute.
References
[1] John Brockman, ed. What to Think About Machines
That Think. Today's Leading Thinkers on the Age of
Machine Intelligence . HarperCollins, 2015.
[2] Stuart J. Russell and Peter Norvig. Articial Intel-
ligence. A Modern Approach . 3rd ed. Upper Saddle
River, NJ: Prentice-Hall, 2010.[3] Nick Bostrom. Superintelligence. Paths, Dangers,
Strategies . New York: Oxford University Press, 2014.
[4] Nick Bostrom. \The Superintelligent Will. Motivation
and Instrumental Rationality in Advanced Articial
Agents". In: Minds and Machines 22.2 (2012): The-
ory and Philosophy of AI . Ed. by Vincent C. M uller.
special issue, pp. 71{85. doi:10.1007/s11023-012-
9281-3 .
[5] Stephen M. Omohundro. \The Basic AI Drives". In:
Articial General Intelligence 2008. Proceedings of the
First AGI Conference . Ed. by Pei Wang, Ben Go-
ertzel, and Stan Franklin. Frontiers in Articial Intel-
ligence and Applications 171. Amsterdam: IOS, 2008,
pp. 483{492.
[6] Eliezer Yudkowsky. Complex Value Systems are Re-
quired to Realize Valuable Futures . The Singularity
Institute, San Francisco, CA, 2011. url:http : / /
intelligence.org/files/ComplexValues.pdf .
[7] Nate Soares. The Value Learning Problem . Tech. rep.
2015{4. Berkeley, CA: Machine Intelligence Research
Institute, 2015. url:https://intelligence.org/
files/ValueLearningProblem.pdf .
[8] Mark R Waser. \Discovering the Foundations of a
Universal System of Ethics as a Road to Safe Arti-
cial Intelligence." In: AAAI Fall Symposium: Biolog-
ically Inspired Cognitive Architectures . Melno Park,
CA: AAAI Press, 2008, pp. 195{200.
[9] Francesco Albert Bosco Cortese. \The Maximally Dis-
tributed Intelligence Explosion". In: AAAI Spring
Symposium Series . AAAI Publications, 2014.
[10] Nate Soares and Benja Fallenstein. Questions of Rea-
soning Under Logical Uncertainty . Tech. rep. 2015{
1. Berkeley, CA: Machine Intelligence Research Insti-
tute, 2015. url:https://intelligence.org/files/
QuestionsLogicalUncertainty.pdf .
[11] John Storrs Hall. Beyond AI. Creating the Conscience
of the Machine . Amherst, NY: Prometheus Books,
2007.
[12] Frank Tipler. \If You Can't Beat 'em, Join 'em". In:
What to Think About Machines That Think. Today's
Leading Thinkers on the Age of Machine Intelligence .
Ed. by John Brockman. HarperCollins, 2015, pp. 17{
18.
[13] Steven Pinker. \Thinking Does Not Imply Subjugat-
ing". In: What to Think About Machines That Think.
Today's Leading Thinkers on the Age of Machine
Intelligence . Ed. by John Brockman. HarperCollins,
2015, pp. 5{8.
[14] Mark Pagel. \They'll Do More Good Than Harm". In:
What to Think About Machines That Think. Today's
Leading Thinkers on the Age of Machine Intelligence .
Ed. by John Brockman. HarperCollins, 2015, pp. 145{
147.
[15] Robert A. Freitas Jr. Some Limits to Global Ecophagy
by Biovorous Nanoreplicators, with Public Policy Rec-
ommendations . Foresight Institute. Apr. 2000. url:
http://www.foresight.org/nano/Ecophagy.html
(visited on 07/28/2013).
10
[16] Stuart Armstrong, Anders Sandberg, and Nick
Bostrom. \Thinking Inside the Box. Controlling and
Using an Oracle AI". In: Minds and Machines 22.4
(2012), pp. 299{324. doi:10 . 1007 / s11023 - 012 -
9282-2 .
[17] Jessica Taylor. \Quantilizers: A Safer Alternative to
Maximizers for Limited Optimization". In: (forthcom-
ing). Submitted to AAAI 2016.
[18] Gerd Gigerenzer and Reinhard Selten, eds. Bounded
Rationality. The Adaptive Toolbox . Dahlem Workshop
Reports. Cambridge, MA: MIT Press, 2001.
[19] Shane Legg and Marcus Hutter. \A Universal Mea-
sure of Intelligence for Articial Agents". In: IJCAI-
05. Proceedings of the Nineteenth International Joint
Conference on Articial Intelligence, Edinburgh, Scot-
land, UK, July 30{August 5, 2005 . Ed. by Leslie
Pack Kaelbling and Alessandro Saotti. Lawrence
Erlbaum, 2005, pp. 1509{1510. url:http : / / www .
ijcai.org/papers/post-0042.pdf .
[20] Laurent Orseau and Mark Ring. \Space-Time Embed-
ded Intelligence". In: Articial General Intelligence.
5th International Conference, AGI 2012, Oxford, UK,
December 8{11, 2012. Proceedings . Lecture Notes in
Articial Intelligence 7716. New York: Springer, 2012,
pp. 209{218. doi:10.1007/978-3-642-35506-6\_22 .
[21] Benja Fallenstein, Jessica Taylor, and Paul F. Chris-
tiano. \Re
ective Oracles: A Foundation for Game
Theory in Articial Intelligence". In: Logic, Ratio-
nality, and Interaction. Fifth International Workshop,
LORI-V, Taipei, Taiwan, October 28{31, 2015. Pro-
ceedings . Ed. by Wiebe van der Hoek, Wesley H. Hol-
liday, and Wen-fang Wang. FoLLI Publications on
Logic, Language and Information. Springer, forthcom-
ing.
[22] Stuart Armstrong. Utility Indierence . 2010-1. Ox-
ford: Future of Humanity Institute, University of Ox-
ford, 2010. url:http : / / www . fhi . ox . ac . uk /
utility-indifference.pdf .
[23] Nate Soares et al. \Corrigibility". Paper presented at
the 1st International Workshop on AI and Ethics, held
within the 29th AAAI Conference on Articial Intel-
ligence (AAAI-2015). Austin, TX, 2015. url:http:
//aaai.org/ocs/index.php/WS/AAAIW15/paper/
view/10124 .
[24] Herbert A. Simon. \Rational Choice and the Struc-
ture of the Environment". In: Psychological Review
63.2 (1956), pp. 129{138.
11