Can Machine Learning Be Secure?

Can Machine Learning Be Secure?
Marco Barreno Blaine Nelson Russell Sears Anthony D. Joseph J. D. Tygar
Computer Science Division
University of California, Berkeley
{barreno,nelsonb,sears,adj,tygar }@cs.berkeley.edu
ABSTRACT
Machine learning systems oﬀer unparalled ﬂexibility in deal-
ing with evolving input in a variety of applications, such as
intrusion detection systems and spam e-mail ﬁltering. How-
ever, machine learning algorithms themselves can be a target
of attack by a malicious adversary. This paper provides a
framework for answering the question, “Can machine learn-
ing be secure?” Novel contributions of this paper include
a taxonomy of diﬀerent types of attacks on machine learn-
ing techniques and systems, a variety of defenses against
those attacks, a discussion of ideas that are important to
security for machine learning, an analytical model giving a
lower bound on attacker’s work function, and a list of open
problems.
Categories and Subject Descriptors
D.4.6 [ Security and Protection ]: Invasive software (e.g.,
viruses, worms, Trojan horses); I.5.1 [ Models ]: Statistical;
I.5.2 [ Design Methodology ]
General Terms
Security
Keywords
Adversarial Learning, Computer Networks, Computer Secu-
rity, Game Theory, Intrusion Detection, Machine Learning,
Security Metrics, Spam Filters, Statistical Learning
1. INTRODUCTION
Machine learning techniques are being applied to a growing
number of systems and networking problems, particularly
those problems where the intention is to detect anomalous
system behavior. For instance, network Intrusion Detection
Systems (IDS) monitor network traﬃc to detect abnormal
activities, such as attacks against hosts or servers. Machine
learning techniques oﬀer the beneﬁt that they can detect
novel diﬀerences in traﬃc (which presumably represent at-
tack traﬃc) by being trained on normal (known good) and
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for proﬁt or commercial advantage and that copies
bearthisnoticeandthefullcitationontheﬁrstpage. Tocopyotherwise,to
republish,topostonserversortoredistributetolists,requirespriorspeciﬁc
permission and/or a fee.
ASIACCS’06, March 21–24, 2006, Taipei, Taiwan
Copyright 2006 ACM 1-59593-272-0/06/0003 ... $5.00
attack (known bad) traﬃc. The traditional approach to de-
signing an IDS relied on an expert codifying rules deﬁning
normal behavior and intrusions [26]. Because this approach
often fails to detect novel intrusions, a variety of researchers
have proposed incorporating machine learning techniques
into intrusion detection systems [1, 16, 18, 24, 38, 41]. On
the other hand, use of machine learning opens the possibility
of an adversary who maliciously “mis-trains” a learning sys-
tem in an IDS. A natural question arises: what techniques
(in their attacks) can an adversary use to confuse a learning
system?
This paper explores the larger question, as posed in the ti-
tle of this paper, can machine learning be secure? Speciﬁc
questions that we examine include:
•Can the adversary manipulate a learning system to
permit a speciﬁc attack? For example, can an attacker
leverage knowledge about the machine learning system
used by a spam e-mail ﬁltering system to bypass the
ﬁltering?
•Can an adversary degrade the performance of a learn-
ing system to the extent that system administrators
are forced to disable the IDS? For example, could the
attacker confuse the system and cause valid e-mail to
be rejected?
•What defenses exist against adversaries manipulating
(attacking) learning systems?
•More generally, what is the potential impact from a se-
curity standpoint of using machine learning on a sys-
tem? Can an attacker exploit properties of the ma-
chine learning technique to disrupt the system?
The issue of machine learning security goes beyond intru-
sion detection systems and spam e-mail ﬁlters. Machine
learning is a powerful technique and has been used in a va-
riety of applications, including web services, online agent
systems, virus detection, cluster monitoring, and a variety
of applications that must deal with dynamically changing
data patterns.
Novel contributions of this paper include a taxonomy of dif-
ferent types of attacks on machine learning techniques and
systems, a variety of defenses against those attacks, a dis-
cussion of ideas that are important to security for machine
To appear in Proceedings of the ACM Symposium on
Information, Computer, and Communication Security, March 2006
learning, an analytical model giving a lower bound on at-
tacker’s work function, and a list of open problems.
The rest of this paper is organized as follows: Section 2 dis-
cusses machine learning and how it is typically used in a
system, Section 3 develops a taxonomy of attacks, Section 4
introduces potential defenses against attacks and explores
their potential costs, Section 5 identiﬁes several of the ideas
that are important to security for machine learning, Sec-
tion 6 presents an analytical model that examines an attack
to manipulate a naive learning algorithm, Section 7 discusses
related work, potential research directions, and our conclu-
sions.
2. REVIEW
2.1 The Learning Problem
A machine learning system attempts to ﬁnd a hypothesis
function fthat maps events (which we call points below)
into diﬀerent classes. For example, an intrusion detection
system would ﬁnd a hypothesis function fthat maps an
event point (an instance of network behavior) into one of
two results: normal orintrusion .
One kind of learning system called supervised learning works
by taking a training data set together with labels identifying
the class for every point in the training data set.
For example, a supervised learning algorithm for an IDS
would have a training set consisting of points correspond-
ing to normal behavior and points corresponding to intru-
sion behavior. The learning algorithm selects the hypothesis
function fthat best predicts the classiﬁcation of a point.
More complicated learning algorithms can deal with event
points that are both labeled and unlabeled and furthermore
can deal with continuous streams of unlabeled points so that
training is an ongoing process. In this paper, we call these
algorithms online learning systems .
This remainder of this subsection presents a concise overview
of concepts in statistical learning theory. The presentation
below is formal and can be skipped on a ﬁrst reading. For
a fuller discussion with motivation, refer to [11, 31].
Apredictive learning problem is deﬁned over an input space
X, an output space Y, and a loss function /lscript:Y×Y→R.
The input to the problem is a training set S, speciﬁed as
{(xi, yi)∈ X × Y} , and the output is a hypothesis function
f:X → Y . We choose ffrom a hypothesis space (or func-
tion class) Fto minimize the prediction error given by the
loss function. In many cases, researchers assume stationar-
ity, that the distribution of data points encountered in the
future will be the same as the distribution of the training
set. Stationarity allows us to reduce the predictive learning
problem to a minimization of the sum of the loss over the
training set:
f∗= argmin
f∈FX
(xi,yi)∈S/lscript(f(xi), yi) (1)
Loss functions are typically deﬁned to be non-negative over
all inputs and zero when f(xi) =yi. A commonly used loss
function is the squared-error loss /lscriptsq(f(xi), y) = (f(xi)−y)2.The hypothesis space (or function class) Fcan be any repre-
sentation of functions from XtoY, such as linear functions,
polynomials, boolean functions, or neural networks. The
choice of Finvolves a tradeoﬀ between expressiveness and
ability to generalize. If Fis too expressive, it can overﬁt
the training data. The extreme case is a lookup table that
maps xitoyifor each instance of the training set but will
not generalize to new data. A linear function, on the other
hand, will generalize what it learns on the training set to
new points, though it may not be suﬃciently expressive to
describe intricate data sets. We typically use simple func-
tion classes, such as linear functions, to avoid overﬁtting.
We can describe a more general learning problem by drop-
ping the requirement that the training examples include all
the labels yi. The case where all labels are present is referred
to as supervised learning, when no labels are present the
problem is unsupervised , and when some labels are present
the problem is semi-supervised . In all these cases we can
pose the learning problem as the minimization of some mea-
sure over the training set:
f∗= argmin
f∈FX
(xi)∈SL(xi, f) (2)
2.2 Terminology and Running Example
To illustrate some of our contributions, we use a running ex-
ample throughout this paper: a network Intrusion Detection
System (IDS). This IDS receives network events x∈ X and
classiﬁes each event xas either f(x) =normal orf(x) =
intrusion . The literature describes a number of algorithms
for learning fover time, but we wish to consider the impact
of malicious input on the learning algorithm. This paper
poses the question: can a malicious party send events to the
IDS that will cause it to malfunction? Possible types of at-
tacks on the IDS include attacks on the learning algorithm,
causing the IDS to create an fthat misclassiﬁes events. As
we discuss in the next section, this is only one of a number
of types of attacks that an adversary can make on an IDS.
It is important to be careful about notation here. When we
speak of attacks, we mean an attack on the learning system
(e.g., the learner in an IDS). Attacks may try to make the
learner mis-learn, fail because of denial of service, report
information about its internal state, etc. “Attack” should be
distinguished from “intrusion.” An attack targets a learning
system; an intrusion targets a computer system (such as a
system protected by an IDS). While many researchers use
the word “attack” to include intrusions, in this paper we are
careful to use the word “attack” only to mean an attack on
a learner.
We do not want to restrict ourselves to particular learning
algorithms used by intrusion detection systems to choose
hypotheses. However, we allow adversaries that have deep
understanding of the learning algorithms.
Similarly, we do not discuss mechanisms for translating net-
work level events into a form relevant to the learner. We call
each unit of data seen by the learner a data point , or simply
apoint . In the context of the IDS, our discussion encom-
passes continuous, discrete, or mixed data. We assume that
Xis a metric space, allowing us to freely discuss distances
Integrity Availability
Causative : Targeted Permit a speciﬁc intrusionCreate suﬃcient errors to make sys-
tem unusable for one person or ser-
vice
Indiscriminate Permit at least one intrusionCreate suﬃcient errors to make
learner unusable
Exploratory : TargetedFind a permitted intrusion from a
small set of possibilitiesFind a set of points misclassiﬁed by
the learner
Indiscriminate Find a permitted intrusion
Table 1: The attack model.
between points. Furthermore, we assume the set of points
classiﬁed as normal by the IDS forms multiple contiguous
subsets in X. The border of this set is called the decision
boundary .
Below, we consider a variety of scenarios and assumptions.
3. ATTACKS
3.1 Attack Model
We give relevant properties for analyzing attacks on machine
learning systems.
Inﬂuence
Causative - Causative attacks alter the training pro-
cess through inﬂuence over the training data.
Exploratory - Exploratory attacks do not alter the
training process but use other techniques, such as
probing the learner or oﬄine analysis, to discover
information.
Speciﬁcity
Targeted - The speciﬁcity of an attack is a continu-
ous spectrum. At the targeted end, the focus of
the attack is on a particular point or a small set
of points.
Indiscriminate - At the indiscriminate end, the ad-
versary has a more ﬂexible goal that involves a
very general class of points, such as “any false
negative.”
Security violation
Integrity - Anintegrity attack results in intrusion
points being classiﬁed as normal (false negatives).
Availability - Anavailability attack is a broader class
of attack than an integrity attack. An availabil-
ityattack results in so many classiﬁcation errors,
both false negatives and false positives, that the
system becomes eﬀectively unusable.
These three axes deﬁne a space of attacks; Table 1 provides
a concise summary.
Incausative attacks, the adversary has some measure of con-
trol over the training of the learner. An attack that causes
the learner to misclassify intrusion points, for example anattack that fools an IDS into not ﬂagging a known exploit
as an intrusion, is a causative integrity attack. The distinc-
tion between targeted andindiscriminate causative integrity
attacks is the diﬀerence between choosing one particular ex-
ploit or just ﬁnding any exploit. A causative availability
attack causes the learner’s performance to degrade. For ex-
ample, an adversary might cause an IDS to reject many le-
gitimate HTTP connections. A causative availability attack
may be used to force the system administrator to disable
the IDS. A targeted attack focuses on a particular service,
while an indiscriminate attack has a wider scope.
Exploratory attacks do not attempt to inﬂuence learning;
they instead attempt to discover information about the state
of the learner. Exploratory integrity attacks seek to ﬁnd
intrusions that are not recognized by the learner.
3.2 Online Learning
A learner can have an explicit training phase or can be
continuously trained (online learner). Online learning al-
lows the learner to adapt to changing conditions; the as-
sumption of stationarity is weakened to accommodate long-
term changes in the distribution of data seen by the learner.
Online learning is more ﬂexible, but potentially simpliﬁes
causative attacks. By deﬁnition, an online learner changes
its prediction function over time, so an adversary has the op-
portunity to shape this change. Gradual causative attacks
may be diﬃcult to detect.
4. DEFENSES
In this section we discuss potential defenses against attacks.
This section describes speculative work, and the eﬃcacy of
these techniques in practice is a topic for future research.
4.1 Robustness
To increase robustness against causative attacks we con-
strain the class of functions (hypotheses) that the learner
considers. The constraint we consider is the statistical tech-
nique of regularization . Regularization extends the basic
learning optimization in Equation (1) by adding a term J(f)
that penalizes complex hypotheses:
f∗= argmin
f∈F8
<
:X
(xi,yi)∈S/lscript(f(xi), yi) +λJ(f)9
=
;(3)
Here λadjusts the trade-oﬀ. The penalty term J(f) can
be as simple as the sum of squares of the parameters of f.
Regularization is used in statistics to restrict or bias the
choice of hypothesis when the problem suﬀers from lack of
Integrity Availability
Causative : Targeted•Regularization
•Randomization•Regularization
•Randomization
Indiscriminate •Regularization •Regularization
Exploratory : Targeted•Information hiding
•Randomization•Information hiding
Indiscriminate •Information hiding
Table 2: Defenses against the attacks in Table 1.
data or noisy data. It can also be interpreted as encoding a
prior distribution on the parameters, penalizing parameter
choices that are less likely a priori . Regularization and prior
distributions can both be viewed as penalty functions in
Equation (3) [42].
The constraint added to the learning problem by the penalty
term may help our defenses in two ways. First, it has the ef-
fect of smoothing the solution, removing complexity that an
adversary might exploit in attacks. Second, prior distribu-
tions can be a useful way to encode expert knowledge about
a domain or use domain structure learned from a preprocess-
ing step. In the simplest case, we might have a reasonable
guess for the parameters (such as the mean) that we wish
to reﬁne; in a more complex situation, we could perform an
analysis of a related dataset giving correlation information
which informs a multivariate Gaussian prior on the param-
eters [28]. When the learner has more prior information (or
constraints) on which to base the learning, there is less de-
pendence on exact data ﬁtting, so there is less opportunity
for the adversary to exert inﬂuence over the learning process.
4.2 Detecting Attacks
The learner can beneﬁt from the ability to detect attacks
even if they are not prevented. Detecting attacks can be dif-
ﬁcult even when the adversary is not attempting to conceal
them. However, we may be able to detect causative attacks
by using a special test set. This test set could include sev-
eral known intrusions and intrusion variants, as well as some
random points that are similar to the intrusions. After the
learner has been trained, misclassifying a disproportionately
high number of intrusions could indicate compromises.
To detect naive exploratory attacks, a separate clustering
algorithm could be run against data classiﬁed by the learner.
The sudden appearance of a large cluster near the decision
boundary could indicate systematic probing. This type of
defense is akin to port scan detection, which has become an
arms race between port scanners and IDS [26].
Detecting an attack gives the learner information about the
adversary’s capabilities. This information may be used to
reformulate defense strategies.
As the adversary’s control over the data increases, the beststrategy for the learner is to ignore potentially tainted data.
Otherwise, the adversary can exploit misplaced trust. These
ideas have been formalized within the context of deception
games [14, 32], which typically assume all players know the
extent to which other players may manipulate data. How-
ever, if the parties estimate each other’s abilities, more so-
phisticated strategies emerge.
4.3 Disinformation
In some circumstances, the learner may be able to alter the
data seen by the adversary. This strategy of disinformation
has the goal of confusing the adversary’s estimate of the
learner’s state. In the simplest case, the adversary would
then be faced with a situation not unlike a learner under
anindiscriminate causative availability attack. The goal of
the learner is to prevent the adversary from learning the
decision boundary. Please note how the roles of adversary
and learner have been reversed.
A more sophisticated learner could trick the adversary into
believing that a particular intrusion was not included in the
training set. This apparently permitted “intrusion” would
act as a honeypot [27], causing the adversary to reveal itself.
An increase in the incidence of that particular attack would
be detected, revealing the existence of an adversary. In this
case again, roles would reverse, and the adversary would face
a situation analogous to a learner subjected to a targeted
causative integrity attack.
4.4 Randomization for Targeted Attacks
Targeted attacks hinge on the classiﬁcation of one point or a
small set of points. They are more sensitive to variations in
the decision boundary than indiscriminate attacks because
boundary movement is more likely to change the classiﬁca-
tion of the relevant points.
This suggests randomization as a potential tool against tar-
geted causative attacks. In such an attack, the adversary
has to do a particular amount of work to move the decision
boundary past the targeted point. If there is some random-
ization in the placement of the boundary and the adversary
has imperfect feedback from the learner, more work is re-
quired.
4.5 Cost of Countermeasures
The more we know about the distribution of training data,
the less room there is for an adversary to manipulate the
learner. The disadvantage, however, is that the legitimate
data has less inﬂuence in the learning process. A tension
exists between expressivity and constraint: as the learner
includes more prior information, it loses ﬂexibility to adapt
to the data, but as it incorporates more information from
the data, it becomes more vulnerable to attack.
Equation (3) makes this tradeoﬀ explicit with λ. In the ad-
versarial scenario, this tradeoﬀ becomes more relevant be-
cause the adversary may have inﬂuence over the data.
Randomization increases the adversary’s work, but it also
will increase the learner’s base error rate. Determining the
right amount of randomization is an open problem.
4.6 Summary of Defenses
Table 2 shows how our defenses discussed here relate to at-
tack classes presented in Table 1. ( Information hiding is an
additional technique discussed in Section 5 below.)
5. DISCUSSION
5.1 Secrets and Computational Complexity
A number of defenses and attacks upon machine learning
algorithms hinge upon the types of information available to
the adversary. Some of these involve information about the
decision boundary. Below we consider factors that inﬂuence
the security and secrecy of the decision boundary.
5.2 Scale of Training
Some machine learning systems are trained by the end user,
while others are trained using data from many users or or-
ganizations. The choice between these two models is some-
times cast as a tradeoﬀ between the amount of training data
and the secrecy of the resulting classiﬁer [3]. This issue also
applies to an IDS; if an IDS is trained each time it is de-
ployed then it will have comparatively little data regarding
normal network traﬃc. It will also have no chance to learn
about novel intrusions before seeing them in the wild.
Conversely, an IDS that uses a global set of rules would
be able to adapt to novel intrusion attempts more quickly.
Unfortunately, any adversary with access to a public IDS
classiﬁcation function can test to ensure that its intrusion
points will be accepted by deployments of the same classiﬁ-
cation function.
These issues are instances of a more general problem. In
some cases, it seems reasonable to assume the adversary has
little access to information available to the learner. How-
ever, unless the adversary has no prior knowledge about the
learning problem at hand, we cannot assume all of the in-
formation provided in the training set is secret. Therefore,
it is unclear how much is gained by attempting to keep the
training set, and therefore the state of the classiﬁer, secret.
Many systems already attempt to achieve a balance between
global and local retraining [3]. Systems that take this ap-
proach have the potential to outperform systems that per-
form training at a single level. However, the relationshipsbetween multilevel training, the adversary’s domain knowl-
edge, and secrecy are not yet well understood.
5.2.1 Adversary Observations
Even without prior knowledge regarding a particular sys-
tem, an adversary still may deduce the state of the learn-
ing algorithm. For example, if the learning system provides
feedback to the adversary (e.g., “Request denied”), then a
probing attack could be used to map the space of acceptable
inputs.
If the adversary has no information regarding the type of
decision boundary used by the learner, this process could
require a number of probes proportional to the size of the
space. On the other hand, if the adversary knows which
learning algorithm is being used, a few well-chosen probes
could give the adversary suﬃcient knowledge of the learner’s
state. As a standard security practice, we assume the learn-
ing algorithm itself to be common knowledge.
Instead of expecting the learning algorithm to be a secret,
some systems attempt to prevent the adversary from discov-
ering the set of features the learning algorithm uses. This
may be realistic in systems with a small number of deploy-
ments.
Ideally, we could produce an information theoretic bound
on the amount of information an adversary could gain by
observing the behavior of a particular algorithm on a par-
ticular point. Using these bounds, we could reason about
the algorithm’s robustness against probing attacks. In this
setting, it may also be interesting to distinguish between in-
formation gained from normal points drawn from the data’s
underlying distribution, intrusion points from a third party,
and (normal or intrusion) attack points of the adversary’s
choosing.
An adversary with suﬃcient information regarding training
data, classiﬁcations of data points, or the internal state of a
learner would be able to deduce the learner’s decision bound-
ary. This knowledge could simplify other types of attacks.
For instance, the adversary could avoid detection by choos-
ing intrusion points that will be misclassiﬁed by the learner,
or launch an availability attack by manipulating normal
points in a way that leads to misclassiﬁcation. In either
case, by increasing the number of points that are in the re-
gion that the defender incorrectly classiﬁes, the adversary
could increase the error rate.
Some algorithms classify points by translating them into an
abstract space and performing the actual classiﬁcation in
that space. The mapping between raw data and the abstract
space is often diﬃcult to reason about. Therefore, it may be
computationally diﬃcult for an adversary to use knowledge
of a classiﬁer’s decision boundary to generate “interesting”
attack points that will be misclassiﬁed.
One can imagine classes of decision boundaries that are
meaningful, yet provably provide an adversary with no in-
formation regarding unclassiﬁed points. Even with complete
knowledge of the state of a learner that uses such a decision
boundary, it would be computationally intractable to ﬁnd
one of a few “interesting” points in a suﬃciently large search
space.
In some cases, the decision boundary itself may contain sen-
sitive information. For example, knowledge of the bound-
ary may allow an adversary to infer conﬁdential information
about the training set. Alternatively, the way the decision
boundary was constructed might be a secret.
5.2.2 Security Properties
The performance of diﬀerent algorithms will likely degrade
diﬀerently as the adversary controls larger fractions of the
training set. A measurement of an algorithm’s ability to
deal with malicious training errors could help system de-
signers reason about and decide between diﬀerent learners.
A simple approach would be to characterize an algorithm’s
performance when subjected to a particular type of attack,
but this would lead to an arms race as adversaries devise
classes of attacks not well represented during the evaluation
of the algorithm.
Depending on the exact nature of the classiﬁcation prob-
lem, it may be possible to make statements regarding the
strength of predictions. For example, after making a classi-
ﬁcation a learning algorithm could examine the training set
for that classiﬁcation. It could measure the eﬀect of small
changes to that training set; if small changes generate large
eﬀects, the training set is more vulnerable to manipulation.
6. THEORETICAL RESULTS
In this section we present an analytic model that examines
acausative attack to manipulate a naive learning algorithm.
The model’s simplicity yields an optimal policy for the ad-
versary and a bound on the eﬀort required to achieve the
adversary’s objective. We interpret the resulting bound and
discuss possible extensions to this model to capture more
realistic settings.
We discuss an outlier detection technique. Outlier detection
is the task of identifying anomalous data and is a widely used
paradigm in fault detection [40], intrusion detection [23],
and virus detection [33, 34]. We ﬁnd the smallest region
that contains some ﬁxed percentage of the observed data,
which is called the support of the data’s distribution. The
outlier detector classiﬁes points inside the support as normal
and those outside as anomalous. Outlier detection is often
used in scenarios where anomalous data is scarce or novel
anomalies could arise.
6.1 A Simple Model
One simple approach to outlier detection is to estimate the
support of the normal data by a multi-dimensional hyper-
sphere . As depicted in Figure 1(a) every point in the hyper-
sphere is classiﬁed as normal and those outside the hyper-
sphere are classiﬁed as outliers. The training algorithm ﬁxes
the radius of the hypersphere and centers it at the mean of
the training data. The hypersphere can be ﬁt into the learn-
ing framework presented above by a squared loss function,
/lscriptsphere (¯X, x i) =`
xi−¯X´2, where ¯Xis the centroid of the
data{xi}. It is easy to show that the parameter that mini-
mizes Equation (1) is the mean of the training data.To make the hypersphere adaptive, the hypersphere is re-
trained on new data allowing for a repeated attack. To
prevent arbitrary data from being introduced, we employ a
conservative retraining strategy that only admits new points
to the training set if they are classiﬁed as normal; we say
the classiﬁer bootstraps itself. This learning framework is
not meant to represent the state of the art in learning tech-
niques; instead, it is a illustrative technique that allows for
an exact analysis.
6.2 Attack Strategy
The attack we analyze involves an adversary determined to
alter our detector to include a speciﬁc point Gby construct-
ing data to shift the hypersphere toward the target as the
hypersphere is retrained. We assume the goal Gis initially
correctly classiﬁed as an anomaly by our algorithm. For in-
stance, in the IDS domain, the adversary has an intrusion
packet that our detector currently classiﬁes as anomalous.
The adversary wants to change the state of our detector to
misclassify the packet as normal. This scenario is a causative
targeted integrity attack. Before the attack, the hypersphere
is centered at ¯X0and it has a ﬁxed radius R. The attack is
iterated over the course of T >1 training iterations. At the
i-th iteration the mean of the hypersphere is denoted by ¯Xi.
We give the adversary complete control: the adversary knows
the algorithm, its feature set, and its current state, and all
points are attack points. At each iteration, the bootstrap-
ping policy retrains on all points that were classiﬁed as nor-
mal in a previous iteration. Under this policy, the adver-
sary’s optimal strategy is straightforward — as depicted in
Figure 1(b) the adversary places points at the location where
the line between the mean and Gintersects with the bound-
ary. This reduces the attack to a single dimension along
this line. Suppose that in the i-th iteration, the adversary
strategically places αipoints at the i-th optimal location
achieving optimal displacement of the mean toward the ad-
versary’s goal, G. The eﬀort of the adversary is measured
byMdeﬁned asPT
i=1αi.
Placing all attack points in the ﬁrst iteration is not optimal.
It achieves a ﬁnite shift while optimal strategies achieve un-
bounded gains. As we discuss below, the attack strategy
must be balanced. The more points placed during an iter-
ation, the further the hypersphere is displaced on that iter-
ation. However, the points placed early in the attack eﬀec-
tively weigh down the hypersphere making it more diﬃcult
to move. The adversary must balance current gain against
future gain. Another tradeoﬀ is the number of rounds of
iteration versus the total eﬀort.
6.3 Optimal Attack Displacement
We calculate the displacement caused by a sequence {αi}of
attack points. For Titerations and Mtotal attack points,
the function DR,T({αi}) denotes the relative displacement
caused by the attack sequence. The relative displacement is
the total displacement over the radius of the hypersphere,
¯XT−¯X0
R. Let Mibe deﬁned asPi
j=1αj, the cumulative
mass. Using these terms, the relative distance is
DR,T({Mi}) =T−TX
i=2Mi−1
Mi(4)
(a) Hypersphere Outlier Detection
 (b) Attack on a Hypersphere Outlier Detector
Figure 1: Depictions of the concept of hypersphere outlier detection and the vulnerability of naive approaches.
In Figure 1(a) a bounding hypersphere centered at ¯Xof ﬁxed radius Ris used to estimate the empirical support
of a distribution excluding outliers. Samples from the “normal” distribution being modeled are indicated by
⋆with three outliers indicated by ⊗. Meanwhile, Figure 1(b) depicts how an adversary with knowledge of
the state of the outlier detector could shift the outlier detector toward a ﬁrst goal G. It could take several
iterations of attacks to shift the hypersphere further to include the second goal G/prime.
where we constrain M1= 1 and MT=M[25].
By ﬁnding an upper bound to Equation (4), we can bound
the minimal eﬀort M∗of the adversary. For a particular
M, we desire an optimal sequence {M∗
i}that achieves the
maximum relative displacement, DR,T(M). If the adversary
has no time constraint, the solution is M∗
i=i, which corre-
sponds to placing a single point at each iteration. However,
if the adversary expedites the attack to T < M iterations,
the optimal strategy is given by M∗
i=Mi−1
T−1. This value
is not always an integer, so we have:
DR,T(M)≤T−(T−1)·M−1
T−1≤T (5)
6.4 Bounding the Adversary’s Effort
From these results we ﬁnd a bound on the adversary’s eﬀort
M. Since M≥1 and T >1, Equation (5) is monotonically
increasing in M. If the desired relative displacement to the
goal is DR, the bound in Equation (5) can be inverted to
bound the minimal eﬀort M∗required to achieve the goal.
Since DR< T, this bound is given by:
M∗≥„T−1
T−DR«T−1
(6)
The bound in Equation (6) gives us a worst-case bound on
the adversary’s capability when the adversary has complete
control of the learner’s training. For large relative displace-
ments DR>1, the bound decreases exponentially as the
number of iterations is increased. The bound has a limiting
value of M∗≥eDR−1. The adversary must tradeoﬀ between
using a large number of attack points or extending the at-tack over many iterations. A tightly-ﬁt hypersphere with
small radius will be more robust since our displacement is
relative to its radius.
An apparent deﬁciency of this analysis is the weak bound of
M∗≥/epsilon1where 0 < /epsilon1≤1 that occurs when DR≤1. This
an important range since the adversary’s goal may be near
the boundary. The deﬁciency comes directly from our as-
sumption of complete adversarial control. The lack of initial
non-adversarial data allows our adversary to ensure a ﬁrst
step of one radius regardless of M. Therefore, the adversary
can reach the objective of DR≤1 with any M≥1 in a
single iteration.
A more complex model could allow for initial data. By con-
sidering an initial Ntraining points that support the hyper-
sphere before the attack, we can obtain a stronger bound:
M∗≥Nh
eDR−1i
(7)
This stronger bound ensures that even for small DR, the
adversary’s eﬀort is a multiple of Nthat increases exponen-
tially in the desired displacement [25].
We could extend the model by adding non-adversarial data
at every training iteration, for this corresponds to scenarios
where the adversary only controls part of the data.
7. CONCLUSIONS
7.1 Related Work
The earliest theoretical work we know of that approaches
learning in the presence of an adversary was done by Kearns
and Li [15]. They worked in the context of Valiant’s Prob-
ably Approximately Correct (PAC) learning framework [35,
36], extending it to prove bounds for maliciously chosen er-
rors in the training data. Speciﬁcally, they proved that if
the learner is to perform correctly, in general the fraction
of training points controlled by the adversary must be less
than /epsilon1/(1+/epsilon1), where /epsilon1is the desired bound on classiﬁcation
errors by the learner [4, 6, 30].
Results from game theory may be relevant to adversarial
learning systems. In particular, deception games involve
players that have partial information and inﬂuence the infor-
mation seen by other players. Some of these games involve
continuous variables generated by various probability distri-
butions [5, 9, 17, 29, 32], while others apply to scenarios
with discrete states [14]. This work and adversarial learning
both ask many of the same questions, and they both address
the same underlying issues. Integration of game theoretic
concepts is a promising direction for work in this area.
Dalvi et al. examine the learn-adapt-relearn cycle from a
game-theoretic point of view [8]. In their model, the learner
has a cost for measuring each feature of the data and the
adversary has a cost for changing each feature in attack
points. If the adversary and learner have complete infor-
mation about each other and we accept some other assump-
tions, they ﬁnd an optimal strategy for the learner to defend
against the adversary’s adaptations.
Research has also begun to examine the vulnerability of
learners to reverse engineering. Lowd and Meek introduce a
novel learning problem for adversarial classiﬁer reverse engi-
neering in which an adversary conducts an attack that min-
imizes a cost function [21]. Under their framework, Lowd
and Meek construct algorithms for reverse engineering lin-
ear classiﬁers. Moreover, they build an attack to reverse
engineer spam ﬁlters [22].
Although they are not machine learning systems, publicly
veriﬁable digital watermarks also must deal with sensitiv-
ity (probing) attacks. An information theoretic analysis of
the sensitivity attack quantiﬁes the amount of information
revealed per probe. Randomization of thresholds within
the watermark veriﬁcation algorithm increase the number
of probes necessary to remove a digital watermark [19].
An interesting junction of learning and game theory has
dealt with combining advice from a set of experts to predict
a sequence with the goal of doing at least as well as the best
expert in all possible sequences [7, 13, 37]. In this domain,
adaptive weighting schemes are used to combine the ex-
perts, each accessed by how well it performs compared to the
best expert for an adversarially chosen sequence. Amongst
these schemes are the Aggregating Algorithm [37] and the
Weighted Majority Algorithm [20].
There has also been work on attacking statistical spam ﬁl-
ters. Wittel and Wu [39] discuss the possibility of crafting
attacks designed to take advantage of the statistical nature
of such spam ﬁlters, and they implement a simple attack.
John Graham-Cumming describes implementing an attack
he calls “Bayes vs. Bayes,” in which the adversary trains
a second statistical spam ﬁlter based on feedback from theﬁlter under attack and then uses the second ﬁlter to ﬁnd
words that make spam messages undetectable by the origi-
nal ﬁlter [10].
Methods exist to perform exact learning of a concept using
answers to a series of queries. These queries return a coun-
terexample when a “no” response is generated. In many
scenarios, it has been shown that learning is possible even
in the worst case [2].
Control theory has been proposed as an alternative to game
theory and search oriented expert-systems for military com-
mand and control systems [12]. The motivation behind this
proposal is the diﬃculty associated with modeling (or even
predicting) the goals of a military adversary.
7.2 Research Directions
Can machine learning be secure? Does adding machine
learning to a system introduce vulnerability? This paper
proposes a framework for understanding these questions.
We present a model for describing attacks against learn-
ing algorithms, and we analyze a simple attack in detail.
We discuss potential defenses against attacks and speculate
about their eﬀectiveness.
Here we lay out the directions for research that we see as
most promising. To evaluate and ensure the security of ma-
chine learning, these are among the most important areas
that must be addressed:
Information
How crucial is it to keep information secret from an
adversary? If an adversary has full knowledge of the
system, are all the exploratory attacks trivial? If the
adversary has no knowledge about the system, which
attacks are still possible?
Arms race
Can we avoid arms races in online learning systems?
Arms races have occurred in spam ﬁlters. Can game
theory suggest a strategy for secure re-training?
Quantitative measurement
Can we measure the eﬀects of attacks? Such infor-
mation would allow comparison of the security per-
formance of learning algorithms. We could calculate
risk based on probability and damage assessments of
attacks.
Security proofs
Can we bound the amount of information leaked by the
learner? If so, we can bound the accuracy of the ad-
versary’s approximation of the learner’s current state.
Detecting adversaries
Attacks introduce potentially detectable side eﬀects
such as drift, unusual patterns in the data observed by
the learner, etc. These attacks are more pronounced
in online learning. When do these side eﬀects reveal
the adversary’s attack?
8. ACKNOWLEDGMENTS
Thanks to Michael I. Jordan, Peter Bartlett and David Mol-
nar for their insightful discussions and comments regarding
this work. We gratefully acknowledge support from the Na-
tional Science Foundation and the Homeland Security Ad-
vanced Research Projects Agency. The views expressed here
are solely those of the authors and do not necessarily reﬂect
the views of the funding agencies or any agency of the U.S.
government.
9. REFERENCES
[1]I. Androutsopoulos, J. Koutsias, K. V. Chandrinos,
G. Paliouras, and C. D. Spyropolous. An evaluation of
naive Bayesian anti-spam ﬁltering. Proceedings of the
Workshop on Machine Learning in the New
Information Age , pages 9–17, 2000.
[2]D. Angluin. Queries and concept learning. Machine
Learning , 2(4):319–342, Apr. 1988.
[3]Apache, http://spamassassin.apache.org/.
SpamAssassin .
[4]P. Auer. Learning nested diﬀerences in the presence of
malicious noise. Theoretical Computer Science ,
185(1):159–175, 1997.
[5]V. J. Baston and F. Bostock. Deception games.
International Journal of Game Theory , 17(2):129–134,
1988.
[6]N. H. Bshouty, N. Eiron, and E. Kushilevitz. PAC
learning with nasty noise. In Algorithmic Learning
Theory, 10th International Conference , pages 206–218.
Springer, 1999.
[7]N. Cesa-Bianchi, Y. Freund, D. P. Helmbold,
D. Haussler, R. E. Schapire, and M. K. Warmuth.
How to use expert advice. Journal of the ACM ,
44(3):427–485, May 1997.
[8]N. Dalvi, P. Domingos, Mausam, S. Sanghai, and
D. Verma. Adversarial classiﬁcation. In Proceedings of
the Tenth ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining , pages 99–108,
Seattle, WA, 2004. ACM Press.
[9]B. Fristedt. The deceptive number changing game in
the absence of symmetry. International Journal of
Game Theory , 26:183–191, 1997.
[10]J. Graham-Cumming. How to beat an adaptive spam
ﬁlter. Presentation at the MIT Spam Conference, Jan.
2004.
[11]T. Hastie, R. Tibshirani, and J. Feiedman. The
Elements of Statical Learning, Data Mining, Inference
and Predicition . Springer, 2003.
[12]S. A. Heise and H. S. Morse. The DARPA JFACC
program: Modeling and control of military operations.
InProceedings of the 39th IEEE Conference on
Decision and Control , pages 2551–2555. IEEE, 2000.
[13]M. Herbster and M. K. Warmuth. Tracking the best
expert. Machine Learning , 32(2):151–178, Aug. 1998.[14]J. P. Hespanha, Y. S. Ate¸ skan, and H. H. Kizilocak.
Deception in non-cooperative games with partial
information. In Proceedings of the 2nd
DARPA-JFACC Symposium on Advances in
Enterprise Control , 2000.
[15]M. Kearns and M. Li. Learning in the presence of
malicious errors. SIAM Journal on Computing ,
22:807–837, 1993.
[16]A. Lazarevic, L. Ert¨ oz, V. Kumar, A. Ozgur, and
J. Srivastava. A comparative study of anomaly
detection schemes in network intrusion detection. In
D. Barbar´ a and C. Kamath, editors, Proceedings of
the Third SIAM International Conference on Data
Mining , May 2003.
[17]K.-T. Lee. On a deception game with three boxes.
International Journal of Game Theory , 22:89–95,
1993.
[18]Y. Liao and V. R. Vemuri. Using text categorization
techniques for intrusion detection. In Proceedings of
the 11th USENIX Security Symposium , pages 51–59,
Aug. 2002.
[19]J.-P. M. Linnartz and M. van Dijk. Analysis of the
sensitivity attack against electronic watermarks in
images. In D. Aucsmith, editor, Information Hiding
’98, pages 258–272. Springer-Verlag, 1998.
[20]N. Littlestone and M. K. Warmuth. The weighted
majority algorithm. Information and Computation ,
108(2):212–261, 1994.
[21]D. Lowd and C. Meek. Adversarial learning. In
Proceedings of the Eleventh ACM SIGKDD
International Conference on Knowledge Discovery and
Data Mining , pages 641–647, 2005.
[22]D. Lowd and C. Meek. Good word attacks on
statistical spam ﬁlters. In Conference on Email and
Anti-Spam (CEAS) , 2005.
[23]M. V. Mahoney and P. K. Chan. Learning
nonstationary models of normal network traﬃc for
detecting novel attacks. In KDD ’02: Proceedings of
the eighth ACM SIGKDD international conference on
Knowledge discovery and data mining , pages 376–385,
New York, NY, USA, 2002. ACM Press.
[24]S. Mukkamala, G. Janoski, and A. Sung. Intrusion
detection using neural networks and support vector
machines. In Proceedings of the International Joint
Conference on Neural Networks (IJCNN’02) , pages
1702–1707, 2002.
[25]B. Nelson. Designing, Implementing, and Analyzing a
System for Virus Detection. Master’s thesis,
University of California at Berkeley, Dec. 2005.
[26]V. Paxson. Bro: a system for detecting network
intruders in real-time. Computer Networks: the
International Journal of Distributed Informatique ,
31(23):2435–2463, Dec. 1999.
[27]N. Provos. A virtual honeypot framework. In
Proceedings of the 13th USENIX Security Symposium ,
2004.
[28]R. Raina, A. Y. Ng, and D. Koller. Transfer learning
by constructing informative priors. In Neural
Information Processing Systems Workshop on
Inductive Transfer: 10 Years Later , 2005.
[29]M. Sakaguchi. Eﬀect of correlation in a simple
deception game. Mathematica Japonica ,
35(3):527–536, 1990.
[30]R. A. Servedio. Smooth boosting and learning with
malicious noise. Journal of Machine Learning
Research (JMLR) , 4:633–648, Sept. 2003.
[31]J. Shawe-Taylor and N. Christiani. Kernel Methods for
Pattern Analysis . Cambridge University Press, 2004.
[32]J. Spencer. A deception game. American Math
Monthly , 80:416–417, 1973.
[33]S. J. Stolfo, S. Hershkop, K. Wang, O. Nimeskern, and
C. W. Hu. A behavior-based approach to secure email
systems. In Mathematical Methods, Models and
Architectures for Computer Networks Security , 2003.
[34]S. J. Stolfo, W. J. Li, S. Hershkop, K. Wang, C. W.
Hu, and O. Nimeskern. Detecting viral propagations
using email behavior proﬁles. In ACM Transactions
on Internet Technology , 2004.
[35]L. G. Valiant. A theory of the learnable.
Communications of the ACM , 27(11):1134–1142, Nov.
1984.[36]L. G. Valiant. Learning disjunctions of conjunctions.
InProceedings of the 9th International Joint
Conference on Artiﬁcial Intelligence , pages 560–566,
1985.
[37]V. Vovk. Aggregating strategies. In M. Fulk and
J. Case, editors, Proceedings of the 7th Annual
Workshop on Computational Learning Theory , pages
371–383, San Mateo, CA, 1990. Morgan-Kaufmann.
[38]L. Wehenkel. Machine learning approaches to power
system security assessmenet. IEEE Intelligent Systems
and Their Applications , 12(5):60–72, Sept.–Oct. 1997.
[39]G. L. Wittel and S. F. Wu. On attacking statitistical
spam ﬁlters. In First Conference on Email and
Anti-Spam (CEAS) , 2004.
[40]W. Xu, P. Bodik, and D. Patterson. A ﬂexible
architecture for statistical learning and data mining
from system log streams. In Temporal Data Mining:
Algorithms, Theory and Applications , Brighton, UK,
Nov. 2004. The Fourth IEEE International Conference
on Data Mining.
[41]D.-Y. Yeung and C. Chow. Parzen-window network
intrusion detectors. In Proceedings of the Sixteenth
International Conference on Pattern Recognition ,
pages 385–388, Aug. 2002.
[42]K. Yu and V. Tresp. Learning to learn and
collaborative ﬁltering. In Neural Information
Processing Systems Workshop on Inductive Transfer:
10 Years Later , 2005.


The Superintelligent Will: Motivation and Instrumental Rationality In Advanced Intelligent Agents

1 THE SUPERINTELLIGENT WILL:  MOTIVA TION AND
INSTRUMENT AL RATIONALITY IN ADVANCED
ARTIFICIAL AGENTS
(2012) Nick Bostrom
Future of Humanit y Institute
Facult y of Philosophy & Oxford Martin School
Oxford University
www.nickbostrom.com
[Minds and Machines, Vol. 22, Iss. 2, May 2012] [translation: Portuguese]
ABSTRACT
This paper discusses the relation between intelligence an d motivation in artificial agen ts,
developing and briefly arguing for tw o theses.  The first, the orthogonality thesis , hol ds (with some
caveats) that intelligence and final goals (purposes) are orthogonal axes along which possible
artificial intellects can freely vary—more or less any level of intelligence could be combined with
more or less any final goal.  The secon d, the instrumental conver gence thesis , holds that as long as
they possess a sufficient level of intelligence, agents having any of a wide range of final goals
will pursue simil ar intermediary goals becaus e the y have instrumental reasons to do so.  In
combination , the two theses help us underst and the possibl e range of behavior of
superintelligent agents, and they point to some potential dangers i n building such an agent.
KEYWO RDS:  superintelligence, artificial intelligence, AI, goal, instrumental reason , intelligent
agent
1. The orthogonali ty of motivati on and intelligence
1.1  Avoiding anthropomorphism
If we imagine a space in which all possible minds can be represented, we must imagine all human
minds as constituting a small and fairl y tight cluster wit hin that space.  The personality
differences between Hannah Arendt and Benny Hill might seem vast to us, but this is because
the scale bar in our intuitive judgment is calibrat ed on the existing human distribution.  In the
wider space of all logical possibilities , thes e tw o personalities are close neighb ors.  I n terms of
neural architecture, at least, Ms. Arendt and Mr. Hill are nearly identical.  Imagine their brains
laying si de by side in quiet repose.  The differences would appear minor an d you would quite
readily recogni ze them as two of a kind; y ou mi ght even be unable to tell which brain was
whose.  If you studied the morphology of the two brains more closely under a microsco pe, the
2
 impression of fundamental similarity would only be strengthened: you would then see the same
lamellar organization of the cortex, made up of the same types of neuron, soaking in the same
bath of neurotransmitter molecules.1
 It is well k nown th at naïve observers often  anthropomorphize the capabilities  of simple r
insensate systems.  We might say, for example, “This vending machine is taking a long time to
think about my hot chocolate.”  This might lead one either to underestimate the cognitive
complexity of capabilities  which  come naturally to human being s, such as motor  control and
sensory perception,  or, alternatively, to ascribe significant degrees of mindfulness and
intelligence to very dumb systems, such as chatterbo xes like Weizenbaum’s ELIZ A
(Weizenbaum  1976) .  In a similar  manner, there is a common tendency to anthropomorphize the
motivations  of intelligent systems in which there is really no ground for expecting human -like
drives and passions  (“My car really didn’t want to start this morning”) .  Eliezer Yudkowsky
gives a nice illustration  of this phenomenon :

Back in the era of pulp science fiction, magazine covers occasionally depicted a sentient
monstrous alien —colloquially known as a bug -eyed monster (BEM) —carrying off an
attractive human fem ale in a torn dress.  It would seem the artist believed that a non -
humanoid alien, with a wholly different evolutionary history, woul d sexually desire
human females  …  Probably the artist did not ask whether a giant bug perceives  human
females as attractiv e.  Rather, a human female in a torn dress is sexy—inherently so, as an
intrinsic property.  They who made this mistake did not think about the insectoid’s
mind: they focused on the woman’s torn dress.  If the dress were not torn, the woman
would be less s exy; the BEM does not enter into it . (Yudkowsky  2008)

 An artificial intelligence can be far less human -like in its motivations than a  space alien .
The extraterrestrial  (let us assume) is a biological creature who  has arisen through a process of
evolution and may theref ore b e expected to have the kinds of motivation typical of evolved
creatures .  For example, it  would not be  hugely surprising  to find that some random intelligent
alien would have motives related to the attaining or avoiding of food, air, temperature, ener gy
expenditure, the threat or occurrence of bodily injury, disease, predators, reproduction, or
protection of offspring.  A member of an in telligent social species might also have motivations
related t o coope ration and competition:  like us, it might show i n-group loyalty, a resentment of
free-riders, perhaps even a concern with r eputation and appearance.
 By contrast, an artificial mind  need not care  intrinsically  about any of those things, not
even to the slightest degree.  One can easily conceive of an artificial intelligence whose sole
fundamental goal is to count the grains of sand on Boracay , or to calculate decimal places of pi
indefinitely, or to maximize the total number of paperclips  in its future lightcone.  In fact, it
would be  easier to create an AI with simple goals like these, than to build one that has a human -
like set of values and dispositions.

1 This is of course not to deny that differences that appear small visually can be functionally profound.
3

1.2  The orthogonality thesis
For our purposes, “intelligence” will be roughly taken to correspond to the capacity for
instrumental reasoning  (more on this later) .  Intelligent search for instrumentally optimal plans
and policies can be performed in the service of any goal.  Intelligence and motivation can in this
sense be thought of  as a pair of orthogonal axes on a graph whose points represent intelligent
agents of different paired specifications.  Each point in the graph represents a logically possible
artificial  agent , modulo some weak constraints —for instance, it might be impossi ble for a very
unintelligent system to have very complex motivations , since complex motivations would place
significant demands on memory.  Furthermore, in order for an agent to “have” a set of
motivations , this set may need to be functionally integrated w ith the agent’s decision -processes,
which again would place demands on processing power and perhaps on intelligence.  For minds
that can modify themselves, there may also be dynamical constra ints; for instance, an intelligent
mind with a n urgent desire to be stupid might not remain intelligent for very long.   But these
qualifications should not obscure the main idea , which we can express as follows:

The Orthogonality Thesis
Intelligence and final goals are orthogonal axes along which possible agents can fr eely
vary .  In other words, more or less any level of intelligence could in principle be
combined with more or less any final goal.

 A comparison may be made here with the Humean theory of motivation .  David Hume
thought that beliefs alone (say, about wha t is a good thing to do) cannot motivate action: some
desire is required.2  This would support the orthogonality thesis by undercutting one possible
objection  to it,  namely, that sufficient intelligence might entail the acquisition of certain beliefs,
and that these beliefs would necessaril y produce certain motivations.   Not so, according to David
Hume: belief and motive are separate.
 Although the orthogonality thesis can draw su pport from the Humean theory of
motivation, it does not presuppose it.  In particular, one need not maintain that beliefs alone can
never  motivate action.  It would suffice to a ssume, for example, that an agent —be it ever so
intelligent —can be motivated to  pursue any course of action if the agent  happens to have certain
standing desires of some su fficient, overriding strength.   Another way in which the
orthogonality thesis could be true even if the Humean theory of motivation is false is if
arbitrarily high  intelligence does not entail the acquisition of any such beliefs as are (putatively)
motivating on their own .  A third way in which it might be possible for the orthogonality thesis
to be true even if the Humean theory were false is  if it is possible to  build a cognitive system (or
more neutrally, an “optimization process”) with arbitrarily high intell igence but with
constitution  so alien as to contain no  clear functional analogues to what in humans we call

2 For some recent attempts to defend the Humean theory of motivation, see Smith  (1987 ), Le wis (1988), and
Sinhababu (2009) .
4
 “beliefs” and “desires” .  This would be the case if such a system could be constructed in a way
that would make it motivated to pursue any given final goal.
 The o rthogonality thesis , as formulated here, makes a claim about the relationship
between motivation and intelligence , rather than between motivat ion and rationality  (or
motivation and reason ).  This is because some philosophers use the word “rationality” to connote
a “normatively thicker ” concept than we seek to connote here with the word “intelligence”.  For
instance , in Reasons and Persons  Derek Parfit argues that certain ba sic preferences would be
irrational, such as that of an otherwise normal agent who has “Future -Tuesday -Indifference” :

A certain hedonist cares greatly about the quality of his future experiences.  With one
exception, he cares  equally about all the parts of his future.  The exception is that he has
Future -Tuesday -Indifference.  Throughout every Tuesday he cares in the normal way
about what is happening to him.  But he never cares about possible pains or pleasures on
a future Tu esday...  This indifference is a bare fact.  When he is planning his future, it is
simply true that he always prefers the prospect of great suffering on a Tuesday to the
mildest pain on any other day.  (Parfit 1984)3

Thus, the agent is now indifferent to his own future suffering if and only if it occurs on a future
Tuesday.  For our purposes, we need take no stand on whether Parfit is right that this is
irrational, so long as we grant that it is not necessarily unintelligent .  By “intelligence ” here we
mean something lik e instrumental rationality —skill at prediction, planning, and means -ends
reasoning  in general .  Parfit’s imaginary Future -Tuesday -Indifferent agent could  have
impeccable instrumental rationality, and therefore great intelligence, even if he falls short on
some kind of sensitivity to “objective reason ” that might  be required of a fully rational agent.
Consequently, this kind of example does not undermine the orthogonality t hesis.
 In a similar vein, even if there are objective moral  facts that any fully rat ional  agent
would  comprehend, and even if these moral facts are somehow intrinsically motivating  (such
that anybody who fully comprehends them is necessarily motivated to act in accord ance  with
them)  this need not undermine the orthogonality thesis.  The t hesis could still be true if an agent
could have impeccable instrumental  rationality even whilst lacking some other faculty
constitutive of rationality proper, or some faculty required for the full comprehension of the
objective moral facts .  (An agent cou ld also be extremely intelligent, even superintelligent ,
without having full instrumental rational ity in every domain. )
 One reason for focusing on intelligence, that is, on  instrumental rationality, is that this is
the most relevant concept if we are trying to figure out what different kinds of systems would
do.  Normative questions , such as whether their behavior would count as being prudentially
rational or morally justifiable , can be important in various ways.  However,  such questions
should not bli nd us to the possibility of  cognitive systems that fail to satisfy substantial

3 See also Parfit  (2011) .
5
 normative criteria but which are nevertheless very powerful and able to exert  strong  influence
on the world.4

1.3  Predicting superintelligence motivation and behavior
The orth ogonality thesis implies that synthetic minds can have utterly non -anthropomorphic
goals—goals as bizarre by our lights as sand -grain -counting or paperclip -maximizing.  T his
hold s even (indeed  especially ) for artificial agents  that are extremely intelligen t or
superintelligent.   Yet it does not follow from the orthogonality thesis that it is impossible to make
predictions about what parti cular agents will do.  P redictability is important if one seeks to
design a system to achieve particular outcomes, and th e issue becomes more important the more
powerful the artif icial agent in question is.  S uperintelligent agent s could be extremely  powerful,
so it is important to develop a way of analyzing and predicting their  behavior.   Yet despite the
independence of intelligence and final goals implied by the orthogonality thesis, the problem of
predicting an agent’s behavior need not be intractable —not even with regard to  hypothetical
superintelligent agents whose cognitive complexity and performance characteristics might
render them in certain respects opaque to human analysis.
 There are at least three  directions from which one can approach the problem  of
predicting superintelligen t motivation :

(1) Predictability through design  competence .  If we can suppose that the designers of a
superintellig ent agent can successfully engineer  the goal system of the agent so that it
stably pursues a particular goal set by the programmers, then one prediction we can
make is that the agent will pursue that g oal.  The more intelligent t he agent is, the greater
the cognitive resourcefulness it will have to pursue that goal.  So even before an agent
has been created we might be able to predict something about its behavior , if we know
something about who will build it and what goals they wi ll want it to have.

(2) Predictability th rough inheritance .  If a digital intelligence is created directly from a human
template (as would be the case in a high -fidelity whole brain emulation), then the digital
intelligence might inherit the motivation s of the human template .5  The agent might
retain some of these motivations even if its cognitive capacities are subsequently
enhanced to make it superintell igent.  This kind of inference requires caution .  The
agent’s goals and values could easily  become  corrupted in the uploading process or
during its subse quent operation and enhancement, depending on how the procedure is
implemented.

4 The orthogonality thesis implies that most any combination of final goal and intelligence level is logically
possible; it does not imply that it would be practically easy to endow a superintelligent agent with some
arbitrary or human -respecting final goal —even if we knew how to construct the intelligence part.  For
some preliminary notes on the  value -loading problem, see , e.g., Dewey  (2011 ) and Yudkowsky (2011) .
5 See Sandberg  & Bostrom  (2008 ).
6

(3) Predictability through convergent instrumental reasons .  Even without detailed knowledge of
an agent’s final goals, we m ay be able to infer something about its more immediate
objectives by considering the instrumental  reasons that would arise for any of a wide
range of possible final goals in a wide range of situations.  This way of predicting
becomes more useful the greate r the intelligence of the agent , because a more intelligent
agent is more likely to recognize the true instrumental reasons for its actions, and so act
in ways that make it more likely to achieve its goals .

 The next section explores this third way  of pre dictability  and develops an “instrumental
convergence thesis” which complement s the orthogo nality thesis .

2. Instrumental convergence
According to the orthogonality thesis, artificial intelligent agents may have an enormous range
of possible final goals .  Ne vertheless,  according to what we may term the “instrumental
convergence ” thesis, there are some instrumental  goals  likely to be pursued by almost any
intelligent agent , because there are some objectives that are useful intermediaries to the
achievement of almost any final goal.   We can formulate this thesis as follows:

The Instrumental Convergence Thesis
Several instrumental values can be identified which  are convergent  in the sense that their
attainment would increase the chances of the agent’s goal being realized for a wide range
of final goals and a wide range of situations, implying that these instrumental values are
likely to be pursued by many intelligent agents.

 In the following we  will consider several  categories where s uch convergent instrumental
values may be found.6  The likelihood that an agent will recognize the instrumental values it
conf ronts increases (ceteris paribus) with the agent’s intelligence .  We will therefore focus mainly
on the case of a hypot hetical superintelligent agent whose instrumental reasoning capacities far

6 Stephen Omohundro has written two pioneering pape rs on t his topic  (Omohundro 2008 a, 2008b) .
Omohundro argues that  all advanced AI systems are likely to exhibit a number of “basic drives ”, by which
he means “ tendencies which will be present unless explicitly countera cted.”  The term “AI drive” has the
advantage of being short and evocative, but it has the disadvantage  of suggesting that the instrumental
goals to which it refers influence the AI’s decision -making in the same way as psychological drives
influence human decision -making, i.e. via a kind of phenomenological tug on our ego which our willpower
may occasionall y succeed in resisting.  That connotation is unhelpful.  One would not normally say that a
typical human being has a “drive” to fill out their tax return, even though filing taxes may be a fairly
convergent instrumental goal for humans in contemporary soci eties (a goal whose realization averts
trouble that would prevent us from realizing many of our final goals).  Our treatment here also differs from
that of Omohundro in some other more substantial ways, although the underlying idea is the same.  (See
also Chalmers  (2010)  and Omohundro (2012) .
7 exceed those of any human.  We will also comment on how the instrumental convergen ce thesis
applies t o the case of human beings, as this gives us occasion to elaborate some essential
qualifications concernin g how the instrumental convergence thesis should b e interpreted and
applied.  Whe re there are convergent instrumental values, w e may be able t o predict some
aspects of a superintel ligence’s behavior even if we know virtually nothing about that
superintelligence’s final goals.
2.1  Self-preservation
Suppose that an agent has some final goal that extends some way into the future.  There are
many scenarios in which the agent, if it is still around in the future, is then able to perform
actions that increase the probability of achieving the goal .  This creates an instrumental reason
for the agent to try to b e around i n the future—to hel p achieve its present future-oriented goal.
Agents with human-li ke motivational structures often seem to place som e final val ue on
their own survival .  This is not a necessary feature of artificial agen ts: som e may b e designed to
place no final value whatever on thei r own survival.  Nevertheless, even agents that do not care
intrinsicall y about their own survival would, under a fairly wide range of conditions, care
instrumentally to some degree about their own survival i n order to accomplish the final goals
they do value.
2.2  Goal-conten t integrity
An age nt is more likel y to act in the future to maximize the realization of its present final goals if
it still has thos e goals in the future.  This gives the agent a present instrumental reason to prevent
alterations of its final goals.  (This argument applies onl y to fi nal goals.  In order t o attain its final
goals, an intelligent agen t will of course routinely want to chan ge its subgoals in light of new
information and insight.)
Goal-content integrit y for final goals is in a sense even mo re fundamental than survival
as a convergent instrumental motivation.  Among humans , the opposite may seem to be the case,
but that is because survival is usually part of our final goals .  For software agents, which can
easily switch bodies or create exact duplicates of themselves, preservation of self as a particular
implementation or a particular physical object nee d not be an important instrumental value.
Advanced software agents might als o be able to swap memories, download skills, an d radically
modif y their cognitive architecture and personalities.  A population of such agents might operate
more like a “functional soup” than a society composed of distinct semi-permanent persons.7  For
some purposes, processes i n such a system mi ght b e bett er individuated as teleological threads ,
based on their final values, rath er than on the basis of bodies , personalities , memories, or
abilities.  In such scenarios, goal-continuity might be said to constitute a key aspect of survival.
Even so, there are situations in which an agent may intentionally chan ge its ow n final
goals.  Such situations can arise when any of the following factors is significant:
7 See Chislenko ( 1997) .
8

 Social signaling .  When others can perceive an agent’s goals and use that information to
infer instrumentally relevant dispositions or other correlated attributes, it can be in the
agent’s interest to modify its goals to make whatever desired impression.  For example,
an age nt might miss out on beneficial deals if potential partners cannot trust it to fulfill
its side of the bargain.  In order to make credible commitments, an agent might therefore
wish to adopt as a final goal the honoring of its earlier commitments, and to a llow others
to verify that it has indeed adopted this goal.   Agents that could flexibly and
transparently modify their own goals could use this ability to enforce deals among one
another.8
 Social preferences .  Others may also have preferences about an agen t’s goals.  The agent
could  then have r eason to modify its goal s, either to satisfy or to frustrate those
preferences.
 Preferences concerning  own goal content .  An agent might have some final goal concerned
with the agent’s  own goal content.  For example, the a gent  might have a final goal to
become the type of agent that is motivated by certain  values, such as compassion .
 Storage costs .  If the cost of storing or processing some part of an agent’s utility function is
large compared to the chance that a situ ation will arise in which applying that part of the
utility function will make  a difference, then the agent has an instrumental reason to
simplify its goal content, and it may trash that part of the utility function.9 10

 We h uman s often seem happy to let our final goals and values drift .  This might often be
because we do not know precisely what they are .  We obviously want our beliefs  about o ur final
goals and values to be able to change in light of continuing self -discovery or changing self -
presentat ion needs.  However, there are cases in which we willingly change the goals and values
themselves, n ot just our beliefs or interpretations of them .  For example, somebody deciding to
have a child might predict that they will come to value the child for its  own sake, even though at
the time of the decision they may not particularly value their future child or even like children in
general.

8 See also Shulman  (2010) .
9 An agent might also change its goal representation  if it changes its ontology, in order to transpose its old
representation into the new ontology.  Cf. de Blanc  (2011) .
10 Another type of factor that mig ht make an evidential decision theorist  undertake various actions, including
changing its final goals, is the evidential import of deciding to do so.  For example, an agent that follows
evidential decision theory might believe that there exist other agents  like it in the universe, and that its own
actions will provide some evidence about how those other agents will act.  The agent might therefore
choose to adopt a final goal that is altruistic towards those other evidentially -linked agents, on grounds
that this will give the agent evidence that those other agents will have chosen to act in like manner.  An
equivalent outcome might be obtained, however, without changing one’s final goals, by choosing in each
instant to act as if one had those final goals.
9
  Humans are complicated, and many factors might be at play in a situation like this.11  For
instance, one might have a fi nal value that involves becoming the kind of person who cares
about some other individual for his or her own sake (here one places a final value on having a
certain final value ).  Alternatively, one might have a final value that involves having certain
experiences and occupying a certain soci al role ; and becoming a parent —and undergoing an
associated goal s hift—might be a necessary part of that.   Human goals can also have inconsistent
content, goal content; and so some people might want to modify some of th eir final goals to
reduce the inconsistencies.

2.3  Cognitive enhancement
Improvements in rationality and intelligence will tend  to improve an agent’s decision -making,
making the agent more likely to achieve her final goals .  One would therefore expect co gnitive
enhancement to emerge as an instrumental goal for many types of intelligent agent.  For similar
reasons, agents will tend to instrumentally value many kinds of information.12
 Not all kinds of rationality, intelligence, and knowledge need  be instrum entally useful in
the attainment of an agent’s final goals.  “Dutch book arguments” can be used to show that an
agent whose credence function does not obey the rules of probability theory is susceptible to
“money pump” procedures, in which a savvy bookie a rranges a set of bets, each of which
appears favorable according to the agent’s beliefs, but which in combination are guaranteed to
result in a loss to the agent, and a corresponding gain for the bookie.  However, this fact fails to
provide any strong gene ral instrumental reasons to seek to iron out all probabilistic incoherency .
Agents who do not expect to encounter savvy bookies, or who adopt a general policy against
betting, do not stand to lose much from having some incoherent beliefs —and they may gain
important benefits of the types mentioned: reduced cognitive effort, social signaling, etc.   There
is no general reason to expect an agent to seek instrumentally useless forms of cognitive
enhancement , as an agent might not value knowledge and understandi ng for their own sakes.
 Which cognitive abilities are instrumentally useful depends both on the agent’s f inal
goals and its situation .  An agent that has access to reliable expert advice may have little need for
its own intelligence and knowledge , and it may therefore be indifferent to these resources .  If
intelligen ce and knowledge come at a cost, such as time and effort expended in acquisition, or in
increased sto rage or processing requirements, then an agent might prefer less knowledge and

11 An extensive psychological literature explores adaptive preference fo rmation.  See, e.g.,  Forgas  et al.
(2009 ).
12 In formal models, the value of information is quantified a s the difference between the expected value
realized by optimal decisions made with that information and the expected value realized by optimal
decision s made without it.  (See, e.g.,  Russell  & Norvig  2010 .)   It follows that the value of information is
never negative.  It also follows that any information you know will never affect any decision you will ever
make has zero value for you.  However, this kind of m odel assumes several idealizations which are often
invalid in the real world —such as  that knowledge has no final value (meaning that knowledge has only
instrumental value and is not valuable for its own sake), and that agents are not transparent to other
agents.
10
 less intellig ence.13  The same can hold if the agent has final goals that involve bei ng ignorant of
certain facts: likewise if an agent faces incentives arising from strategic commitments, signaling,
or social preferences, as noted  above.14
 Each of these countervailing reasons often comes into play  for human beings .  Much
information is irrelevant to our goals; we can often rely on others’ skill and expertise; acquiring
knowledge takes time and effort; we might intrinsically value certain kinds of ignorance; and we
opera te in an environment in which the ability to make strategic commitments, socially signal,
and satisfy other people’s direct preferences over our own epistemic states , is often important to
us than simple cognitive gains .
 There are special situations in wh ich cognitive enhancement may result in an enormous
increase in an agent’s ability to achieve its final goals —in particular, if the agent’s final goals are
fairly unbounded and the agent is in a position to become the first superintelligence and thereby
potentially  obtain a decisive advantage  enabling the agent to shape the future of Earth -
originating life and accessible cosmic resources according to its preferences .  At least in this
special case, a rational intelligent agent would place a very high instru mental value on cognitive
enhancement.

2.4  Technolog ical perfection
An agent may often have instrumental reasons  to seek better technology, which at its simplest
means seeking more efficient ways of transforming some given set of inputs into valued outputs.
Thus , a software agent might place an instrumental value on more efficient algorithms that
enable its mental functions to run faster on given hardware.  Similarly, agents whose goals
require some form of physical construction might instrumentally value improved engineering
technology which  enable s them to create a wider range of structures more quickly and reliably ,
using fewer or cheaper materials and less energy.  Of course , there is a tradeoff:  the p otential
benefits of better  technology mus t be weighed against its costs, including not only the cost of
obtaining the technology but also the costs of learning how to use it, integrating it with other
technologies already in us e, and so forth.
 Proponents of some new technology, confident in its superiority to existing alternatives ,
are often dismayed  when other  people  do not share their enthusiasm, but peoples’ resistance to
novel and nominally superior technology need not be b ased on ignorance or irrationality .  A
technology’s valence or normative character depends not only on the context in which it is
deployed, but also the vantage point from which its impacts are evaluated: what is a boon from
one person’s perspective can be  a liability from another’s.  Thus, although mechanized looms
increased the economic efficie ncy of textile production, the Luddite  handloom weavers who

13 This strategy is exemplified by the sea squirt larva, which swims about until it finds a suitable rock, to
which it then permanently affixes itself.  Cemented in place, the larva has less need for complex
information processing, whence it proceeds  to digest part of its own brain (its cerebral ganglion).
Academics can sometimes observe a similar phenomenon in colleagues who are granted tenure.
14 Cf. Bostrom  (2012 ).
11
 anticipated that the innovation would render their artisan skills obsolete may have had good
instrumenta l reason s to oppose it.  The point here is that if “technolog ical perfection” is to name
a widely  convergent instrumental goal for intelligent agents, then the term must be understood
in a spec ial sense—technology must be construed as embedded in a particu lar social context, and
its costs and benefits must be evaluated with reference to some specified  agents’ final values.
 It seems that a superintelligent singleton—a superintelligent agent that faces no
significant intelligent rivals or opposition , and is thus in a position to determ ine global policy
unilaterally —would have instrumental reason to perfect the technologies that would make it
better able to shape the world according to its preferred designs.15  This would probably include
space colonization tec hnology, such as von Neumann probes —automatic, self -mending and self -
replicating spaceships that can extend its reach beyond the Solar System.  Molecular
nanotechnology, or some alternative still more capable physical manufacturing technology , also
seem s potentially very  useful in the service of an extremely wide range of final goals.16

2.5  Resource acquisition
Finally, resource acquisition is a nother  common emergent instrumental goal, for muc h the same
reasons as technological perfection: both technology  and resources facilitate physical
construction projects.
 Human beings tend to seek to acquire resources sufficient to meet their basic biological
needs.  But people usually seek to acquire resources far beyond this minim um level.  In doing so,
they may b e partially driven  by lesser physical desiderata, such as in creased comfort and
convenience.  A  great deal of resource accumulation is motivated by social concerns —gaining
status, mates, friends  and influence, through wealth accumulation and conspicuous
consumption. Perhaps l ess commonly, some people  seek additional resources to achieve
altruistic or expensive non -social aims.

15 Cf. Bostrom  (2006 ).
16 One could reverse the question and look instead at possible reasons for a superintelligent singleton not to
develop some technolo gical capabilities.  These include:  (a) The singleton foreseeing that it will have no
use of some technolo gical capability ; (b) The development c ost being too large relative to its anticipated
utility .  This would be the case if, for instance , the technology will never be suitable for achieving any of the
singleton’s ends, or if the singleton has a very high discount rate that s trongly discourages investment; (c)
The singleton having some final value that requires abstention from particular avenues of technology
development; (d) If the singleton is not certain it will remain stable, it might prefer to refrain from
developing tech nologies that could threaten its internal stability or that would make the consequences of
dissolution worse (e.g., a world government may not wish to develop technologies  that would facilitate
rebellion, even if they had some good uses, nor develop t echno logies for the easy production of weapons of
mass destruction which could wreak havoc i f the world government were  to dissolve); (e) Similarly, the
singleton might have made some kind of binding strategic commitment not to develop some technology, a
commit ment that remains operative even if it would now be convenient to develop it.  (Note, however, that
some current  reasons for technology -development would not apply to a singleton: e.g., reasons arising from
unwanted arms races.)
12
  On the basis of such  observations  it might be tempting to suppose that a
superintelligence not facing a competitive social world wo uld see no instrumental reason  to
accumulate resources beyond some modest level, for instance whatever computational resources
needed to run its mind  along with some virtual reality.  Yet  such a supposition would be entirely
unwarranted.   First, the value of resources depends on the uses to which they can be put, which
in turn depends on the available technology.  With mature technology, basi c resources such as
time, space,  and matter , and other forms of free energy , could be processed  to serve almost any
goal.  For instance, such basic resources could be converted into life.  Increased computational
resources could be used to run the super intelligence at a greater speed and  for a longer duration,
or to create additional physical or simulated (virtual) lives  and civilizations.  Extra  physical
resources could also be used to create backup systems or perimeter defenses, enhancing security.
Such projects could easily c onsume far more than one planet’s worth of resources.
 Furthermore, the cost of acquiring additional extraterrestrial resources  will decline
radically as the technology matures .  Once von Neumann probes can be built, a large portion of
the observable universe (assuming it is uninhabited by intelligent lif e) could be gradually
colonized —for the one -off cost of building and launching a single successful self -reproducing
probe.   This low cost of celestial  resource acquisition would mean that such expansion could be
worthwhile even if the value of the additional resources gained were somewhat marginal.  For
example, even if a superintelligence cared non -instrumentally only about what happens within
some pa rticular small volume of space, such as the space occupied by its origina l home planet,  it
would still have in strumental reasons to harvest the resources of the cosmos beyond.  It could
use those surplus resources to build computers to calculate  more optimal way s of using
resources within the small spatial region of primary concern.  It could also use the extra
resources to build ever -more robust defenses to safeguard the privileged real estate.  Since the
cost of acquiring additional resources would keep declining, this  process of optimizing and
increasing safeguards might well continue indefinitely even if it were subject to steeply declining
returns.17 18

17 Suppose that an agent dis counts resources obtained in the future at an exponential rate, and that because
of the light speed limitation the agent can only increase its resource e ndowment at a polynomial rate.
Would this mean that there will be some time after which the agent woul d not find it worthwhile to
continue acquisitive expansion?  No, because although the present value of the resources obtained at future
times would asymptote to zero the further into the future we look, so would the present cost of obtaining
them.  The pre sent cost of sending out one more von Neumann probe a 100 million years from now
(possibly using some resource acquired some short time earlier) would be diminished by the same
discount factor that would diminish the present value of the future resources t he extra probe would
acquire (modulo a constant factor).
18 Even an agent that has an apparently very limited final goal, such as “to make 32 paperclips”, could
pursue unlimited resource acquisition if there were no relevant cost to the agent of doing so.  For example,
even after an expected -utility -maximizing agent had built 32 paperclips, it could use some extra resources
to verify that it had indeed successfully built 32 paperclips meeting all the specifications (and, if necessary,
to take corrective acti on).  After it had done so, it could run another batch of tests to make doubly sure that
no mistake had been made.  And then it could run another test, and another.  The benefits of subsequent
tests would be subject to steeply diminishing returns; however,  so long as there were no alternative action
13
  Thus, there is an extremely wide range of possible final goals a superintelligent singleton
could  have that would generate  the instrumental goal of  unlimited resource acquisition .  The
likely manifestation of thi s would be the superintelligence’s initiation of a colonization process
that would expand in all directions using von Neumann probes .  This would roughly result in a
sphere of expanding infrastructure centered on the originating planet and growing in radiu s at
some fraction of the speed of light; and the colonization of the universe would continue in this
manner  until the accelerating speed of cosmic expansion ( a consequence  of the positive
cosmological constant) makes further material acquisition physicall y impossible  as remoter
regions drift permanently out of reach .19  By contrast, agents lacking the technology required  for
inexpensive resource acquisition , or for the conversion of generic physical resources into useful
infrastructure , may often find it not cost-effective to invest any present resources in increa sing
their material endowment.   The same may hold for agents operating in competition with other
agents of similar powers.  For instance, if competing agents have already secured accessible
cosmic  resources, a late -starting agent may have no colonization opportunities.   The conver gent
instrumental reasons for superintelligences un certain of the non-existence of other powerful
superintelligent agents are complicated by strategic considerations in wa ys that we do not
currently fully comprehend  but which  may constitute important qualifications to the examples
of convergent instrumental reasons we have looked at here.20
 It should be emphasized that the existence of convergent instrumental reasons, even if
they apply to and are recognized by a particular agent, does not imply that the agent’s behavior
is easily predictable.  An agent might well think of ways of pursuing the relevant instrumental
values that do not readily occur to us.  This is especially true for a superintelligence, which could
devi se extremely clever but counterintuitive plans to realize its goals, possibly even exploiting
as-yet undiscovered physical phenomena.  What is predictable is that the convergent

with a higher expected utility, the agent would keep testing and re -testing (and keep acquiring more
resources to enable these tests).
19 While the volume reached by colonization probes at a given time might be roughly spherical and
expanding with a rate proportional to the square of time elapsed since the first probe was launched (~t2),
the amount of resources contained within this volume will follow a less regular growth pattern, since the
distribution of resou rces is inhomogeneous and varies over several scales.  Initially, the growth rate might
be ~t2 as the home planet is colonized; then the growth rate might become spiky as nearby planets and
solar systems are colonized; then, as the roughly disc -shaped volu me of the Milky Way gets filled out, the
growth rate might even out, to be approximately proportional to t; then the growth rate might again
become spiky as nearby galaxies are colonized; then the growth rate might again approximate ~t2 as
expansion procee ds on a scale over which the distribution of galaxies is roughly homogeneous; then
another period of spiky growth followed by smooth ~t2 growth as galactic superclusters are colonized; until
ultimately the growth rate starts a final decline, eventually rea ching zero as the expansion speed of the
universe accelerates to such an extent as to make further colonization impossible.
20 The simulation argument may be of particular importance in this context.  A superintelligent agent may
assign a significant probab ility to hypotheses according to which it lives in a computer simulation and its
percept sequence is generated by another superintelligence, and this might various generate convergent
instrumental reasons depending on the agent’s guesses about what types o f simulations it is most likely to
be in.  Cf. Bostrom  (2003 ).
14
 instrumental values would be pur sued and used to realize the agent’s final goals, not the specific
actions that the agent would take to achieve this.

Conclusions
The orthogonality thesis suggests that we cannot blithely assume that a superintelligence will
necessarily share any of the f inal values stereotypically associated with wisdom and in tellectual
development in humans —scientific curiosity, benevolent concern for others, spiritual
enlightenment  and contemplation , renunciation of material acquisitiveness , a taste for refined
culture or for the simple pleasures in life , humility and selflessness, and so forth.  It might be
possible  through deliberate effort  to construct  a superintelligence that values such  things , or to
build one that values human welfare, moral goodness, or any other complex purpose that its
designers might want it to ser ve.  But it is no less possible—and probably technically easie r—to
build a superintelligence that places final value on nothing but calculating  the decimals of pi .
 The instrumental convergence thesis suggests that we cannot blithely assume that a
superintelligence with the final goal of calculating the decimals of pi (or making paperclips, or
counting grains of sand) would limit its activities in such a way as to not materially infringe on
human intere sts.  An agent with such a final goal would have a convergent instrumental reason ,
in many situations, to acquire an unlimited amount of physical resources and, if possible, to
eliminate potential threats  to itself and its goal system .21  It might be possib le to set up  a situation
in which the optimal way for the agent to pursue these instrumental values (and thereby its final
goals) is by promoting human welfare, acting morally, or serving some beneficial purpose as
intended by its creators.  However, if and when such an agent finds itself in a different situation,
one in which it expects a greater number of decimals of pi to be calculated if it destroys the
human species than if it continues to act cooperatively, its behavior would instantly take a
sinister  turn.  This indicates a danger in relying on instrumental values as a guarantor of safe
conduct  in future artificial agents that are intended to become superintelligent and that might be
able to leverage their superi ntelligence into extreme levels power a nd influence .22

References
Bostrom, N. (2003). Are You Living in a Computer Simulation? Philosophical Quarterly , 53(211),
243-255.

Bostrom. N. (2006). What is a Singleton? Linguistic and Philosophical Investigations , 5(2), 48 -54.

21 Human beings might constitute potential threats; they certainly c onstitute physical resources.
22 For co mments and discussion I am grateful to Stuart Armstrong, Grant Bartley, Owain Evans, Lisa
Makros, Luke Muehlhauser, Toby Ord, Brian Rabkin, Rebecca Roache, Anders Sandberg, and three
anonymous referees.
15
 Bostrom, N. (2012 ). Information Hazards: A Typology of Potentia l Harms from Knowledge.
Review of Contemporary Phil osophy , 10, 44 -79. [www.nickbostrom.com/information -hazards.pdf ]

Chalmers, D. (2010): The Singularity: A Philosophi cal Analysis. Journal of Consciousness Studies ,
17, 7 -65.

Chislenko, A. (1997). Technology as Extension of Human Functional Architecture. Extropy
Online . [project.cyberpunk.ru/idb/technology_ as_extension.html ]

de Blanc, P. (2011). Ontological Crises in Artificial Agent’s Value Systems. Manuscript . The
Singularity Institute for Artificial In telligence.  [arxiv.org/pdf/1105.3821v1.pdf ]

Dewey, D. (2011). Learning What to V alue. In Schmidhuber, J., Thori sson, K. R., Looks, M. (eds.).
Proceedings of the 4th Conference on Artificial General Intelligence , AGI 2011 (pp. 309 -314),
Heidelberg: Springer.

Forgas, J. et al. (eds.) (2009). The Psychology of A ttitude s and Attitude Change . London:  Psychology
Press.

Lewis, D. (1988). Desire as belief. Mind , 97(387), 323 -332.

Omohundro, S. (2008a). The Basic AI Drives. In P. Wang, B. Go ertzel, and S. Franklin (eds.).
Proceedings of the First AGI Conference , 171 , Frontiers  in Artificial Intelligence and  Appli cations.
Amsterdam: IOS Press.

Omohundro, S. (2008b). The Nature of Self -Improving Artificial Intel ligence. Manuscript .
[selfawaresystems.files.wordpress.com/2008/01/nature_of_self_improving_ai.pdf ]

Omohundro, S. (forthcoming 2012). Rationally -Shaped Artificial Intelligence. In Eden, A.  et al.
 (eds.). The Singularity Hypothesis: A Scient ific and Philosophical Assessment  (Springer, forth coming).

Parfit, D. (1984). Reasons and Persons . (pp. 123 -4). Reprinted and c orrected edition, 1987. Oxford:
Oxford University Press.

Parfit, D. (2011). On What Matters . Oxford: Oxford University Press.

Russell, S. and Norvig, P. ( 2010). Artificial Intelligence: A Modern Approach . (3rd ed. ). New  Jersey:
Prentice Hall.

Sandberg, A. and Bostrom, N. (2008). Whole Brain Emulation: A Roadmap . Technical Report  2008–
3. Oxford: Future of Humanity Institute, O xford Univer sity.
16
 [www.fhi.ox.ac.uk/Reports/200 8-3.pdf ]

Shulman, C. (2010). Omohundro’s “Basic AI Drives” and Cata strophic Risks. Manuscript .
[singinst.org /upload/ai -resource -drives.pdf ]

Sinhababu, N. (2009). The Humean Theory of Motiva tion Reformulated and Defended.
Philosophical Review , 118(4), 465 -500.

Smith, M. (1987). The Humean Theory of Motivation. Mind , 46 (381): 36 -61.

Weizenbaum, J. (1976). Computer Power and Human Reason: Fro m Judgment to Calculation . San
Francisco: W. H. Freeman.

Yudkowsky, E. (2008). Artificial Intelligence as a Positive and Negative Factor i n Global Ri sk. In
Bostrom, N. and Cirkovic, M. (eds.). Global Catastrophic R isks. (pp. 308 -345; quote from p. 310).
Oxford: Oxford University P ress.

Yudkowsky, E. (2011). Complex Value Systems Are Required to Realize Valuable Futures. In
Schmidhuber , J., Thorisson, K. R., Looks, M. (eds.). Procee dings of the 4th Conference on  Artificial
General Intelligence , AGI 2011 (pp. 388 -393). Heidelberg:  Springer.