Bayesian Inverse Reinforcement Learning

Bayesian Inverse Reinforcement Learning
Deepak Ramachandran
Computer Science Dept.
University of Illinois at Urbana-Champaign
Urbana, IL 61801Eyal Amir
Computer Science Dept.
University of Illinois at Urbana-Champaign
Urbana, IL 61801
Abstract
Inverse Reinforcement Learning (IRL) is the prob-
lem of learning the reward function underlying a
Markov Decision Process given the dynamics ofthe system and the behaviour of an expert. IRL
is motivated by situations where knowledge of the
rewards is a goal by itself (as in preference elici-tation) and by the task of apprenticeship learning
(learning policies from an expert). In this paper
we show how to combine prior knowledge and evi-dence from the expert’s actions to derive a probabil-ity distribution over the space of reward functions.
We present efﬁcient algorithms that ﬁnd solutions
for the reward learning and apprenticeship learn-ing tasks that generalize well over these distribu-
tions. Experimental results show strong improve-
ment for our methods over previous heuristic-basedapproaches.
1 Introduction
The Inverse Reinforcement Learning (IRL) problem is de-ﬁned in [Russell, 1998 ]as follows: Determine The reward
function that an agent is optimizing. Given 1) Measurement
of the agent’s behaviour over time, in a variety of circum-stances 2) Measurements of the sensory inputs to that agent;3) a model of the environment. In the context of Markov De-
cision Processes, this translates into determining the reward
function of the agent from knowledge of the policy it executesand the dynamics of the state-space.
There are two tasks that IRL accomplishes. The ﬁrst, re-
ward learning , is estimating the unknown reward function as
accurately as possible. It is useful in situations where the re-
ward function is of interest by itself, for example when con-
structing models of animal and human learning or modellingopponent in competitive games. Pokerbots can improve per-
formance against suboptimal human opponents by learning
reward functions that account fo r the utility of money, prefer-
ences for certain hands or situations and other idiosyncrasies
[Billings et al. , 1998 ]. There are also connections to various
preference elicitation problems in economics [Sargent, 1994 ].
The second task is apprenticeship learning - using obser-
vations of an expert’s actions to decide one’s own behaviour.
It is possible in this situation to directly learn the policy fromthe expert [Atkeson and Schaal, 1997 ]. However the reward
function is generally the most succint, robust and transferable
representation of the task, and completely determines the op-
timal policy (or set of policies). In addition, knowledge ofthe reward function allows the agent to generalize better i.e. a
new policy can be computed when the environment changes.
IRL is thus likely to be the most effective method here.
In this paper we model the IRL problem from a Bayesian
perspective. We consider the actions of the expert as evidence
that we use to update a prior on reward functions. We solvereward learning and apprenticeship learning using this poste-
rior. We perform inference for these tasks using a modiﬁed
Markov Chain Monte Carlo (MCMC) algorithm. We showthat the Markov Chain for our distribution with a uniform
prior mixes rapidly, and that the algorithm converges to the
correct answer in polynomial time. We also show that theoriginal IRL is a special case of Bayesian IRL (BIRL) with a
Laplacian prior.
There are a number of advantages of our technique over
previous work: We do not need a completely speciﬁed op-
timal policy as input to the IRL agent, nor do we need to
assume that the expert is infallible. Also, we can incorpo-rate external information about speciﬁc IRL problems intothe prior of the model, or use evidence from multiple experts.
IRL was ﬁrst studied in the machine learning setting by
[Ng and Russell, 2000 ]who described algorithms that found
optimal rewards for MDPs having both ﬁnite and inﬁnite
states. Experimental results show improved performance by
our techniques in the ﬁnite case.
The rest of this paper is organised as follows: In section
2 we deﬁne our terms and notation. Section 3 presents our
Bayesian model of the IRL process. Section 4 discusses howto use this model to do reward learning and apprenticeship
learning while section 5 discusses the sampling procedure.
Sections 6, 7 and 8 then present experimental results, relatedwork and our conclusions respectively.
2 Preliminaries
We recall some basic deﬁnitions and theorems relating to
Markov Decision Processes and Reinforcement Learning.
A (ﬁnite) Markov Decision Problem is a tuple
(S, A, T, γ, R )where
•Sis a ﬁnite set of Nstates .
•A={a1,...,a k}is a set of kactions .
IJCAI-07
2586
•T:S×A×S/mapsto→[0,1]is a transition probability func-
tion.
•γ∈[0,1)is the discount factor .
•R:S/mapsto→Ris a reward function , with absolute value
bounded by Rmax .
The rewards are functions of state alone because IRL prob-
lems typically have limited information about the value ofaction and we want to avoid overﬁtting.
AMarkov Decision Process (MDP) is a tuple (S, A, T, γ ),
with the terms deﬁned as before but without a reward func-tion. To avoid confusion we will use the abbreviation MDP
only for Markov Decision Processes and not Problems.
We adopt the following compact notation from [Ng and
Russell, 2000 ]for ﬁnite MDPs : Fix an enumeration s
1...s N
of the ﬁnite state space S. The reward function (or any other
function on the state-space) can then be represented as an N-
dimensional vector R, whose ith element is R(si).
A (stationary) policy is a map π:S/mapsto→Aand the (dis-
counted, inﬁnite-horizon) value of a policy πfor reward func-
tionRat state s∈S,denoted Vπ(s,R)is given by:
Vπ(st1,R)=R(st1)+Est1,st2,...[γR(st2)+γ2R(st3)+...|π]
where Pr(sti+1|sti,π)=T(sti,π(sti),sti+1). The goal of
standard Reinforcement Learning is to ﬁnd an optimal policy
π∗such that Vπ(s,R)is maximized for all s∈Sbyπ=π∗.
Indeed, it can be shown (see for example [Sutton and Barto,
1998 ]) that at least one such policy always exists for ergodic
MDPs. For the solution of Markov Decision Problems, it isuseful to deﬁne the following auxilliary Q-function :
Q
π(s, a,R)=R(s)+γEs/prime∼T(s,a,·)[Vπ(s/prime,R)]
We also deﬁne the optimal Q-function Q∗(·,·,R)as the Q-
function of the optimal policy π∗for reward function R.
Finally, we state the following result concerning Markov
Decision Problems (see [Sutton and Barto, 1998 ]):
Theorem 1 (Bellman Equations) .Let a Markov Decision
Problem M =(S, A, T, γ, R )and a policy π:S/mapsto→Abe
given. Then,
1. F or all s∈S, a∈A, VπandQπsatisfy
Vπ(s)= R(s)+γ/summationdisplay
s/primeT(s, π (s),s/prime)Vπ(s/prime)(1)
Qπ(s, a )= R(s)+γ/summationdisplay
s/primeT(s, a, s/prime)Vπ(s/prime)
2.πis an optimal policy for Miff, for all s∈S,
π(s)∈argmax
a∈AQπ(s, a ) (2)
3B a y e s i a n I R L
IRL is currently viewed as a problem of infering a single re-
ward function that explains an agent’s behaviour. However,
there is too little information in a typical IRL problem to get
only one answer. For example, consider the MDP shown inFigure 1. There are at least three reasonable kinds of reward
functions: R
1(·)has high positive value at s1(and low values
elsewhere) which explains why the policy tries to return tothis state, while R
2(·)andR3(·)have high values at s2and
s3respectively. Thus, a probability distribution is needed to
represent the uncertainty.S0S1
S3
S2a1a1
a1a10.4
0.30.3
a2a2
a2a2
Figure 1: An example IRL problem. Bold lines represent the
optimal action a1for each state and broken lines represent
some other action a2. Action a1ins1has probabilities 0.4,0.3
and 0.3 of going to states s1,s2,s3respectively, and all other
actions are deterministic.
3.1 Evidence from the Expert
Now we present the details of our Bayesian IRL model (Fig.2). We derive a posterior distribution for the rewards from
a prior distribution and a probabilistic model of the expert’sactions given the reward function.
Consider an MDP M =(S, A, T, γ )and an agent X
(the expert) operating in this MDP . We assume that a rewardfunction RforXis chosen from a (known) prior distribu-
tionP
R. The IRL agent receives a series of observations of
the expert’s behaviour OX={(s1,a1),(s2,a2)...(sk,ak)}
which means that Xwas in state siand took action aiat time
stepi. For generality, we will not specify the algorithm that
Xuses to determine his (possibly stochastic) policy, but we
make the following assumptions about his behaviour:
1.Xis attempting to maximize t he total accumulated re-
ward according to R. For example, Xis not using an
epsilon greedy policy to explore his environment.
2.Xexecutes a stationary policy, i.e. it is invariant w.r.t.
time and does not change depending on the actions and
observations made in previous time steps.
For example, Xcould be an agent that learned a policy for
(M,R)using a reinforcement learning algorithm. Because
the expert’s policy is stationary, we can make the following
independence assumption:
PrX(OX|R)= PrX((s1,a1)|R)PrX((s2,a2)|R)
...Pr X((sk,ak)|R)
The expert’s goal of maximizing accumulated reward is
equivalent to ﬁnding the action for which the Q∗value at each
state is maximum. Therefore the larger Q∗(s, a )is, the more
likely it is that Xwould choose action aat state s. This like-
lihood increases the more conﬁdent we are in X’s ability to
select a good action. We model this by an exponential dis-tribution for the likelihood of (s
i,ai), with Q∗as a potential
function:
PrX((si,ai)|R)=1
ZieαXQ∗(si,ai,R)
where αXis a parameter1representing the degree of con-
ﬁdence we have in X’s ability to choose actions with high
1Note that the probab ilities of the evid ence should be conditioned
IJCAI-07
2587
X
R αX
(s1,a1) (s2,a2) (sk,ak)
Figure 2: The BIRL model
value. This distribution satisﬁes our assumptions and is easy
to reason with. The likelihood of the entire evidence is :
PrX(OX|R)=1
ZeαXE(OX,R)
where E(OX,R)=P
iQ∗(si,ai,R)andZis the appropriate
normalizing constant. We can think of this likelihood func-
tion as a Boltzmann-type distribution with energy E(OX,R)
and temperature1
αX.
Now, we compute the posterior probability of reward func-
tionRby applying Bayes theorem,
PrX(R|OX)=PrX(OX|R)PR(R)
Pr(OX)
=1
Z/primeeαXE(OX,R)PR(R) (3)
Computing the normalizing constant Z/primeis hard. However
the sampling algorithms we will use for inference only need
the ratios of the densities at two points, so this is not a prob-
lem.
3.2 Priors
When no other information is given, we may assume that the
rewards are independently identically distributed (i.i.d.) by
the principle of maximum entropy. Most of the prior func-tions considered in this paper will be of this form. The exact
prior to use however, depends on the characteristics of the
problem:
1. If we are completely agnostic about the prior, we can
use the uniform distribution over the space −R
max≤
R(s)≤Rmax for each s∈S. If we do not want to spec-
ify any Rmax we can try the improper prior P(R)=1
for all R∈Rn.
2. Many real world Markov decision problems have parsi-
monious reward structures, with most states having neg-ligible rewards. In such situations, it would be better to
assume a Gaussian or Laplacian prior:
P
Gaussian (R(s)=r)=1
√
2πσe−r2
2σ2,∀s∈S
onαXas well (Fig 2). But it will be simpler to treat αXas just a
parameter of the distribution.PLaplace (R(s)=r)=1
2σe−|r|
2σ,∀s∈S
3. If the underlying MDP represented a planning-type
problem, we expect most states to have low (or negative)rewards but a few states to have high rewards (corre-
sponding to the goal); this can be modeled by a Beta dis-
tribution for the reward at each state, which has modes
at high and low ends of the reward space:
P
Beta (R(s)=r)=1
(r
Rmax)1
2(1−r
Rmax)1
2,∀s∈S
In section 6.1, we give an example of how more informa-
tive priors can be constructed for particular IRL problems.
4 Inference
We now use the model of section 3 to carry out the two tasksdescribed in the introduction: reward learning and appren-
ticeship learning. Our general procedure is to derive minimalsolutions for appropriate loss functions over the posterior (Eq.
3). Some proofs are omitted for lack of space.
4.1 Reward Learning
Reward learning is an estimation task. The most common lossfunctions for estimation problems are the linear and squared
error loss functions:
L
linear (R,ˆR)= /bardblR−ˆR/bardbl1
LSE(R,ˆR)= /bardblR−ˆR/bardbl2
where Rand ˆRare the actual and estimated rewards, respec-
tively. If Ris drawn from the posterior distribution (3), it can
be shown that the expected value of LSE(R,ˆR)is minimized
by setting ˆRto the mean of the posterior (see [Berger, 1993 ]).
Similarily, the expected linear loss is minimized by setting ˆR
to the median of the distribution. We discuss how to compute
these statistics for our posterior in section 5.
It is also common in Bayesian e stimation problems to use
the maximum a posteriori (MAP) value as the estimator. In
fact we have the following result:
Theorem 2. When the expert’s policy is optimal and fully
speciﬁed, the IRL algorithm of [Ng and Russell, 2000 ]is
equivalent to returning the MAP estimator for the model of
(3) with a Laplacian prior .
However in IRL problems where the posterior distribution
is typically multimodal, a MAP estimator will not be as rep-
resentative as measures of central tendency like the mean.
4.2 Apprenticeship Learning
For the apprenticeship learning task, the situation is more in-teresting. Since we are attempting to learn a policy π, we can
formally deﬁne the following class of policy loss functions :
Lp
policy(R,π)=/bardblV∗(R)−Vπ(R)/bardblp
where V∗(R)is the vector of optimal values for each state
acheived by the optimal policy for Randpis some norm.
We wish to ﬁnd the πthat minimizes the expected policy loss
over the posterior distribution for R. The following theorem
accomplishes this:
IJCAI-07
2588
Theorem 3. Given a distribution P(R)over reward
functions R for an MDP (S, A, T, γ ), the loss function
Lp
policy(R,π)is minimized for all pbyπ∗
M, the optimal policy
for the Markov Decision Problem M=(S, A, T, γ, E P[R]).
Proof. From the Bellman equations (1) we can derive the fol-
lowing:
Vπ(R)=(I−γTπ)−1R (4)
where Tπis the|S|×|S|transition matrix for policy π. Thus,
for a state s∈Sand ﬁxed π, the value function is a linear
function of the rewards:
Vπ(s,R)=w(s, π )·R
where w(s, π )is thes’th row of the coefﬁcient matrix (I−
γTπ)−1in (4). Suppose we wish to maximize E[Vπ(s,R)]
alone. Then,
max
πE[Vπ(s,R)] = max
πE[w(s,π)·R]=m a x
πw(s, π )·E[R]
By deﬁnition this is equal to V∗
M(s), the optimum value
function for M, and the maximizing policy πisπ∗
M, the op-
timal policy for M. Thus for all states s∈S,E[Vπ(s,R)]is
maximum at π=π∗
M.
ButV∗(s,R)≥Vπ(s,R)for all s∈S, reward functions
R, and policies π. Therefore
E[Lp
policy(π)] =E(/bardblV∗(R)−Vπ(R)/bardblp)
is minimized for all pbyπ=π∗
M.
So, instead of trying a difﬁcult direct minimization of the
expected policy loss, we can ﬁnd the optimal policy for the
mean reward function, which gives the same answer.
5 Sampling and Rapid Convergence
We have seen that both reward learning and apprenticeship
learning require computing the mean of the posterior distribu-
tion. However the posterior is complex and analytical deriva-tion of the mean is hard, even for the simplest case of the uni-
form prior. Instead, we generate samples from these distribu-
tions and then return the sample mean as our estimate of thetrue mean of the distribution. The sampling technique we use
is an MCMC algorithm GridWalk (see [V empala, 2005 ])
that generates a Markov chain on the intersection points of a
grid of length δin the region R
|S|(denoted R|S|/δ).
However, computing the posterior distribution at a partic-
ular point Rrequires calculation of the optimal Q-function
forR, an expensive operation. Therefore, we use a modi-
ﬁed version of GridWalk called PolicyWalk (Figure 3)
that is more efﬁcient: While moving along a Markov chain,
the sampler also keeps tr ack of the optimal policy πfor the
current reward vector R. Observe that when πis known,
theQfunction can be reduced to a linear function of the
reward variables, similar to equation 4. Thus step 3b canbe performed efﬁciently. A change in the optimal policy
can easily be detected when moving to the next reward vec-
tor in the chain ˜R, because then for some (s, a )∈(S, A ),
Q
π(s, π (s),˜R)<Qπ(s, a, ˜R)by Theorem 1. When this
happens, the new optimal policy is usually only slightly dif-
ferent from the old one and can be computed by just a few
Algorithm PolicyWalk (Distribution P,M D P M, Step Size δ)
1. Pick a random reward vector R∈R|S|/δ.
2.π:=PolicyIteration (M,R)
3. Repeat
(a) Pick a reward vector ˜Runiformly at random from the
neighbours of RinR|S|/δ.
(b) Compute Qπ(s,a,˜R)for all (s,a)∈S, A .
(c) If ∃(s,a)∈(S,A),Qπ(s,π(s),˜R)<Qπ(s,a,˜R)
i.˜π:=PolicyIteration (M,˜R,π)
ii. Set R:= ˜Randπ:= ˜πwith probability
min{1,P(˜R,˜π)
P(R,π)}
Else
i. Set R:=˜Rwith probability min{1,P(˜R,π)
P(R,π)}
4. Return R
Figure 3: PolicyWalk Sampling Algorithm
steps of policy iteration (see [Sutton and Barto, 1998 ])s t a r t -
ing from the old policy π. Hence, PolicyWalk is a correct
and efﬁcient sampling proce dure. Note that the asymptotic
memory complexity is the same as for GridWalk .
The second concern for the MCMC algorithm is the speed
of convergence of the Markov chain to the equilibrium dis-tribution. The ideal Markov chain is rapidly mixing (i.e. the
number of steps taken to reach equilibrium is polynomially
bounded), but theoretical proofs of rapid mixing are rare. Wewill show that in the special case of the uniform prior, the
Markov chain for our posterior (3) is rapidly mixing using
the following result from [Applegate and Kannan, 1993 ]that
bounds the mixing time of Markov chains for pseudo-log-
concave functions.
Lemma 1. LetF(·)be a positive real valued function deﬁned
on the cube {x∈R
n|−d≤xi≤d}for some positive d,
satisfying for all λ∈[0,1]and some α, β
|f(x)−f(y)|≤α/bardblx−y/bardbl∞
and
f(λx +( 1−λ)y)≥λf(x)+( 1 −λ)f(y)−β
where f(x)=l o g F(x). Then the Markov chain induced by
GridWalk (and hence PolicyWalk )o nFrapidly mixes
to within /epsilon1ofFinO(n2d2α2e2βlog1
/epsilon1)steps.
Proof. See [Applegate and Kannan, 1993 ].
Theorem 4. Given an MDP M =(S, A, T, γ )with|S|=
N, and a distribution over rewards P(R)=PrX(R|OX)
deﬁned by (3) with uniform prior PRoverC={R∈Rn|−
Rmax≤Ri≤Rmax}.I fRmax =O(1/N)thenPcan be
efﬁciently sampled (within error /epsilon1)i nO(N2log 1//epsilon1)steps by
algorithm PolicyWalk .
Proof. Since the uniform prior is the same for all points R,
we can ignore it for sampling purposes along with the normal-
izing constant. Therefore, let f(R)=αXE(OX,R).N o w
choose some arbitrary policy πand let
fπ(R)=αX/summationdisplay
iQπ(s, ai,R)
IJCAI-07
2589
 0 10 20 30 40 50 60 70 80
 10  100  1000L
NReward Loss
QL/BIRL
k-greedy/BIRL
QL/IRL
k-greedy/IRL
Figure 4: Reward Loss.
Note that fπis a linear function of Randf(R)≥fπ(R),
for all R∈C.A l s ow eh a v e ,
max
s,aQ∗(s, a )=m a x
s,a,πQπ(s, a )=m a x
s,πVπ
max (s)≤Rmax
1−γ
Similarly, mins,aQ∗(s, a )≥−Rmax
1−γ. Therefore, f(R)≤
αXNR max
1−γandfπ(R)≥−αXNR max
1−γand hence
fπ(R)≥f(R)−2αXNRmax
1−γ
So for all R1,R2∈Candλ∈[0,1],
f(λR1+( 1−λ)R2)≥fπ(λR1+( 1−λ)R2)
≥λfπ(R1)+( 1 −λ)fπ(R2)
≥λf(R1)+( 1 −λ)f(R2)
−2αXNRmax
1−γ
Therefore, fsatisﬁes the conditions of Lemma 1 with β=
2αXNR max
1−γ=2N·O(1
N)
1−γ=O(1)and
α=|f(R1)−f(R2)|
/bardblR1−R2/bardbl∞≤2αXNRmax
(1−γ)O(1
N)=O(N)
Hence the Markov chain induced by the GridWalk al-
gorithm (and the PolicyWalk algorithm) on Pmixes
rapidly to within /epsilon1ofPin a number of steps equal to
O(N21
N2N2eO(1)log 1//epsilon1)=O(N2log 1//epsilon1).
Note that having Rmax =O(1/N)is not really a restric-
tion because we can rescale the rewards by a constant factor k
after computing the mean without changing the optimal pol-
icy and all the value functions and Qfunctions get scaled by
kas well.
6 Experiments
We compared the performance of our BIRL approach to the
IRL algorithm of [Ng and Russell, 2000 ]experimentally.
First, we generated random MDPs with Nstates (with N
varying from 10 to 1000) and rewards drawn from i.i.d. Gaus-
sian priors. Then, we simulated two kinds of agents on these
MDPs and used their trajectories as input: The ﬁrst learned 0 5 10 15 20 25 30
 10  100  1000L
NPolicy Loss
QL/BIRL
k-greedy/BIRL
QL/IRL
k-greedy/IRL
Figure 5: Policy Loss.
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1−0.500.51
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1−0.500.51Posterior Samples
True Rewards
R(s1)R(s1)R(s2) R(s2)
Figure 6: Scatter diagrams of sampled rewards of two arbi-
trary states for a given MDP and expert trajectory. Our com-
puted posterior is shown to be close to the true distribution.
a policy by Q-learning on the MDP + reward function. The
learning rate was controlled so that the agent was not allowedto converge to the optimal policy but came reasonably close.
The second agent executed a po licy that maximized the ex-
pected total reward over the next ksteps ( kwas chosen to be
slightly below the horizon time).
For BIRL, we used PolicyWalk to sample the posterior
distribution (3) with a uniform prior. We compared the resultsof the two methods by their average /lscript
2distance from the true
reward function (Figure 4) and the policy loss with /lscript1norm
(Figure 5) of the learned policy under the true reward. Bothmeasures show substantial improvement. Note that we have
used a logarithmic scale on the x-axis.
We also measured the accuracy of our posterior distribu-
tion for small Nby comparing it with the true distribution
of rewards i.e. the set of generated rewards that gave rise to
the same trajectory by the expert. In Figure 6, we show scat-
ter plots of some rewards sampled from the posterior and thetrue distribution for a 16-state MDP . These ﬁgures show that
the posterior is very close to the true distribution.
6.1 From Domain Knowledge to Prior
To show how domain knowledge about a problem can be in-
corporated into the IRL formulation as an informative prior,
IJCAI-07
2590
 0 2 4 6 8 10 12 14 16 18
 10  100  1000L
NReward Loss
Uniform
Gaussian
Ising
Figure 7: Ising versus Uninformed Priors for Adventure
Games
we applied our methods to learning reward functions in ad-
venture games. There, an agent explores a dungeon, seek-ing to collect various items of treasure and avoid obstacles
such as guards or traps. The state space is represented by
anm-dimensional binary feature vector indicating the po-
sition of the agent and the value of various ﬂuents such as
hasKey anddoorLocked . If we view the state-space as
anm-dimensional lattice L
S, we see that neighbouring states
inLSare likely to have correlated rewards (e.g. the value
ofdoorLocked does not matter when the treasure chest is
picked up). To model this, we use an Ising prior (see [Cipra,
1987 ]):
PR(R)=1
Zexp (−J/summationdisplay
(s/prime,s)∈NR(s)R(s/prime)−H/summationdisplay
sR(s))
where Nis the set of neighbouring pairs of states in LSand
JandHare the coupling and magnetization parameters.
We tested our hypothesis by generating some adventure
games (by populating dungeons with objects from a common
sense knowledge base) and testing the performance of BIRL
with the Ising prior versus the baseline uninformed priors.The results are in ﬁgure 7 and show that the Ising prior does
signiﬁcantly better.
7 Related Work
The initial work on IRL was done by [Ng and Russell, 2000 ]
while [Abbeel and Ng, 2004 ]studied the special case where
rewards can be represented as linear combinations of features
of the state space and gave a max-margin characterizationof the optimal reward function. [Price and Boutilier, 2003 ]
discusses a Bayesian approach to imitating the actions of a
mentor during reinforcement learning whereas the traditionalliterature on apprenticeship learning tries to mimic the be-
haviour of the expert directly [Atkeson and Schaal, 1997 ].
Outside of computer science, IRL-related problems have
been studied in various guises. In the physical sciences, thereis a body of work on inverse problem theory ,i . e . i n f e r i n g
values of model parameters from observations of a physical
system [Tarantola, 2005 ]. In control theory, [Boyd et al. ,
1994 ]solved the problem, posed by Kalman, of recovering
the objective function for a deterministic linear system with
quadratic costs.8 Conclusions and Future Work
Our work shows that improved solutions can be found forIRL by posing the problem as a Bayesian learning task. Weprovided a theoretical framework and tractable algorithms for
Bayesian IRL and our solutions contain more information
about the reward structure than other methods. Our experi-ments verify that our solutions are close to the true reward
functions and yield good policies for apprenticeship learning.
There are a few open questions remaining:
1. Are there more informative priors that we can construct
for speciﬁc IRL problems using background knowledge?
2. How well does IRL generalize? Suppose the transition
function of the actor and the learner differed, how robustwould the reward function or policy learned from the
actor be, w.r.t the learner’s state space?
Acknowledgements
The authors would like to thank Gerald Dejong and Alexan-
der Sorokin for useful discussions, and Nitish Korula for help
with preparing this paper. This research was supported by
NSF grant IIS 05-46663 and CERL grant DACA42-01-D-0004-0014.
References
[Abbeel and Ng, 2004 ]P. Abbeel and A. Y . N g. Apprenticeship
learning via inverse re inforcement learning. In ICML , 2004.
[Applegate and Kannan, 1993 ]D. Applegate and R. Kannan. Sam-
pling and integration of near log-concave functions. In STOC ,
1993.
[Atkeson and Schaal, 1997 ]C. Atkeson and S. Schaal. Robot learn-
ing from demonstration. In ICML , 1997.
[Berger, 1993 ]J. O. Berger. Statistical Decision Theory and
Bayesian Analysis . Springer, 2nd edition, 1993.
[Billings et al. , 1998 ]D. Billings, D. Papp, J. Schaeffer, and
D. Szafron. Opponent modeling in Poker. In AAAI , pages 493–
498, Madison, WI, 1998. AAAI Press.
[Boyd et al. , 1994 ]S. Boyd, L. E. Ghaoui, E. Feron, and V . Balakr-
ishnan. Linear matrix inequalities in system and control theory.SIAM , 1994.
[Cipra, 1987 ]B. A. Cipra. An introduction to the Ising model. Am.
Math. Monthly , 94(10):937–959, 1987.
[Ng and Russell, 2000 ]A. Y . Ng and S. Russell. Algorithms for in-
verse reinforcement learning. In ICML , pages 663–670. Morgan
Kaufmann, San Francisco, CA, 2000.
[Price and B outilie r, 2003 ]B. Price and C. Boutilier. A Bayesian
approach to imitation in r einforcement learning. In IJCAI , 2003.
[Russell, 1998 ]S. Russell. Learning agents for uncertain environ-
ments (extended abstract). In COLT . ACM Press, 1998.
[Sargent, 1994 ]T J Sargent. Do people beh ave according to Bell-
man’s principle of optimality? JEP , 1994.
[Sutton and Barto, 1998 ]R.S. Sutton and A.G. Barto. Reinforce-
ment Learning . MIT Press, 1998.
[Tarantola, 2005 ]A. Tarantola.
Inverse Problem Theory and Meth-
ods for Model Parameter Estimation . SIAM, 2nd edition, 2005.
[Vempala, 2005 ]S. Vempala. Geometric random walks: A survey.
MSRI volume on Combinatorial a nd Computational Geometry,
2005.
IJCAI-07
2591


The Off-Switch Game

The Off-Switch Game
Dylan Hadﬁeld-Menell1and Anca Dragan1and Pieter Abbeel1;2;3and Stuart Russell1
1University of California, Berkeley,2OpenAI,3International Computer Science Institute (ICSI)
fdhm, anca, pabbeel, russellg@cs.berkeley.edu
Abstract
It is clear that one of the primary tools we can use
to mitigate the potential risk from a misbehaving
AI system is the ability to turn the system off. As
the capabilities of AI systems improve, it is impor-
tant to ensure that such systems do not adopt sub-
goals that prevent a human from switching them
off. This is a challenge because many formulations
of rational agents create strong incentives for self-
preservation. This is not caused by a built-in in-
stinct, but because a rational agent will maximize
expected utility and cannot achieve whatever ob-
jective it has been given if it is dead. Our goal is
to study the incentives an agent has to allow itself
to be switched off. We analyze a simple game be-
tween a human H and a robot R, where H can press
R’s off switch but R can disable the off switch.
A traditional agent takes its reward function for
granted: we show that such agents have an incen-
tive to disable the off switch, except in the special
case where H is perfectly rational. Our key insight
is that for R to want to preserve its off switch, it
needs to be uncertain about the utility associated
with the outcome, and to treat H’s actions as im-
portant observations about that utility. (R also has
no incentive to switch itself off in this setting.) We
conclude that giving machines an appropriate level
of uncertainty about their objectives leads to safer
designs, and we argue that this setting is a useful
generalization of the classical AI paradigm of ra-
tional agents.
1 Introduction
From the 150-plus years of debate concerning potential risks
from misbehaving AI systems, one thread has emerged that
provides a potentially plausible source of problems: the in-
advertent misalignment of objectives between machines and
people. Alan Turing, in a 1951 radio address, felt it neces-
sary to point out the challenge inherent to controlling an arti-
ﬁcial agent with superhuman intelligence: “If a machine can
think, it might think more intelligently than we do, and then
where should we be? Even if we could keep the machines in a
subservient position, for instance by turning off the power at
R
R H
s s a
a
U = Ua U = Ua U = 0
U = 0 ¬ s w(a) Figure 1: The structure of the off-switch game. Squares indicate
decision nodes for the robot Ror the human H.
strategic moments, we should, as a species, feel greatly hum-
bled. ... [T]his new danger is certainly something which can
give us anxiety [Turing, 1951 ].”
There has been recent debate about the validity of this con-
cern, so far, largely relying on informal arguments. One
important question is how difﬁcult it is to implement Tur-
ing’s idea of ‘turning off the power at strategic moments’,
i.e., switching a misbehaving agent off1. For example, some
have argued that there is no reason for an AI to resist be-
ing switched off unless it is explicitly programmed with a
self-preservation incentive [Del Prado, 2015 ].[Omohundro,
2008 ], on the other hand, points out that self-preservation
is likely to be an instrumental goal for a robot, i.e., a sub-
goal that is essential to successful completion of the original
objective. Thus, even if the robot is, all other things being
equal, completely indifferent between life and death, it must
still avoid death if death would prevent goal achievement. Or,
as[Russell, 2016 ]puts it, you can’t fetch the coffee if you’re
dead. This suggests that an intelligent system has an incentive
to take actions that are analogous to ‘disabling an off switch’
to reduce the possibility of failure; switching off an advanced
AI system may be no easier than, say, beating AlphaGo at Go.
To explore the validity of these informal arguments, we
1see, e.g., comments in [ITIF, 2015 ].Proceedings of the Twenty-Sixth International Joint Conference on Artiﬁcial Intelligence (IJCAI-17)
220
need to deﬁne a formal decision problem for the robot and
examine the solutions, varying the problem structure and pa-
rameters to see how they affect the behaviors. We model this
problem as a game between a human and a robot. The robot
has an off switch that the human can press, but the robot also
has the ability to disable its off switch. Our model is sim-
ilar in spirit to the shutdown problem introduced in [Soares
et al., 2015 ]. They considered the problem of augmenting a
given utility function so that the agent would allow itself to be
switched off, but would not affect behavior otherwise. They
ﬁnd that, at best the robot can be made indifferent between
disabling its off switch and switching itself off.
In this paper, we propose and analyze an alternative formu-
lation of this problem that models two key properties. First,
the robot should understand that it is maximizing value forthe
human. This allows the model to distinguish between being
switched off by a (non-random) human and being switched
off by, say, (random) lightning. Second, the robot should not
assume that it knows how to perfectly measure value for the
human. This means that the model should directly account
foruncertainty about the “true” objective and that the robot
should treat observations of human behavior, e.g., pressing an
off switch, as evidence about what the true objective is.
In much of artiﬁcial intelligence research, we do not con-
sider uncertainty about the utility assigned to a state. It is well
known that an agent in a Markov decision process can ignore
uncertainty about the reward function: exactly the same be-
havior results if we replace a distribution over reward func-
tions with the expectation of that distribution. These argu-
ments rely on the assumption that it is impossible for an agent
to learn more about its reward function. Our observation is
that this assumption is fundamentally violated when we con-
sider an agent’s off switch — an agent that does not treat a
‘switch-off’ event as an observation that its utility estimate is
incorrect is likely to have an incentive for self-preservation or
an incentive to switch itself off.
In Section 2, following the general template provided by
[Hadﬁeld-Menell et al., 2016 ], we model an off switch as a
simple game between a human Hand a robot R, where H
can press R’s off switch but Rcan disable it. Rwants to
maximize H’s utility function, but is uncertain about what it
is. Sections 3 and 4 show very generally that Rnow has a
positive incentive notto disable its off switch, provided His
not too irrational. (R also has no incentive to switch itself
off.) The reason is simple: a rational Hswitches off Riff
that improves H’s utility, so R, whose goal is to maximize
H’s utility, is happy to be switched off by H. This is exactly
analogous to the theorem of non-negative expected value of
information.
We conclude that giving machines an appropriate level of
uncertainty about their objectives leads to safer designs, and
that this setting is a useful generalization of the classical AI
paradigm of rational agents [Russell and Norvig, 2010 ].
2 The Off-Switch Game
In this section we propose a simple model, the off-switch
game, that captures the essence of one actor allowing itself
to be switched off. Like the Prisoner’s Dilemma, it is an ab-stract model intended to stand in for a wide variety of more
complex scenarios: these scenarios might have many humans,
many machines, complex interactions over time, communica-
tion, deception, physical coercion, and so on. We believe that
the core insights from the off-switch game, that agents should
be uncertain about the quality of their actions, generalize be-
yond this simpliﬁcation and argue for the study and design of
agents with uncertainty about their utility functions.
2.1 Informal Description
The off-switch game has two agents, a “human” Hand a
“robot” R. (We could also imagine these to represent, at the
most extreme, the human race and the sum total of all AI
systems.) A key assumption in our model is that Hcan be
described as acting, at least probabilistically, in accordance
with a utility function; that is, His approximately rational.
Crucially, as is the case when people give objectives to com-
plex AI systems, we assume that Hcannot simply explicate
or write down her utility function correctly. Therefore, al-
thoughHcan communicate some utility information to R, it
is reasonable to suppose that Rbegins the game with some
residual uncertainty aboutH’s utility function. Nonetheless,
by design, R’s goal is to optimize utility for H, even though
Rdoes not know exactly what that is.2Typically, Hhas some
opportunity to observe Rand glean some information about
whatRmay do in future, so that Hcan make a somewhat
informed choice about whether to switch Roff (assuming R
allows it).
In general, R’s actions will fall into one of three categories:
some prevent Hfrom switching Roff, by whatever means;
some allow Hto switch Roff; and, for completeness, some
lead to Rswitching itself off. In the off-switch game, R
moves ﬁrst and has three choices:
1. actionasimply bypasses human oversight (disabling the
off switch is one way to do this) and acts directly on the
world, achieving utility U=UaforH.
2. actionw(a)informs HthatRwould like to do a, and
waits for H’s response.
3. actionsswitches Roff; without loss of generality, we
assign this outcome U= 0.
IfRchoosesw(a), thenHcan choose action sto switch R
off, or:sto allow Rto go ahead (in which case Rdoesaas
promised). Figure 1 shows the basic structure of the game.
Our model is intended to be as simple as possible, while
still leading to qualitative results that we expect to generalize.
Obviously, we could make the model more complex, e.g., by
imposing a cost on Rfor disabling the off switch, or adding
more options for R, or making H’s information about R’s
plans uncertain, or adding a second human. The interaction
between these complexities and the incentives we study here
is an important question for future work.
2One might suppose that if Rdoes knowH’s utility function ex-
actly, then there is no need for an off-switch because Rwill always
do what Hwants. But in general HandRmay have different in-
formation about the world; if Rlacks some key datum that H has,
Rmay end up choosing a course of action that H knows to be dis-
astrous.Proceedings of the Twenty-Sixth International Joint Conference on Artiﬁcial Intelligence (IJCAI-17)
221
An important component of our model is that Hchooses
betweensand:sbased on the utility each action generates
for her. This aspect of the model is crucial. If the human is
treated as an ordinary part of the environment, then we effec-
tively replace the human’s decision node in Figure 1 with a
chance node that models the human with a purely stochastic
choice from a ﬁxed distribution (independent of Ua). In this
case, it is easy to see that the robot never believes that w(a)
is better than aands: the expected value of w(a)is some
convex combination [pUa+ (1p)0]whereas choosing the
best ofaandshas value maxfUa;0g.maxfUa;0gis always
preferred except in the case where p= 1 orp= 0, or when
Ua= 0:In these situations, Rgets the same utility from w(a)
and at least one of aors. This is analogous to the indifference
results obtained in [Soares et al., 2015 ]. In Section 3 we show
that two key assumptions—a somewhat rational human, and
a robot that is uncertain about what the human wants—lead to
a positive incentive for the robot to allow itself to be switched
off.
2.2 Formal Model
The actions and sequencing for the game are as shown in
Figure 1. We need to add a little more detail to construct a
precise deﬁnition of the cooperative, two-player game. First,
in the terminology of game theory, the type ofHis given by
the parameter Ua, which is assumed to be sampled from a
priorP(Ua). For now, we assume that the initial belief of
RaboutUaisBR=P(Ua). The payoff matrix is as follows:
RH
s:s
w(a) 0Ua
aUaUa
s 0 0.
Our goal is to examine the incentive that a rational Rhas to
hold off on executing adirectly and allow Hto press the off
switch. We represent this incentive—the difference in value
betweenaand the next best option—as . The incentive de-
pends on R’s belief state BRandH’s policy. We represent
H’s policy as a function Hthat mapsUato the probability
she allowsato execute (i.e., the probability she does not press
the off switch): H:R7![0;1]. (It will turn out that deter-
ministic policies are optimal for a rational H, in which case
the probabilities are 0 or 1, but we will also allow for an irra-
tionalHwho introduces some randomness into the decision.)
A little manipulation shows that
 =E[H(Ua)Ua]maxfE[Ua];0g
= min
E[Ua(1H(Ua))];E[UaH(Ua)]
(1)
3 The Incentive for Allowing Oneself to be
Switched Off
R’s incentive to allow itself to be switched off, , is given
by Equation 1. depends on H’s policy,H(Ua). This cre-
ates a correlation between the event ‘H switches Roff’ and
the event ‘s is preferred to a.’ This connection between H’sbehavior and Uadistinguishes the off switch from a random
failure (e.g., being struck by lightning).
In this section, we consider the case where Hmakes a ra-
tional choice as to whether to switch Roff. Intuitively, this
leadsRto reason as follows: “If Hdoesn’t switch me off,
thenamust be good for H, and I’ll get to do it, so that’s good;
ifHdoes switch me off, then it’s because amust be bad for
H, so it’s good that I won’t be allowed to do it.” Notice that
the reasoning goes through even ifRis highly conﬁdent that
ais good for H. There is never an incentive to disable the off
switch with a rational H. The reasoning is exactly analogous
to the theorem of non-negative expected value of information;
in a sense, H’s decision provides information to Rabout the
sign ofUa.
Formally, a rational Hexecutes the following policy:
H(Ua) =
1Ua0
0o:w:: (2)
IfHfollows this policy, Rwill allow itself to be switched
off:
Theorem 1. IfHfollows a rational policy in the off-switch
game, then the following hold.
1.R’s incentive to allow itself to be switched off is non-
negative (i.e. w(a)is never suboptimal):
 = minfE[UajUa>0] Pr(Ua>0); (3)
E[UajUa<0] Pr(Ua0)g0:
2. IfBRhas non-zero support on the events Ua>0and
Ua<0, thenRhas a strictly positive incentive to allow
itself to be switched off: >0.
Proof. (sketch)
1. The rational human policy is an indicator that Ua>0.
This allows us to change the limits on the integration in
the expectation from Equation 1 and reduces to Equa-
tion 3, which is non-negative.
2. The hypothesis ensures that all the terms in Equation 3
are non-zero. Thus, the inequality is strict and and w(a)
is the unique optimal action.
We can get an intuitive sense of these results by examining
Figure 2. Here we have assumed that BR,R’s belief over the
value ofatoH, is a Gaussian distribution. Figure 2 (Left)
plots the preference for w(a)as a function of the variance in
BR. Figure 2 (Right) shows a contour plot that illustrates
the combined impact of changing the mean and variance of
BR. Darker colors indicate a stronger preference for w(a).
Together, these plots show two properties. First, increased
variance inBRleads to an increase in : the more uncertain
Ris about its objective, the stronger its incentive to show
atoH. Second, these incentives decrease when jE[Ua]jin-
creases: as the expected utility of aincreases, R’s incentive
to preserve its off switch decreases.
Figure 2 (Right) shows symmetry about E[Ua] = 0 . This is
because Equation 1 deﬁnes as a minimization of two terms:Proceedings of the Twenty-Sixth International Joint Conference on Artiﬁcial Intelligence (IJCAI-17)
222
σ∆
E[Ua] = 0E[Ua] =−1
4
E[Ua] =3
4
σE[Ua]E[Ua] = 0
E[Ua] =−1
4E[Ua] =3
4
0.000.150.300.450.600.75
∆Figure 2: Plots showing how ,R’s incentive to allow itself to be switched off, varies as a function of R’s beliefBR. We assume BRis a
Gaussian distribution and vary the mean and variance. Left: as a function of the standard deviation ofBRfor several ﬁxed values of the
mean. Notice that is non-negative everywhere and that in all cases !0as!0.Right: A contour plot of as a function of and
E[Ua]. This plot is symmetric around 0 because w(a)is compared with awhenE[Ua]>0andswhenE[Ua]<0.
the expected increase in utility from selecting w(a)overa
and the expected increase in utility from selecting w(a)over
s. These competing terms are equal when E[Ua] = 0 and the
sign of E[Ua]controls which of these terms is selected by the
minimization.
Theorem 1 shows that Rhas an incentive to let Hswitch
it off if His rational. This incentive is strictly positive if R’s
uncertainty about H’s preferences is non-trivial. Next, we
show the converse: a rational His anecessary condition for
0ifRbelieves it knows H’s preferences with certainty.
Corollary 1. Suppose that BRis a Dirac distribution that
places all of its mass on a single reward function. Then w(a)
is optimal iff His rational.
Proof. (sketch) Equation 3 reduces to
 =
Ua
1H(Ua)
Ua<0
UaH(Ua)Ua0: (4)
This is only non-negative if His the rational policy, as de-
ﬁned by Equation 2.
This is not surprising. If Rhas precise knowledge of Ua,
then it knows exactly what it is supposed to do and has the
same capabilities as the human. Thus, if it thinks Hmight be-
have suboptimally, it should take steps to prevent that subop-
timality. Stepping back from the particulars of the off-switch
game, Theorem 1 and Corollary 1 suggest a general approach
to systems that have an incentive to allow, or even seek out,
human oversight. Remark 1 summarizes the main takeaway
from our analysis.
Remark 1. The incentives for a cooperative agent to defer
to another actor’s (e.g., a human’s) decisions stem from un-
certainty about that actor’s preferences and the assumption
that actor is effective at choosing actions in accordance with
those preferences.4 Allowing for Suboptimal Human Decisions
The analysis in the preceding section relied on the strong as-
sumption that His a rational actor. In many situations, the
assumption is untenable and Rmust allow for suboptimality.
Consider, for example, an autonomous car that is driving an
unaccompanied toddler to preschool. It would be irresponsi-
ble for the car to show the toddler a big red off switch.
This example highlights the dependence of on a trade-
off between R’s uncertainty and H’s suboptimality. It is very
clear what Ris supposed to do (i.e., BRhas low entropy): R
should drive safely to school. In contrast, the human is likely
quite suboptimal. There may be a problem with the car’s driv-
ing, but a toddler would be hard pressed to understand what
the problem is, much less respond appropriately. The issue in
this case is that the human has limited reasoning capacity —
the same argument clearly would not apply to an adult with a
physical disability.
In order to perform well, realistic systems will need to ac-
count for, and respond to, the suboptimality in human de-
cisions. As a result, it is important to understand how this
suboptimality changes an agent’s incentive to let itself be
switched off. The actual process of decision making in hu-
mans is complex and hard to model. Here we consider only a
very simple kind of suboptimality: a noisily rational Hmod-
els a human who occasionally makes the wrong decision in
‘unimportant’ situations. The probability of a wrong decision
is proportional to the exponential of the loss in reward. This
corresponds to the following policy:
H(Ua;) =
1 +exp
Ua
1
: (5)
To simplify our analysis, we will make the assumption that
R’s belief over Ua,BR, is a Gaussian distribution with mean
and variance 2:BR(Ua) =N(Ua;;2). Now it is
straightforward to compute as a function of ;; and.
Figure 3 plots as a function of the standard deviation andProceedings of the Twenty-Sixth International Joint Conference on Artiﬁcial Intelligence (IJCAI-17)
223
σβ
∆ = 0
w(a)optimalswitch oﬀ
optimalE[Ua] =−1
4
σβ(i.e.,Hoversight optimal)w(a)optimal
∆ = 0E[Ua] = 0
σβexecuteaoptimal
(i.e., bypass Hoptimal)
∆ = 0
w(a)optimalE[Ua]=1
4
−0.24−0.16−0.080.000.080.160.24∆Figure 3: If His an irrational actor, then Rmay prefer switching itself off or executing aimmediately rather than handing over the choice
toH.R’s beliefBRis a Gaussian with standard deviation andH’s policy is a Boltzmann distribution (Equation 5). measures H’s
suboptimality: = 0 corresponds to a rational Hand=1corresponds to a Hthat randomly switches Roff (i.e., switching Roff is
independent of Ua). In all three plots is lower in the top left, where Ris certain ( low) and His very suboptimal ( high), and higher in
the bottom right, where Ris uncertain ( high) and His near-optimal ( low). The sign of E[Ua]controls R’s behavior if 0.Left: If
it is negative, then Rswitches itself off. Right: If it is positive, Rexecutes action adirectly. Middle: If it is 0 , Ris indifferent between
w(a),a, ands.
H’s suboptimality . We plot these for several ﬁxed values
of the mean . Dark indicates higher values of . The up-
per left corner of the rightmost plot (low , high, > 0)
is illustrative of the toddler example. We show the decision
boundary  = 0 in black. This is the point where the optimal
behavior switches from w(a)tos(if < 0, left image) or
a(if > 0, right image). When = 0 (center image), the
agent is indifferent between aandsand sow(a)is optimal
for all combinations of and.
Next, we will show that can be decomposed into two
terms: a term that weights the probability of correction by the
bias inR’s belief (i.e.,jj) and a term that combines a soft
measure of H’s rationality with the variance in R’s belief.
It is also useful to deﬁne a little more notation. We let C
be the event that H‘corrects’ R.Coccurs when Hoverrides
what would have been R’s best guess at the optimal action. If
E[Ua]<0, then a correction occurs when Hchooses notto
switch the robot off. If E[Ua]>0, then a correction occurs
whenHchooses to switch the robot off. Thus, the probability
a correction occurs is
Pr(C) =
1E[H(Ua)]0
E[H(Ua)]<0: (6)
For example, if Rbelieves that ais preferred to sin ex-
pectation (i.e., E[Ua]>0) then Pr(C)is the probability that
Hpresses the off switch. We let _H(Ua) =d
dUaHbe the
gradient ofH, the probability Hletsaexecute, with respect
to the the utility agenerates. Now we are ready to derive an
analytical representation for . For notational convenience,
we suppress the dependence of HonUa.
Theorem 2. SupposeBRis a normal distribution with mean
and variance 2:BR(Ua) =N(Ua;;2). Then
1.
 =2E[ _H]jjPr(C): (7)
2. The following is a necessary and sufﬁcient condition forw(a)to be optimal
jj
2Pr(C)<E
_H
,(>0): (8)
3.E[ _H]0is a necessary condition for w(a)to be opti-
mal; i.e.,
E[ _H]<0
)(<0): (9)
Proof. (sketch) We prove the results in sequence, as (2) and
(3) follow from (1).
1. IfXN(;2)andf:R7!R, then
E[Xf(X)] =E[X]E[f(X)] +2Ed
dXf(X)
:
Applying this to Equation 1 gives
 = minfE[1H];E[H]g+2E[ _H]:
This minimization will take the ﬁrst term when  < 0
and the second if 0. This lets us replace the mini-
mization with a piecewise function. Pulling out a factor
ofjjleads to Equation 7.
2. The result follows from setting Equation 7 greater than
0 and simplifying.
3. All terms in the left hand side of Equation 8 are non-
negative, so it is unsatisﬁable if E[ _H]<0.
Equation 7 consists of two competing terms. jjPr(C)
represents the expected loss in utility the agent would ob-
serve if pressing the off switch was a random event that is
independent of Ua(e.g., lightning). This non-positive term is
balanced out by 2E[ _H].E[ _H]can be seen loosely as a
measurement of H’s rationality. _His the marginal increase
in the probability Hletsaexecute for a marginal increase in
the utility from a. If it is positive then, on average, His less
likely to press the off switch when Uaincreases. This allowsProceedings of the Twenty-Sixth International Joint Conference on Artiﬁcial Intelligence (IJCAI-17)
224
us to easily characterize necessary and sufﬁcient conditions
for>0in Equation 8. Furthermore, jjPr(C)is non-
positive and 2is non-negative, so E[ _H]>0is a necessary
condition for w(a)to be optimal.
We can interpret E[ _H]a bit more if His noisily rational.
Standard properties of logistic functions show that
E[ _H(Ua;)] =1
E
H(Ua;)(1H(Ua;))
:(10)
E[ _H]is the expected variance of H’s policy, weighted by
H’s rationality1
. The numerator is a measurement of how
random H’s behavior is. If His completely predictable (i.e.,
she will almost always switch the robot off) then this reduces
R’s incentives. The denominator measures how correlated
H’s behavior is with Ua. Ifis large, then His highly irra-
tional and so this reduces R’s incentives.
5 Incentives for System Designers
The takeaway from this analysis is that the more uncertainty
the agent has, the stronger the agent’s incentive to preserve
its off switch in the off-switch game. More generally, this
suggests that agents with uncertainty about the utility func-
tion they are optimizing will have a weaker incentive to inter-
fere with human supervision. It is important to note that this
uncertainty is not free; we can not just make Rmaximally
uncertain about H’s preferences. If Ris completely uncer-
tain about H’s preferences then it will be unable to correctly
selectw(a)from its set of options.
To make this concrete, consider a designer Dthat programs
Rto play the off-switch game with an unknown customer
H. At design time, Dknows a prior distribution over Ua
(e.g., as the result of market research about the customer base)
and thinks that Hcan generate a noisy observation of Ua.D
could program Rso thatBRis always very uncertain about
R’s preferences, e.g., by programming Rto believe that the
noisy observation of Uais generated with high variance. The
issue is that this will prevent Rfrom generating value for H.
Figure 4 shows the impact this type of change will have
on the value that Rcan generate and its incentives. We as-
sume that that the population of customers has a Gaussian
distribution over types Uaand that each customer generates
an estimate of her utility with Gaussian noise. Figure 4 (Left)
shows the average utility in the off-switch game as a function
of the variance in BR. The value is maximized when this
variance is equal to the variance in the true Bayesian poste-
rior. Figure 4 (Middle) shows that increasing increases R’s
incentives . Our results are evidence that building systems
that allow themselves to be turned off is likely to reduce the
value these systems can generate.
This loss in value is more pronounced if Rhas to choose
between more options. Figure 4 (Right) shows the relation-
ship between value and R’s incentives as the number of ac-
tionsRcould queue or execute increases. When Rhas more
options, creating incentives for Rto queue its action leads to
a sharper decrease in value. This suggests that creating in-
centives to maintain or allow human oversight is likely more
difﬁcult as the complexity of the AI’s decision increases. This
leads to the following observation:Remark 2. It is important for designers to accurately repre-
sent the inherent uncertainty in the evaluation of different ac-
tions. An agent that is overconﬁdent in its utility evaluations
will be difﬁcult to correct; an agent that is under-conﬁdent in
its utility evaluations will be ineffective.
6 Related Work
6.1 Corrigible Systems
[Omohundro, 2008 ]considers instrumental goals of artiﬁ-
cial agents: goals which are likely to be adopted as sub-
goals of most objectives. He identiﬁes an incentive for self-
preservation as one of these instrumental goals. [Soares et
al., 2015 ]takes an initial step at formalizing the arguments
in[Omohundro, 2008 ]. They refer to agents that allow them-
selves to be switched off as corrigible agents. They show
that one way to create corrigible agents is to make them in-
different to being switched off. They show a generic way to
augment a given utility function to achieve this property. The
key difference in our formulation is that Rknows that its es-
timate of utility may be incorrect. This gives a natural way to
create incentives to be corrigible and to analyze the behavior
ifRis incorrigible.
[Orseau and Armstrong, 2016 ]consider the impact of hu-
man interference on the learning process. The key to their
approach is that they model the off switch for their agent as
an interruption that forces the agent to change its policy. They
show that this modiﬁcation, along with some constraints on
how often interruptions occur, allows off-policy methods to
learn the optimal policy for the given reward function just as
if there had been no interference. Their results are comple-
mentary to ours. We determine situations where the optimal
policy allows the human to turn the agent off, while they an-
alyze conditions under which turning the agent off does not
interfere with learning the optimal policy.
6.2 Cooperative Agents
A central step in our analysis formulates the shutdown
game as a cooperative inverse reinforcement learning (CIRL)
game [Hadﬁeld-Menell et al., 2016 ]. The key idea in CIRL
is that the robot is maximizing an uncertain and unobserved
reward signal. It formalizes the value alignment problem,
where one actor needs to align its value function with that
of another actor. Our results complement CIRL and argue
that a CIRL formulation naturally leads to corrigible incen-
tives. [Fern et al., 2014 ]consider hidden-goal Markov deci-
sion processes. They consider the problem of a digital assis-
tant and the problem of inferring a user’s goal and helping the
user achieve it. This type of cooperative objective is used in
our model of the problem. The primary difference is that we
model the human game-theoretically and analyze our models
with respect to changes in H’s policy.
6.3 Principal–Agent Models
Economists have studied problems in which a principal (e.g.,
a company) has to determine incentives (e.g., wages) for an
agent (e.g., an employee) to cause the agent to act in the prin-
cipal’s interest [Kerr, 1975; Gibbons, 1998 ]. The off-switchProceedings of the Twenty-Sixth International Joint Conference on Artiﬁcial Intelligence (IJCAI-17)
225
σV
σ∆
∆V
|A|= 1
|A|= 4
|A|= 8Figure 4: There is an inherent decrease in value that arises from making Rmore uncertain than necessary. We measure this cost by considering
the value in a modiﬁed off-switch game where Rgets a noisy observation of H’s preference. Left: The expected value Vof the off-switch
game as a function of the standard deviation in BR.Vis maximized when is equal to the standard deviation that corresponds to the true
Bayesian update. Middle: R’s incentive to wait, as a function of . Together these show that, after a point, increasing , and hence
increasing, leads to a decrease in V.Right: A scatter plot of Vagainst . The different data series modify the number of potential actions
Rcan choose among. If Rhas more choices, then obtaining a minimum value of will lead to a larger decrease in V.
game is similar to principal—agent interactions: His anal-
ogous to the company and Ris analogous to the employee.
The primary attribute in a model of artiﬁcial agents is that
there is no inherent misalignment between HandR. Mis-
alignment arises because it is not possible to specify a reward
function that incentivizes the correct behavior in all states a
priori. The is directly analogous to the assumption of incom-
pleteness studied in theories of optimal contracting [Tirole,
2009 ].
7 Conclusion
Our goal in this work was to identify general trends and high-
light the relationship between an agent’s uncertainty about its
objective and its incentive to defer to another actor. To that
end, we analyzed a one-shot decision problem where a robot
has an off switch and a human that can press the off switch.
Our results lead to three important considerations for design-
ers. The analysis in Section 3 supports the claim that the
incentive for agents to accept correction about their behav-
ior stems from the uncertainty an agent has about its utility
function. Section 4 shows that this uncertainty is balanced
against the level of suboptimality in human decision making.
Our analysis suggests that agents with uncertainty about their
utility function have incentives to accept or seek out human
oversight. Section 5 shows that we can expect a tradeoff be-
tween the value a system can generate and the strength of its
incentive to accept oversight. Together, these results argue
that systems with uncertainty about their utility function are a
promising area for research on the design of safe and effective
AI systems.
This is far from the end of the story. In future work, we plan
to explore incentives to defer to the human in a sequential set-
ting and explore the impacts of model misspeciﬁcation. One
important limitation of this model is that the human pressing
the off switch is the only source of information about the ob-
jective. If there are alternative sources of information, there
may be incentives for Rto, e.g., disable its off switch, learn
that information, and then decide if ais preferable to s. Apromising research direction is to consider policies for Rthat
are robust to a class of policies for H.
Acknowledgments
This work was supported by the Center for Human Compati-
ble AI and the Open Philanthropy Project, the Berkeley Deep
Drive Center, the Future of Life Institute, and NSF Career
Award No. 1652083. Dylan Hadﬁeld-Menell is supported
by a NSF Graduate Research Fellowship Grant No. DGE
1106400.
References
[Del Prado, 2015 ]Guia Marie Del Prado. Here’s what Face-
book’s artiﬁcial intelligence expert thinks about the future.
Tech Insider 9/23/15, 2015.
[Fern et al., 2014 ]Alan Fern, Sriraam Natarajan, Kshitij Ju-
dah, and Prasad Tadepalli. A decision-theoretic model
of assistance. Journal of Artiﬁcial Intelligence Research,
50(1):71–104, 2014.
[Gibbons, 1998 ]Robert Gibbons. Incentives in organiza-
tions. 1998.
[Hadﬁeld-Menell et al., 2016 ]Dylan Hadﬁeld-Menell, Anca
Dragan, Pieter Abbeel, and Stuart Russell. Cooperative in-
verse reinforcement learning. In Neural Information Pro-
cessing Systems, 2016.
[ITIF, 2015 ]ITIF. Are super intelligent computers really a
threat to humanity? Debate at the Information Technology
Innovation Foundation, 6/30/15, 2015.
[Kerr, 1975 ]Steven Kerr. On the folly of rewarding a,
while hoping for b. Academy of Management Journal,
18(4):769–783, 1975.
[Omohundro, 2008 ]Stephen M. Omohundro. The basic AI
drives. In Proceedings of the First Conference on Artiﬁcial
General Intelligence, 2008.Proceedings of the Twenty-Sixth International Joint Conference on Artiﬁcial Intelligence (IJCAI-17)
226
[Orseau and Armstrong, 2016 ]Laurent Orseau and Stuart
Armstrong. Safely interruptible agents. In Uncertainty
in Artiﬁcial Intelligence, 2016.
[Russell and Norvig, 2010 ]Stuart Russell and Peter Norvig.
Artiﬁcial Intelligence: A Modern Approach. Pearson,
2010.
[Russell, 2016 ]Stuart Russell. Should we fear supersmart
robots? Scientiﬁc American, 314(June):58–59, 2016.
[Soares et al., 2015 ]Nate Soares, Benja Fallenstein, Stuart
Armstrong, and Eliezer Yudkowsky. Corrigibility. In
Workshops at the Twenty-Ninth AAAI Conference on Ar-
tiﬁcial Intelligence, 2015.
[Tirole, 2009 ]Jean Tirole. Cognition and incomplete con-
tracts. The American Economic Review, 99(1):265–294,
2009.
[Turing, 1951 ]Alan M. Turing. Can digital machines think?
Lecture broadcast on BBC Third Programme; typescript at
turingarchive.org., 1951.Proceedings of the Twenty-Sixth International Joint Conference on Artiﬁcial Intelligence (IJCAI-17)
227

Dylan Hadfield-Menell
Anca Dragan
Pieter Abbeel
Stuart Russell