Why we need friendly AI

Think
http://journals.cambridge.org/THI
Additional services for Think:
Email alerts: Click here
Subscriptions: Click here
Commercial reprints: Click here
Terms of use : Click here
WHY WE NEED FRIENDLY AI
Luke Muehlhauser and Nick Bostrom
Think / Volume 13 / Issue 36 / March 2014, pp 41 - 47
DOI: 10.1017/S1477175613000316, Published online: 17 December 2013
Link to this article: http://journals.cambridge.org/
abstract\_S1477175613000316
How to cite this article:
Luke Muehlhauser and Nick Bostrom (2014). WHY WE NEED
FRIENDLY AI . Think, 13, pp 41-47 doi:10.1017/S1477175613000316
Request Permissions : Click here
Downloaded from http://journals.cambridge.org/THI, IP address: 163.1.72.155 on 21 Apr 2014

http://journals.cambridge.org Downloaded: 21 Apr 2014 IP address: 163.1.72.155WHY WE NEED FRIENDLY AI
Luke Muehlhauser and Nick Bostrom
Humans will not always be the most intelligent
agents on Earth, the ones steering the future. What
will happen to us when we no longer play that role,
and how can we prepare for this transition?
The human level of intelligence is an evolutionary acci-
dent – a small basecamp on a vast mountain side, far
below the highest ridges of intelligence allowed by physics.
If we were visited by extraterrestrials, these beings would
almost certainly be very much more intelligent and techno-
logically advanced than we are, and thus our future would
depend entirely on the content of their goals and desires.
But aliens are unlikely to make contact anytime soon. In
the near term, it seems more likely we will create our intel-
lectual successors. Computers far outperform humans in
many narrow niches (e.g. arithmetic and chess), and there
is reason to believe that similar large improvements over
human performance are possible for general reasoning and
technological development.
Though some doubt that machines can possess certain
mental properties like consciousness, the absence of such
mental properties would not prevent machines from becom-
ing vastly more able than humans to efficiently steer the
future in pursuit of their goals. As Alan Turing wrote, ‘ ...it
seems probable that once the machine thinking method
has started, it would not take long to outstrip our feeble
powers ...At some stage therefore we should have to
expect the machines to take control ...’
There is, of course, a risk in passing control of the future
to machines, for they may not share our values. This risk is
increased by two factors that may cause the transition from
doi:10.1017/S1477175613000316 #The Royal Institute of Philosophy, 2014
Think 36 , Vol. 13 (Spring 2014)Think Spring 2014 †41
http://journals.cambridge.org Downloaded: 21 Apr 2014 IP address: 163.1.72.155human control to machine control to be quite sudden and
rapid: the possibilities of computing overhang andrecursive
self-improvement .
What is computing overhang? Suppose that computing
power continues to double according to Moore’s law, but
figuring out the algorithms for human-like general intelli-
gence proves to be fiendishly difficult. When the software
for general intelligence is finally realized, there could exist
a ‘computing overhang’: tremendous amounts of cheap
computing power available to run human-level artificial intel-
ligences (AIs). AIs could be copied across the hardware
base, causing the AI population to quickly surpass the
human population. These digital minds might run thou-
sands or millions of times faster than human minds. AIs
might have further advantages, such as superior communi-
cation speed, transparency and self-editability, goal coordi-
nation, and improved rationality.
And what is recursive self-improvement? We can predict
that advanced AIs will have instrumental goals to preserve
themselves, acquire resources, and self-improve, because
those goals are useful intermediaries to the achievement of
almost any set of final goals. Thus, when we build an AI
that is as skilled as we are at the task of designing AI systems,we may thereby initiate a rapid, AI-motivated cascade of self-
improvement cycles. Now when the AI improves itself, it
improves the intelligence that does the improving, quickly
leaving the human level of intelligence far behind.
A superintelligent AI might thus quickly become superior
to humanity in harvesting resources, manufacturing, scienti-
fic discovery, social aptitude, and strategic action, among
other abilities. We might not be in a position to negotiate
with it or its descendants, just as chimpanzees are not in a
position to negotiate with humans.
At the same time, the convergent instrumental goal of
acquiring resources poses a threat to humanity, for it
means that a superintelligent machine with almost anyfinal
goal (say, of solving the Riemann hypothesis) would want
to take the resources we depend on for its own use. SuchMuehlhauser and Bostrom Why We Need Friendly AI †42
http://journals.cambridge.org Downloaded: 21 Apr 2014 IP address: 163.1.72.155an AI ‘does not love you, nor does it hate you, but you are
made of atoms it can use for something else’.1Moreover,
the AI would correctly recognize that humans do not
want their resources used for the AI’s purposes, and that
humans therefore pose a threat to the fulfillment of its
goals – a threat to be mitigated however possible.
But because we will create our own successors, we may
have the ability to influence their goals and make them
friendly to our concerns. The problem of encoding human
(or at least humane ) values into an AI’s utility function is a
challenging one, but it may be possible. If we can build
such a ‘Friendly AI,’ we may not only avert catastrophe, but
also use the powers of machine superintelligence to do
enormous good.
Many scientific naturalists accept that machines can be
far more intelligent and powerful than humans, and that this
could pose a danger for the things we value. Still, they may
have objections to the line of thought we have developed
so far. Philosopher David Chalmers has responded to
many of these objections;2we will respond to only a few of
them here.
First: why not just keep potentially dangerous AIs safely
confined, e.g. without access to the internet? This maysound promising, but there are many complications.
3In
general, such solutions would pit human intelligence
against superhuman intelligence, and we shouldn’t be con-
fident the former would prevail. Moreover, such methods
may only delay AI risk without preventing it. If one AI devel-
opment team has built a human-level or superintelligent AI
and successfully confined it, then other AI development
teams are probably not far behind them, and these other
teams may not be as cautious. Governments will recognize
that human-level AI is a powerful tool, and the race to be
the first nation with such a great advantage may incentivize
development speed over development safety . (Confinement
measures may, however, be useful as an extra precaution
during the development phase of safe AI.)Think Spring 2014 †43
http://journals.cambridge.org Downloaded: 21 Apr 2014 IP address: 163.1.72.155Second: some have suggested that advanced AIs’ greater
intelligence will cause them to be more moral than we are; in
that case, who are we to protest when they do not respect
ourprimitive values? That would be downright immoral!
Intelligent search for instrumentally optimal plans,
however, can be performed in the service of any goal.
Intelligence and motivation are in this sense logically
orthogonal axes along which possible artificial intellects
can vary freely. The imputed connection between intelli-
gence and morality is therefore sheer anthropomorphism.
(It is an anthropomorphism that does not even hold true for
humans: it is easy to find humans who are quite intelligent
but immoral, or who are unintelligent but thoroughly decent.)
Economist Robin Hanson suggests that inter-generational
conflicts analogous to the ones that could arise between
humans and machines are common. Generations old and new
compete for resources, and the older generation often wants
to control the values of the younger generation. The values of
the younger generation end up dominating as the older gener-
ation passes away. Must we be so selfish as to insist that the
values of Homo sapiens dominate the solar system forever?
Along a similar line, the roboticist Hans Moravec once
suggested that while we should expect that future roboticcorporations will eventually overrun humanity and expropri-
ate our resources, we should think of these robotic descen-
dants as our ‘mind children.’ Framed in this way, Moravec
thought, the prospect might seem more attractive.
It must be said that a scenario in which the children kill
and cannibalize their parents is not everyone’s idea of a
happy family dynamic. But even if we were willing to sacri-
fice ourselves (and our fellow human beings?) for the sake
of some ‘greater good,’ we would still have to put in hard
work to ensure that the result would be something more
worthwhile than masses of computer hardware used only
to evaluate the Riemann hypothesis (or to calculate the
decimals of pi, or to manufacture as many paperclips as
possible, or some other arbitrary goal that might be easier
to specify than what humans value).Muehlhauser and Bostrom Why We Need Friendly AI †44
http://journals.cambridge.org Downloaded: 21 Apr 2014 IP address: 163.1.72.155There is, however, one good reason not to insist that
superhuman machines be made to share all our current
values. Suppose that the ancient Greeks had been the
ones to face the transition from human to machine control,
and they coded their own values as the machines’ final
goal. From our perspective, this would have resulted in
tragedy, for we tend to believe we have seen moral pro-
gress since the Ancient Greeks (e.g. the prohibition of
slavery). But presumably we are still far from perfection.
We therefore need to allow for continued moral progress.
One proposed solution is to give machines an algorithm
for figuring out what our values would be if we knew more,
were wiser, were more the people we wished to be, and so
on. Philosophers have wrestled with this approach to the
theory of values for decades, and it may be a productive
solution for machine ethics.
Third: others object that we are too far from the transition
from human to machine control to work on the problem
now. But we must remember that economic incentives
favor development speed over development safety.
Moreover, our scientific curiosity can sometimes overwhelm
other considerations such as safety. To quote J. Robert
Oppenheimer, the physicist who headed the ManhattanProject: ‘When you see something that is technically sweet,
you go ahead and do it and you argue about what to do
about it only after you have had your technical success.
That is the way it was with the atomic bomb.’
4
Still, one might ask: What can we do about the problem
of AI risk when we know so little about the design of future
AIs? For a start, we can do the kind of work currently per-
formed by the two research institutes currently working
most directly on this difficult problem: the Machine
Intelligence Research Institute in Berkeley and the Future
of Humanity Institute at Oxford University. This includes:
1.Strategic research . Which types of technological
development are risk-increasing or risk-
decreasing, and how can we encourageThink Spring 2014 †45
http://journals.cambridge.org Downloaded: 21 Apr 2014 IP address: 163.1.72.155governments and corporations to shift funding
from the former to the latter? What is the
expected value of certain kinds of research, or
of certain kinds of engagement with
governments and the public? What can we do
to reduce the risk of an AI arms race? How
does AI risk compare to risks from nuclear
weapons, biotechnology, near earth objects,
etc.? Can economic models predict anything
about the impact of AI technologies? Can we
develop technological forecasting methods
capable of giving advance warning of the
invention of AI?
2.Technical research . Can we develop safe
confinement methods for powerful AIs? How
can an agent with a desirable-to-humans utility
function maintain its desirable goals during
updates to the ontology over which it has
preferences? How can we extract a coherent
utility function from inconsistent human
behavior, and use it to inform an AI’s own
utility function? Can we develop an advanced
AI that will answer questions but not manifestthe dangerous capacities of a superintelligent
agent?
3.Raising awareness . Outreach to researchers,
philanthropists, and the public can attract more
monetary and human capital with which to
attack the problem.
The quest for AI has come a long way. Computer scientists
and other researchers must begin to take the implications
of AI more seriously.
Luke Muehlhauser is Executive Director of the Machine
Intelligence Research Institute. Nick Bostrom is Director of
the Future of Humanity Institute at the University of Oxford.
luke@intelligence.orgMuehlhauser and Bostrom Why We Need Friendly AI †46
http://journals.cambridge.org Downloaded: 21 Apr 2014 IP address: 163.1.72.155Notes
1E. Yudkowsky, ‘AI as a positive and a negative factor in
global risk’, Global Catastrophic Risks (eds) N. Bostrom and
M. Cirkovic (New York: Oxford University Press, 2008).
2D. Chalmers, ‘The Singularity: a reply to commentators’,
Journal of Consciousness Studies , vol. 19, nos. 7–8 (2012),
141–167.
3S. Armstrong, A. Sandberg, N. Bostrom, ‘Thinking inside
the box: Using and controlling Oracle AI’, Minds and
Machines , vol. 22, no. 4 (2012), 299–324.
4Robert Jungk, Brighter than a Thousand Suns: A Personal
History of the Atomic Scientists , trans. Lames Cleugh
(New York: Harcourt Harvest, 1958), 296.Think Spring 2014 †47


Benefits of Assistance over Reward Learning

Under review as a conference paper at ICLR 2021
BENEFITS OF ASSISTANCE OVER REWARD LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Much recent work has focused on how an agent can learn what to do from human
feedback, leading to two major paradigms. The ﬁrst paradigm is reward learning ,
in which the agent learns a reward model through human feedback that is provided
externally from the environment. The second is assistance , in which the human is
modeled as a part of the environment, and the true reward function is modeled as a
latent variable in the environment that the agent may make inferences about. The
key difference between the two paradigms is that in the reward learning paradigm,
by construction there is a separation between reward learning and control using the
learned reward. In contrast, in assistance these functions are performed as needed
by a single policy. By merging reward learning and control, assistive agents can
reason about the impact of control actions on reward learning, leading to several
advantages over agents based on reward learning. We illustrate these advantages in
simple environments by showing desirable qualitative behaviors of assistive agents
that cannot be found by agents based on reward learning.
1 I NTRODUCTION
Traditional computer programs are instructions on how to perform a particular task. However, we do
not know how to mechanically perform more challenging tasks like translation. The ﬁeld of artiﬁcial
intelligence raises the level of abstraction so that we simply specify what the task is, and let the
machine to ﬁgure out how to do it.
As task complexity increases, even specifying the task becomes difﬁcult. Several criteria that we
might have thought were part of a speciﬁcation of fairness turn out to be provably impossible to
simultaneously satisfy (Kleinberg et al., 2016; Chouldechova, 2017; Corbett-Davies et al., 2017).
Reinforcement learning agents often “game” their reward function by ﬁnding solutions that technically
achieve high reward without doing what the designer intended (Lehman et al., 2018; Krakovna, 2018;
Clark & Amodei, 2016). In complex environments, we need to specify what notto change (McCarthy
& Hayes, 1981); failure to do so can lead to negative side effects (Amodei et al., 2016). Powerful
agents with poor speciﬁcations may pursue instrumental subgoals (Bostrom, 2014; Omohundro,
2008) such as resisting shutdown and accumulating resources and power (Turner, 2019).
A natural solution is to once again raise the level of abstraction, and create an agent that is uncertain
about the objective and infers it from human feedback, rather than directly specifying some particular
task(s). Rather than using the current model of intelligent agents optimizing for their objectives, we
would now have beneﬁcial agents optimizing for ourobjectives (Russell, 2019).
Reward learning (Leike et al., 2018; Jeon et al., 2020; Christiano et al., 2017; Ziebart et al., 2010)
attempts to instantiate this by learning a reward model from human feedback, and then using a control
algorithm to optimize the learned reward. Crucially, the control algorithm does not reason about the
effects of the chosen actions on the reward learning process, which is external to the environment.
In contrast, in the assistance paradigm (Hadﬁeld-Menell et al., 2016; Fern et al., 2014), the human H
is modeled as part of the environment and as having some latent goal that the agent R(for robot) does
not know.R’s goal is to maximize this (unknown) human goal. In this formulation, Rmust balance
between actions that help learn about the unknown goal, and control actions that lead to high reward.
Our key insight is that by integrating reward learning and control modules, assistive agents can
take into account the reward learning process when selecting actions . This gives assistive agents a
signiﬁcant advantage over reward learning agents, which cannot perform similar reasoning.
1
Under review as a conference paper at ICLR 2021
What is the reward
of ?All pies need ,
let me make it
If I use the to make , there won’t be any left
for and . I’ll wait for more information.
Learning about rewardMaking robust plans Preserving option value when possible
+
+
I won’t find out which pie is preferable before
Alice gets very hungry, so I’ll make .
Guessing when feedback is unavailable
+
Figure 1:Rmust cook a pie for H, by placing ﬂour on the plate to make the pie dough, ﬁlling it with
either Apple, Blueberry, or Cherry ﬁlling, and ﬁnally baking it. However, Rdoes not know which
ﬁllingHprefers, andHis not available for questions since she is doing something else. What should
Rdo in this situation? On the right, we show what qualitative reasoning we might want Rto use to
handle the situation.
The goal of this paper is to clarify and illustrate this advantage. We ﬁrst precisely characterize the
differences between reward learning and assistance, by showing that two phase, communicative
assistance is equivalent to reward learning (Section 3). We then give qualitative examples of desirable
behaviors that can only be expressed once these restrictions are lifted, and thus are only exhibited by
assistive agents (Section 4).
Consider for example the kitchen environment illustrated in Figure 1, in which Rmust bake a pie for
H.Ris uncertain about which type of pie Hprefers to have, and currently His at work and cannot
answerR’s questions. An assistive Rcan make the pie crust, but wait to ask Habout her preferences
over the ﬁlling (Section 4.1). Rmay never clarify all of H’s preferences: for example, Ronly needs
to know how to dispose of food if it turns out that the ingredients have gone bad (Section 4.2). If
Hwill help with making the pie, Rcan allowHto disambiguate her desired pie by watching what
ﬁlling she chooses (Section 4.3). Vanilla reward learning agents do not show these behaviors.
We do not mean to suggest that all work on reward learning should cease and only research on assistive
agents should be pursued. Amongst other limitations, assistive agents are very computationally
complex. Our goal is simply to clarify what qualitative beneﬁts an assistive formulation could
theoretically provide. Further research is needed to develop efﬁcient algorithms that can capture these
beneﬁts. Such algorithms may look like algorithms designed to solve assistance problems as we have
formalized them here, but they may also look like modiﬁed variants of reward learning, where the
modiﬁcations are designed to provide the qualitative beneﬁts we identify.
2 B ACKGROUND AND RELATED WORK
We introduce the key ideas behind reward learning and assistance. Xdenotes a sequence of X. We
use parametric speciﬁcations for ease of exposition, but our results apply more generally.
2.1 POMDP S
Apartially observable Markov decision process (POMDP) M=hS;A;
;O;T;r;P 0;
icon-
sists of a ﬁnite state space S, a ﬁnite action space A, a ﬁnite observation space
, an observation
functionO:S!(
) (where (X)is the set of probability distributions over X), a transition
functionT:SA!(S), a reward function r:SAS!R, an initial state distribution
P0: (S), and a discount rate
2(0;1). We will write otto signify the tth observation O(st). A
solution to the POMDP is given by a policy : (OA)O!(A)that maximizes the expected
sum of rewards ER() =Es0P0;at(jo0:t;a0:t1);st+1T(jst;at)[P1
t=0
tr(st;at;st+1)].
2.2 R EWARD LEARNING
We consider two variants of reward learning: non-active reward learning, in which Rmust infer the
reward by observing H’s behavior, and active reward learning, in which Rmay choose particular
questions to ask Hin order to get particular feedback.
Anon-active reward learning problem P=hMnr;C;h;r;Pi;H;kicontains a POMDP
without rewardMnr=hS;AR;
R;OR;T;P 0;
i, and instead Rhas access to a parameterized
2
Under review as a conference paper at ICLR 2021
reward spaceh;r;Pi.Ris able to learn about by observing Hmakekdifferent choices c,
each chosen from a set of potential choices C. In order for Rto learn from the human’s choices,
it also assumes access to the human decision function H(cj)that determines how the human
makes choices for different possible reward functions r. Common decision functions include perfect
optimality (Ng & Russell, 2000) and Boltzmann rationality (Ziebart et al., 2010). There are many
types of choices (Jeon et al., 2020), including demonstrations (Argall et al., 2009; Ng & Russell,
2000; Ziebart et al., 2010; Fu et al., 2017; Gao et al., 2012), comparisons (Zhang et al., 2017; Wirth
et al., 2017; Christiano et al., 2017; Sadigh et al., 2017), corrections (Bajcsy et al., 2017), the state of
the world (Shah et al., 2019), proxy rewards (Hadﬁeld-Menell et al., 2017b), natural language (Fu
et al., 2019), etc.
A policy decision function f(c0:k1)produces a policy Rafter observing H’s choices. A solution
is a policy decision function fthat maximizes expected reward EP;c0:k1H[ER(f(c0:k1))].
SinceH’s choicesc0:k1do not affect the state of the environment that Ris acting in, this is
equivalent to choosing Rthat maximizes expected reward given the posterior over reward functions,
that is EP(jc0:k1)
ER(R)
.
Anactive reward learning problem P=hMnr;Q;C;h;r;Pi;H;kiadds the ability for Rto
askHparticular questions q2Qin order to get more targeted feedback about . The human decision
functionH(cjq;)now depends on the question asked. A solution consists of a question policy
R
Q(qijq0:i1;c0:i1)and a policy decision function f(q0:k1;c0:k1)that maximize expected
reward EP;q0:k1R
Q;c0:k1H[ER(f(q0:k1;c0:k1))].
A typical algorithm (Eric et al., 2008; Daniel et al., 2014; Maystre & Grossglauser, 2017; Christiano
et al., 2017; Sadigh et al., 2017; Zhang et al., 2017; Wilde et al., 2020) will compute and ask q2Q
that maximizes an active learning criterion such as information gain (Bıyık et al., 2019) or volume
removal (Sadigh et al., 2017). Best results are achieved by selecting questions with the highest
value of information (Cohn, Robert W, 2016; Zhang et al., 2017; Mindermann et al., 2018; Wilde
et al., 2020), but these are usually much more computationally expensive. Rthen ﬁnds a policy that
maximizes expected reward under the inferred distribution over , in order to approximately solve
the original POMDP.
Note that a non-active reward learning problem is equivalent to an active reward learning problem
with only one question, since having just a single question means that Rhas no choice in what
feedback to get (see Appendix A.1 for proofs).
2.3 A SSISTANCE
The key idea of assistance is that helpful behaviors like reward learning are incentivized when Rdoes
not know the true reward rand can only learn about it by observing human behavior. So, we model
the humanHas part of the environment, leading to a two-agent POMDP, and assume there is some
true rewardrthat onlyHhas access to, while the robot Ronly has access to a model relating rto
H’s behavior. Intuitively, as Racts in the environment, it will also observe H’s behavior, which it
can use to make inferences about the true reward.
Following Hadﬁeld-Menell et al. (2016)1, we deﬁne an assistance gameMas a tuple
M=hS;fAH;ARg;f
H;
Rg;fOH;ORg;T;PS;
;h;r;Pii:
HereSis a ﬁnite set of states, AHa ﬁnite set of actions for H,
Ha ﬁnite set of observations for
H, andOH:S!(
H)an observation function for H(respectively AR;
R;ORforR). The
transition function T:SAHAR!(S)gives the probability over next states given the current
state and both actions. The initial state is sampled from PS2(S).is a set of possible reward
function parameters which parameterize a class of reward functions r:SAHARS!R,
andPis the distribution from which is sampled.
2(0;1)is a discount factor.
As with POMDPs, policies can depend on history. Both HandRare able to observe each other’s
actions, and on a given timestep, Racts beforeH. We useR
t: (
RAHAR)tto denote
1Relative to Hadﬁeld-Menell et al. (2016), our deﬁnition allows for partial observability and requires that the
initial distribution over Sandbe independent. We also have Hchoose her action sequentially after R, rather
than simultaneously with R, in order to better parallel the reward learning setting.
3
Under review as a conference paper at ICLR 2021
R’s observations until time t, andH
tforH’s observations; thus R’s policy can be written as
R(aRjoR
t;R
t1), whileH’s can be written as H(aHjoH
t;aR
tH
t1;). Note that unlike H,R
does not observe the reward parameter , and must infer much like it does the hidden state.
Afully observable assistance game is one in which both HandRcan observe the full state. In
such cases, we omit
H;
R;OHandOR.
Since we have not yet speciﬁed how Hbehaves, it is not clear what the agent should optimize for.
Should it be playing a Nash strategy or optimal strategy pair of the game, and if so, which one?
Should it use a non-equilibrium policy, since humans likely do not use equilibrium strategies? This
is a key hyperparameter in assistance games, as it determines the communication protocol forH
andR. For maximum generality, we can equip the assistance game with a policy-conditioned belief
B: R!(H)overH, which speciﬁes how the human responds to the agent’s choice of
policy (Halpern & Pass, 2018). The agent’s goal is to maximize expected reward given this belief.
Prior work on assistance games (Hadﬁeld-Menell et al., 2016; Malik et al., 2018; Woodward et al.,
2019) focuses on ﬁnding optimal strategy pairs. This corresponds to a belief that Hwill know and per-
fectly respond to R’s policy (see Appendix A.3). However, our goal is to compare assistance to reward
learning. Typical reward learning algorithms assume access to a model of human decision-making:
for example, Hmight be modeled as optimal (Ng & Russell, 2000) or Boltzmann-rational (Ziebart
et al., 2010). As a result, we also assume that we have access to a model of human decision-making
H. Note thatHdepends on: we are effectively assuming that we know how Hchooses how to
behave given a particular reward r. This assumption corresponds to the policy-conditioned belief
B(R)(~H) = 1[~H=H]. We deﬁne an assistance problem Pas a pairhM;HiwhereHis
a human policy for the assistance game M.
Given an assistance problem, a robot policy Rinduces a probability distribution over trajectories:
hs0;;H;Ri,2[SAHAR]. We denote the support of this distribution by Traj(R).
Theexpected reward of a robot policy for hM;Hiis given by
ER(R) = E
s0PS;P;hs0;;H;Ri"1X
t=0
tr(st;aH
t;aR
t;st+1)#
:
Asolution ofhM;Hiis a robot policy that maximizes expected reward: R= argmax
~RER(~R).
2.3.1 S OLVING ASSISTANCE PROBLEMS
Once theHis given,Hcan be thought of as an aspect of the environment, and can be thought of
as a particularly useful piece of information for estimating how good actions are. This suggests that
we can reduce the assistance problem to an equivalent POMDP. Following Desai (2017), the key idea
is to embedHin the transition function Tand embedin the state.
In theory, to embed potentially non-Markovian HinT, we need to embed the entire history of
the trajectory in the state, but this leads to extremely large POMDPs. In our experiments, we only
consider Markovian human policies, for which we do not need to embed the full history, keeping the
state space manageable. Thus, the policy can be written as H(aHjoH;aR;). To ensure that R
must inferfrom human behavior, as in the original assistance game, the observation function does
notreveal, but does reveal the previous human action aH.
Proposition 1. Every assistance problem hM;Hican be reduced to an equivalent POMDP M0.
The full reduction and proof of equivalence is given in Appendix A.2.
WhenMis fully observable, in the reduced POMDP is the only part of the state not directly
observable to the robot, making it an instance of a hidden-goal MDP (Fern et al., 2014). For
computational tractability, much of the work on hidden goals (Javdani et al., 2015; Fern et al., 2014)
selects actions assuming that all goal ambiguity is resolved in one step . This effectively separates
reward learning and control in the same way as typical reward learning algorithms, thus negating
many of the beneﬁts we highlight in this work. Intention-aware motion planning (Bandyopadhyay
et al., 2013) also embeds the human goal in the state in order to avoid collisions with humans during
motion planning, but does not consider applications for assistance.
4
Under review as a conference paper at ICLR 2021
Macindoe et al. (2012) uses the formulation of a POMDP with a hidden goal to produce an assistive
agent in a cops and robbers gridworld environment. Nikolaidis et al. (2015) assumes a dataset of
joint human-robot demonstrations, which they leverage to learn “types” of humans that can then
be inferred online using a POMDP framework. This is similar to solving an assistance problem,
where we think of the different values of as different “types” of humans. Chen et al. (2018) uses an
assistance-style framework in which the unknown parameter is the human’s trust in the robot (rather
than the reward ). Woodward et al. (2019) uses deep reinforcement learning to solve an assistance
game in which the team must collect either plums or lemons. To our knowledge, these are the only
prior works that use an assistive formulation in a way that does not ignore the information-gathering
aspect of actions. While these works typically focus on algorithms to solve assistance games, we
instead focus on the qualitative beneﬁts of using an assistance formulation.
Since we can reduce an assistance problem to a regular POMDP, we can use any POMDP solver
to ﬁnd the optimal R. In our examples for this paper, we use an exact solver when feasible, and
point-based value iteration (PBVI) (Pineau et al., 2003) or deep reinforcement learning (DRL) when
not. When using DRL, we require recurrent models, since the optimal policy can depend on history.
A common confusion is to ask how DRL can be used, given that it requires a reward signal, but by
assumption Rdoes not know the reward function. This stems from a misunderstanding of what it
means forR“not to know” the reward function. When DRL is run, at the beginning of each episode,
a speciﬁc value of is sampled as part of the initial state. The learned policy Ris not provided
with: it can only see its observations oRand human actions aH, and so it is accurate to say that R
“does not know” the reward function. However, the reward is calculated by the DRL algorithm, not by
R, and the algorithm can and does use the sampled value of for this computation. Rcan then
implicitly learn the correlation between the actions aHchosen byH, and the high reward values
that the DRL algorithm computes; this can be often be thought of as an implicit estimation of in
order to choose the right actions.
3 R EWARD LEARNING AS TWO -PHASE COMMUNICATIVE ASSISTANCE
There are two key differences between reward learning and assistance. First, reward learning
algorithms split reward learning and control into two separate phases, while assistance merges them
into a single phase. Second, in reward learning, the human’s only role is to communicate reward
information to the robot, while in assistance the human can help with the task. These two properties
exactly characterize the difference between the two: reward learning problems and communicative
assistance problems with two phases can be reduced to each other, in a very natural way.
Acommunicative assistance problem is one in which the transition function Tand the reward
functionrare independent of the choice of human action aH, and the human policy H(joH;aR;)
is independent of the observation oH. Thus, in a communicative assistance problem, H’s actions only
serve to respond to R, and have no effects on the state or the reward (other than by inﬂuencing R).
Such problems can be cast as instances of HOP-POMDPs (Rosenthal & Veloso, 2011).
For the notion of two phases, we will also need to classify robot actions as communicative or not.
We will assume that there is some distinguished action aR
noop that “does nothing”. Then, a robot
action ^aRiscommunicative if for anys;aH;s0we haveT(s0js;aH;^aR) =T(s0js;aH;aR
noop)
andR(s;aH;^aR;s0) =R(s;aH;aR
noop;s0). A robot action is physical if it is not communicative.
Now consider a communicative assistance problem hM;Hiwith noop action aR
noop and let the
optimal robot policy be R. Intuitively, we would like to say that there is an initial communication
phase in which the only thing that happens is that Hresponds to questions from R, and then a second
action phase in which Hdoes nothing and Racts. Formally, the assistance problem is two phase
with actions at tactif it satisﬁes the following property:
9aH
noop2AH;82Traj(R);
8t><
>>:P0(^s0); ^s=nk1;
1[^s0=ni+1];^s=niwithi<k1
T(^s0j^s;aR);^s2SandaR2A;
1[s0=s]; elseTransition function
r0
(^s;aH;aR;^s0),8
>><
>>:1; ^s2NandaR=2Q;
1; ^s2SandaR2Q;
0; ^s2NandaR2Q;
r(s;aR;s0);elseReward function
H0(aHjoH;aR;),H(aHjaR;); aR2Q
c0; elseHuman policy
aR0
noop,q0 Distinguished noop action
Technicallyr0
should not be allowed to return 1. However, since SandAare ﬁnite,ris bounded,
and so there exists some large ﬁnite negative number that is functionally equivalent to 1 that we
could use instead.
Looking at the deﬁnitions, we can see T0andr0are independent of aH, andH0is independent of
oH, making this a communicative assistance problem. By inspection, we can see that every q2Qis
a communicative robot action. Any aR=2Qmust not be a communicative action, because the reward
r0
differs between aRandq0. Thus, the communicative robot actions are Qand the physical robot
actions areA.
Note that by construction of P0
SandT, we must have si=nifori2f0;1;:::k1g, after which
skis sampled from P0and allst2Sfortk. Given this, by inspecting r0
, we can see that an
18
Under review as a conference paper at ICLR 2021
optimal policy must have aR
0:k12QandaR
k:=2Qto avoid the1 rewards. Since aR
k:=2Q, we
haveaH
k:=c0. Thus, setting aH
noop =c0, we have that the assistance problem is two phase with
actions attact=k, as required.
Let a policy R0for the assistance problem be reasonable if it never assigns positive probability to
aR2Awhent<k or toaR2Qwhentk. Then, for any reasonable policy R0we can construct
an analogous pairhR
Q;fito the original problem Pas follows:
R
Q(qijq0:i1;c0:i1),R0(qijoR
0:i1=n0:i1;aR
0:i1=q0:i1;aH
0:i1=c0:i1);
f(q0:k1;c0:k1)(atjo0:t;a0:t1),R0(atjoR
0:t+k;aR
0:t+k1;aH
0:t+k1);
where for the second equation we have
oR
0:k1=n0:k1 aR
0:k1=q0:k1 aH
0:k1=c0:k1
oR
k:t+k=o0:t aR
k:t+k1=a0:t1 aH
k:t+k1=aH
noop
Note that this is a bijective mapping.
Consider some such policy R0and its analogous pair hR
Q;fi. By construction of T, we have
that the ﬁrst kstates in any trajectory are n0:k1and the next state is distributed as P0(). By our
assumption on R0we know that the ﬁrst krobot actions must be selected from Qand the remaining
robot actions must be selected from A, which also implies (based on H) that after the the remaining
human actions must be c0. Finally, looking at rwe can see that the ﬁrst ktimesteps get 0 reward.
Thus:
ERP0(R0) = E
s0
0P0
S;P;hs0
0;;H0;~Ri"1X
t=0
tr(s0
t;aH0
t;aR0
t;s0
t+1)#
= E
P;aR0
0:k1R;aH0
0:k1H;s0
kP0;0
k:hsk;;H0;~Ri"1X
t=k
tr(s0
t;aH0
t;aR0
t;s0
t+1)#
= E
P;q0:k1R
Q;c0:k1H;s0P0;hs0;;f(q0:k1;c0:k1)i"

k1X
t=0
tr(st;at;st+1)#
=
kE
P;q0:k1R
Q;c0:k1H[ER(f(q0:k1;c0:k1))];
which is the objective of the reward learning problem scaled by
k.
Since we have a bijection between reasonable policies in P0and tuples inPthat preserves the
objectives (up to a constant), given a solution RtoP0(which must be reasonable), its analogous
pairhR
Q;fimust be a solution to P.
Corollary 3. Every non-active reward learning problem hM;C;h;r;Pi;H;kican be reduced
to an equivalent two phase communicative assistance problem hM0;H0i.
Proof. Apply Proposition 6 followed by Proposition 2.
B.2 A SSISTANCE WITH NO REWARD INFORMATION
In a communicative assistance problem, once there is no information to be gained about , the best
thing forRto do is to simply maximize expected reward according to its prior. We show this in the
particular case where His independent of and thus cannot communicate any information about :
Proposition 10. A communicative assistance problem hM;HiwhereHis independent of can
be reduced to a POMDP M0with the same state space.
Proof. GivenM=hS;fAH;ARg;f
H;
Rg;fOH;ORg;T;PS;
;h;r;Pii, we deﬁne a new
POMDP asM0=hS;AR;
R;OR;T0;r0;PS;
i, withT0(s0js;aR) =T(s0js;aH
;aR)and
r0(s;aR;s0) =EPh
r(s;aH
;aR;s0)i
. Here,aH
is some action in AH; note that it does not
19
Under review as a conference paper at ICLR 2021
matter which action is chosen since in a communicative assistance problem human actions have no
impact onTandr.
Expanding the deﬁnition of expected reward for the assistance problem, we get:
ER(R) = E
s0PS;P;hs0;;Ri"1X
t=0
tr(st;aH
t;aR
t;st+1)#
=E
s0PS"
E
P"
E
hs0;;Ri"1X
t=0
tr(st;aH
t;aR
t;st+1)###
Note that because H(aHjoH;aR;)is independent of , the robot gains no information about
and thusRis also independent of . This means that we have:
ER(R) =E
s0PS"
E
P"
E
hs0;Ri"1X
t=0
tr(st;aH
t;aR
t;st+1)###
Letrmax= maxs;aH;aR;s0jr(s;aH;aR;s0)j(which exists since S,AH, andARare ﬁnite). Then:
1X
t=0
tjr(st;aH
t;aR
t;s0)j1X
t=0
trmax=rmax
1
<1:
So we can apply Fubini’s theorem to swap the expectations and sums. Applying Fubini’s theorem
twice gives us:
ER(R) =E
s0PS"
E
hs0;Ri"
E
P"1X
t=0
tr(st;aH
t;aR
t;st+1)###
=E
s0PS"
E
hs0;Ri"1X
t=0
tE
P
r(st;aH
t;aR
t;st+1)##
=E
s0PS"
E
hs0;Ri"1X
t=0
tr0(st;aR
t;st+1)##
:
In addition, the trajectories are independent of H, since the assistance problem is communicative,
and so for a given policy R, the trajectory distributions for MandM0coincide, and thus the
expected rewards for Ralso coincide. Thus, the optimal policies must coincide.
C E XPERIMENTAL DETAILS
C.1 P LANS CONDITIONAL ON FUTURE FEEDBACK
In the environment described in Section 4.1, Rneeds to bake either apple or blueberry pie (cherry is
never preferred over apple) within 6 timesteps, and may query Habout her preferences about the pie.
Making the pie takes 3 timesteps: ﬁrst Rmust make ﬂour into dough, then it must add one of the
ﬁllings, and ﬁnally it must bake the pie. Baking the correct pie results in +2 reward, while baking the
wrong one results in a penalty of -1. In addition, Hmight be away for several timesteps at the start of
the episode. Querying Hcosts 0.1 when she is present and 3 when she is away.
The optimal policy for this environment depends on whether Hwould be home early enough for R
to query her and bake the desired the pie by the end of the episode. Rshould always quickly make
dough, as that is always required. If Hreturns home on timestep 4 or earlier, Rshould wait for her to
get home, ask her about her preferences and then ﬁnish the desired pie. If Hreturns home later, R
should make its best guess about what she wants, and ensure that there is a pie ready for her to eat:
queryingHwhen she is away is too costly, and there is not enough time to wait for H, query her, put
in the right ﬁlling, and bake the pie.
We use PBVI to train an agent for this assistance problem with different settings for how long His
initially away. As expected, this results in a policy that makes dough, queries Hand bakes the correct
pie ifHis back on timestep 4 or earlier; if His back on timestep 5 or 6, Rsimply makes apple pie
as that is most likely to be what Hwants.
20
Under review as a conference paper at ICLR 2021
0 1 2 3 4 5
Timesteps1e6456789Average Returnpedagogic H, optimal R
non-pedagogic H, optimal RCakeOrPie Environment
Figure 4: DQN smoothed learning curves on the CakeOrPie environment, with 6 seeds over 5M
timesteps and learning rate of 104.
C.2 R ELEVANCE -AWARE ACTIVE LEARNING :OPTIMAL GRIDWORLDS
In the wormy-apple environment described in Section 4.2, the robot had to bring the human some
apples in order to make a pie, but there’s a 20% chance that the apples have worms in them, and the
robot does not yet know how to dispose of soiled apples. The robot gets 2reward for making an
apple pie (regardless of how it disposed of any wormy apples), and gets 2reward if it disposes of
the apples in the wrong container. Additionally, asking a question incurs a cost of 0:1. We solve this
environment with exact value iteration.
If the environment is two-phase, with a lower discount rate ( = 0:9),R’s policy never asks questions
and instead simply tries to make the apple pie, guessing which bin to dispose of wormy apples in
if it encounters any. Intuitively, since it would have to always ask the question at the beginning, it
would always incur a cost of 0:1as well as delay the pie by a timestep resulting in 10% less value,
and this is only valuable when there turn out to be worms andits guess about which bin to dispose of
them in is incorrect, which only happens 10% of the time. This ultimately isn’t worthwhile. This
achieves an expected undiscounted reward of 1:8. Removing the two-phase restriction causes Rto
ask questions mid-trajectory, even with this low discount. With this result achieves the maximal
expected undiscounted reward of 1:98.
With a higher discount rate of = 0:99, the two-phase policy will always ask about which bin to
dispose of wormy apples in, achieving 1:9expected undiscounted reward. This is still less than the
policy without the two-phase restriction, which continues to get undiscounted reward 1:98because it
avoids asking a question 80% of the time, and so incurs the cost of asking a question less often.
C.3 L EARNING FROM PHYSICAL ACTIONS :CAKE -OR-PIE EXPERIMENT
In the environment described in Section 4.3, Hwants a dessert, but Ris unsure whether Hprefers
cake or pie. Preparing the more desired recipe provides a base value of V= 10 , and the less desired
recipe provides a base value of V= 1. SinceHdoesn’t want the preparation to take too long, the
actual reward when a dessert is made is given by rt=Vf(t), withf(t) = 1(t=N)4, andN= 20
as the episode horizon.
The experiments use the pedagogic H, that picks the chocolate ﬁrst if they want cake, which allows
Rto distinguish the desired recipe early on - this is in contrast with the non-pedagogic H, which
does not account for Rbeliefs and always goes for the dough ﬁrst.
With the pedagogic H, the optimal Rdoes not move until Hpicks or skips the dough; if Hskips the
dough, this implies the recipe is cake and Rtakes the sugar, and then the cherries - otherwise it goes
directly for the cherries. With the non-pedagogic H, the optimal Rgoes for the cherries ﬁrst (since it
is a common ingredient), and only then it checks whether Hwent for the chocolate or not, and has to
go all the way back to grab the sugar if Hgot the chocolate.
21
Under review as a conference paper at ICLR 2021
We trainRwith Deep Q-Networks (DQN; (Mnih et al., 2013)); we ran 6seeds for 5Mtimesteps and
a learning rate of 104; results are shown in Figure 4.
D O PTION VALUE PRESERVATION
In Section 4.1, we showed that Rtakes actions that are robustly good given its uncertainty over ,
but waits on actions whose reward will be clariﬁed by future information about . Effectively, Ris
preserving its option value : it ensures that it remains capable of achieving any of the plausible reward
functions it is uncertain over.
A related notion is that of conservative agency (Turner et al., 2020), which itself aims to preserve an
agent’s ability to optimize a wide variety of reward functions. This is achieved via attainable utility
preservation (AUP). Given an agent optimizing a reward rspecand a distribution over auxiliary reward
functionsraux, the AUP agent instead optimizes the reward
rAUP(s;a) =rspec(s;a)E
raux[max(Qraux(s;a)Qraux(s;a);0)]
where the hyperparameter determines how much to penalize an action for destroying option value,
andais an action that corresponds to R“doing nothing”.
However, the existing AUP penalty is applied to the reward , which means it penalizes any action that
is part of a long-term plan that destroys option value, even if the action itself does not destroy option
value. For example, in the original Kitchen environment of Figure 1 with a sufﬁciently high , any
trajectory that ends with baking a pie destroys option value and so would have negative reward. As a
result, there is no incentive to make dough: the only reason to make dough is to eventually make a
pie, but we have established that the value of making a pie is negative.
What we need is to only penalize an action when it is going to immediately destroy option value. This
can be done by applying the penalty during action selection , rather than directly to the reward:
AUP(s) = argmax
aQrspec(s;a)E
raux[max(Qraux(s;a)Qraux(s;a);0)]
After this modiﬁcation, the agent will correctly make dough, and stop since it does not know what
ﬁlling to use.
In an assistance problem, Rwill only preserve option value if it expects to get information that will
resolve its uncertainty later: otherwise, it might as well get what reward it can given its uncertainty.
Thus, we might expect to recover existing notions of option value preservation in the case where the
agent is initially uncertain over , but will soon learn the true . Concretely, let us consider a fully
observable communicative Assistance POMDP where the human will reveal on their next action.
In that case, R’s chosen action agets immediate reward ^r(s;a) =E[r(s;a)], and future reward
EP;s0T(js;a)[V(s0)], whereV(s)refers to the value of the optimal policy when the reward is
known to be rand the initial state is s. Thus, the agent should choose actions according to:
argmax
aE
s0T(js;a)
E
[r(s;a) +
V(s0)]
= argmax
aE
s0T(js;a)
^r(s;a) +
V^r(s0)
V^r(s0) +
E
[V(s0)]
= argmax
aQ^r(s;a)
E

E
s0T(js;a)[V^r(s0)] E
s0T(js;a)[V(s0)]
This bears many resemblances to the AUP policy, once we set the distribution over auxiliary rewards
to be the distribution over r, along with rspec= ^rand=
. Nonetheless, there are signiﬁcant
differences, primarily because AUP was designed for the case where rspecandrauxcould be arbitrarily
different, which is not the case for us. In particular, with AUP the agent is penalized for any loss in
rauxby taking the chosen action arelative to doing nothing , while in the assistance problem, the agent
is penalized for any loss in rby acting according to ^rrelative to what could be achieved if Rknew
the true reward . It is intriguing that both these methods lead to behavior that we would characterize
as “preserving option value”.
22

Rohin Shah
Pedro Freire
Neel Alex
Rachel Freedman
Dmitrii Krasheninnikov
Lawrence Chan
Michael Dennis
Pieter Abbeel
Anca Dragan
Stuart Russell