Extracting and Using Preference Information from the State
of the World
Rohin Shah
Electrical Engineering and Computer Sciences
University of California at Berkeley
Technical Report No. UCB/EECS-2020-210
http://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-210.html
December 17, 2020
Copyright © 2020, by the author(s).
All rights reserved.

Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission.
Extracting and Using Preference Information from the State of the World
by
Rohin Monish Shah
A dissertation submitted in partial satisfaction of the
requirements for the degree of
Doctor of Philosophy
in
Computer Science
in the
Graduate Division
of the
University of California, Berkeley
Committee in charge:
Professor Anca Dragan, Co-chair
Professor Stuart Russell, Co-chair
Professor Pieter Abbeel
Professor David Ahn
Fall 2020
Extracting and Using Preference Information from the State of the World
Copyright 2020
by
Rohin Monish Shah
1
Abstract
Extracting and Using Preference Information from the State of the World
by
Rohin Monish Shah
Doctor of Philosophy in Computer Science
University of California, Berkeley
Professor Anca Dragan, Co-chair
Professor Stuart Russell, Co-chair
Typically when learning about what people want and don't want, we look to human action as
evidence: what reward they specify, how they perform a task, or what preferences they express
can all provide useful information about what an agent should do. This is essential in order
to build AI systems that do what we intend them to do. However, existing methods require a
lot of expensive human feedback in order to learn even simple tasks. This dissertation argues
that there is an additional source of information that is rather helpful: the state of the world.
The key insight of this dissertation is that when a robot is deployed in an envi-
ronment that humans have been acting in, the state of the environment is already
optimized for what humans want, and is thus informative about human preferences.
We formalize this setting by assuming that a human Hhas been acting in an environment for
some time, and a robot Robserves the nal state produced. From this nal state, Rmust
infer as much as possible about H's reward function. We analyze this problem formulation
theoretically and show that it is particularly well suited to inferring aspects of the state that
should notbe changed { exactly the aspects of the reward that His likely to forget to specify.
We develop an algorithm using dynamic programming for tabular environments, analogously
to value iteration, and demonstrate its behavior on several simple environments. To scale
to high-dimensional environments, we use function approximators judiciously to allow the
various parts of our algorithm to be trained without needing to enumerate all possible states.
Of course, there is no point in learning about H's reward function unless we use it to guide
R's decision-making. While we could have Rsimply optimize the inferred reward, this suers
from a \status quo bias": the inferred reward is likely to strongly prefer the observed state,
since by assumption it is already optimized for H's preferences. To get Rto make changes to
2
the environment, we will usually need to integrate the inferred reward with other sources
of preference information. In order to support such reward combination, we use a model
in whichRmust maximize an unknown reward function known only to H. Learning from
the state of the world arises as an instrumentally useful behavior in such a setting, and can
serve to form a prior belief over the reward function that can then be updated after further
interaction with H.
i
For humanity,
that we may navigate the challenging times ahead.
ii
Contents
Contents ii
List of Figures iv
List of Tables v
1 Introduction 1
1.1 Why is inaction a safe default? . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Learning from the state of the world . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Using the learned reward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Thesis overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Background 7
2.1 Sequential decision-making . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Reward misspecication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Minimizing side eects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.4 Learning from human feedback . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.5 Maximum Causal Entropy Inverse Reinforcement Learning . . . . . . . . . . 13
3 Formalism: Dening a state of the world problem 18
3.1 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2 Choice of problem parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4 Theory: What can we learn from the state of the world? 22
4.1 Formalism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2 Preserving the observed state . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.3 Features as a form of prior information . . . . . . . . . . . . . . . . . . . . . 30
4.4 Information from irreversibility . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5 An exact algorithm for tabular environments 36
5.1 MCMC sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5.2 Reward Learning by Simulating the Past . . . . . . . . . . . . . . . . . . . . 37
iii
5.3 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6 Function approximation for high-dimensional environments 43
6.1 Gradient as backwards-forwards consistency . . . . . . . . . . . . . . . . . . 44
6.2 Learning a latent MDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.3 Deep RLSP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
7 Evaluation: Correcting misspecied rewards and imitating skills 51
7.1 Algorithm details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.2 Conceptual environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.3 Skill learning with Deep RLSP . . . . . . . . . . . . . . . . . . . . . . . . . . 59
7.4 Investigating the prior distribution . . . . . . . . . . . . . . . . . . . . . . . 63
7.5 Robustness to H's planning horizon . . . . . . . . . . . . . . . . . . . . . . . 64
7.6 Con
icts between the inferred reward and H's desires . . . . . . . . . . . . . 65
8 Using assistance instead of reward learning to integrate information 68
8.1 Reward learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
8.2 Assistance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
8.3 Reward learning as two-phase communicative assistance . . . . . . . . . . . . 76
8.4 Qualitative improvements for general assistance . . . . . . . . . . . . . . . . 82
8.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
8.6 Modeling the state of the world with assistance . . . . . . . . . . . . . . . . 89
9 Conclusion 91
9.1 Avenues for improvement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
9.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
9.3 Closing thoughts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Bibliography 96
iv
List of Figures
1.1 An environment which should not be disturbed: the house of cards . . . . . . . 2
1.2 Learning preferences from the state of the world: the vase environment . . . . . 3
4.1 An MDP where short-term reward con
icts with long-term reward . . . . . . . . 27
4.2 A simple deterministic chain MDP . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.3 Fire extinguisher MDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
6.1 Learning to imitate a balanced Cheetah . . . . . . . . . . . . . . . . . . . . . . 43
7.1 Behavior of RLSP on a suite of gridworld environments . . . . . . . . . . . . . . 56
7.2 Policy recovered by Deep RLSP for the balanced Cheetah . . . . . . . . . . . . 62
7.3 Robustness of RLSP to Alice's planning horizon . . . . . . . . . . . . . . . . . . 64
7.4 Comparing Additive and Bayesian methods of combining reward information . . 66
8.1 Desired behaviors from an AI assistant . . . . . . . . . . . . . . . . . . . . . . . 69
8.2 The wormy-apples variant of the kitchen environment . . . . . . . . . . . . . . . 84
8.3 The CakeOrPie variant of the kitchen environment . . . . . . . . . . . . . . . . 86
8.4 Learning curves for DQN on the CakeOrPie environment . . . . . . . . . . . . . 87
8.5 The oce variant of the kitchen environment . . . . . . . . . . . . . . . . . . . 90
v
List of Tables
7.1 Performance of algorithms on the gridworld test suite . . . . . . . . . . . . . . . 55
7.2 Results for imitation of MuJoCo robot policies . . . . . . . . . . . . . . . . . . . 60
vi
Acknowledgments
My journey through grad school has been quite a roller coaster, and I am indebted to many
people for making it both enjoyable and successful. For even starting me on this path, I
owe my thanks to Ras Bodik. He encouraged me to consider research as an opportunity to
explore during undergrad, and later took me on as his PhD student to work on program
synthesis. While my dissertation has turned out to be on a completely dierent topic, I owe
my success in this eld to the research skills I learned working with Ras.
During those early PhD years, I was especially lucky to be surrounded by the Bodik
group: Mangpo Phothilimthana, Sarah Chasins, Julie Newcomb, Ali Koksal, Shaon Barman,
and Emina Torlak. While it is only now that I really see how little I knew, and how much I
could expect to improve over the course of my PhD, their advice was invaluable in helping
me come to terms with my lack of productivity during that rst year.
After this year, Ras transferred to the University of Washington, and I followed him a
year later. I am especially grateful to the UW PLSE lab for making me feel so welcome; Zach
Tatlock, Anna Kornfeld Simpson, and James Wilcox deserve special thanks.
It was around this time that I noticed the signicant progress being made in articial
intelligence, and the notable lack of research on how to ensure its safety. I made the dicult
decision to switch the focus of my PhD from program synthesis to AI safety, at the Center for
Human-Compatible AI (CHAI). Thanks in particular to Ajeya Cotra for several conversations
that fed into this decision.
At CHAI, I was especially lucky to work with three advisors: Anca Dragan, Stuart Russell
and Pieter Abbeel. While I'm not going to cover the wealth of wisdom they have imparted
to me, I want to highlight one thing they did that I am especially thankful for: they gave me
a large amount of freedom in deciding what I should spend my time on. During my rst year
at CHAI, I spent most of my time simply reading papers and getting up to speed in my new
eld of study; at the end of the year I started a newsletter that consumed over ten hours of
my time every week. In today's culture of \publish or perish", it feels quite unusual for me
to have had this aordance at all.
The greatest benet of CHAI was the people I got to work with. Thanks in particular to
Daniel Filan, Adam Gleave, Michael Dennis, and Dylan Hadeld-Menell for shaping many of
my views about AI safety. I would also like to thank Alex Turner, Andrew Critch, Jaime Fisac,
Lawrence Chan, Andreea Bobu, Smitha Milli, and Cody Wild for fascinating conversations
on the topic. And of course I am indebted to many collaborators, including Ben Cottier,
Cynthia Chen, David Lindner, Dmitrii Krasheninnikov, Jordan Alexander, Mark Ho, Micah
Carroll, Neel Alex, Noah Gundotra, Paul Knott, Pedro Freire, Rachel Freedman, Richard
Ngo, Sam Toyer, Scott Emmons, S oren Mindermann, Steven Wang and Sumith Kulal.
Of course, I don't just owe thanks to those who worked specically on research with me. I
am also indebted to the excellent administrative sta, who made university bureaucracy much,
much easier to navigate. Rosie Campbell, Martin Fukui, Lydia Raya and Jean Nguyen were
particularly helpful in this regard, but I must also thank Angie Abbatecola, Lena Lau-Stewart,
Shirley Salanio, Audrey Sillers, Susanne Kauer, Laura Greeneld, and Logan Baldini.
vii
This PhD would have been so much harder to do without the support of many friends
(beyond those I was working with). While it would be quite a challenge to list them all, I
want to specically name Andrew Huang, Anisha Sensa Mauze, Bill Zito, Caroline Jeanmaire,
Chigozie Nri, Daniel Ziegler, Davis Foote, Dmitriy Khripkov, Eli Marrone, Ellen Anderson,
Jen Chung, Jonathan Mustin, Joseph Kijewski, Nathan Mandi, Ozzie Gooen, Patrick Brinich-
Langlois, Richard Yannow, Sean Kelly, Sindy Li, Valmik Prabhu, and Veronica Boyce.
And last but certainly not least, I must thank my parents Lena and Monish, my brother
Nihal, and my partner Lynette; they have been remarkably supportive, even as my expected
graduation date slipped further and further into the future. I cannot say that they were
all thrilled at my choice to pursue a PhD, which makes me all the more grateful that I can
nevertheless count on them to support me anyway.
1
Chapter 1
Introduction
Traditional computer programs are instructions on how to perform a particular task: to
compute a factorial, we tell the machine to enumerate the integers from 1 to n, and multiply
them together. However, we do not know how to mechanically perform more challenging
tasks like translation. The eld of articial intelligence raises the level of abstraction so that
we simply show what the task is, and let the machine to gure out how to do it. For instance,
we present pairs of sentences which provide examples of translation, and let the machine
determine how to translate well.
Unfortunately, as we apply our techniques to increasingly complex tasks, even specifying
the task becomes dicult. When we specify reward functions for reinforcement learning
by hand, the resulting agents often \game" their reward function by nding solutions that
technically achieve high reward without doing what the designer intended. In a particularly
famous example, an agent trained to maximize its score in the boat racing game CoastRunners
got stuck in a loop collecting high-scoring speed boosts instead of winning the game (Clark
and Amodei, 2016). Both Lehman et al. (2018) and Krakovna (2018) collect several examples
of similar behavior from a variety of sources.
As the environment becomes more complex, it becomes important not just to specify what
should be done, but also what should not be changed (McCarthy and Hayes, 1981). As put
by Stuart Russell, \A system that is optimizing a function of nvariables, where the objective
depends on a subset of size k  0such that
max
s2SjjPt(s;)jjTVCt:
While the theorem considers deterministic initial conditions where we start at a partic-
ular states, the bound also applies to stochastic initial conditions, since they are convex
combinations of the deterministic cases.
Ergodic MDPs
Given an MDPM=hS;AH;T;R;PS;
iand a policy H, we say that Hinduces a Markov
chain inMgiven by:
P(s0js) =X
aH2AHH(aHjs)T(s0js;aH):
Intuitively, if we observe an agent with policy Hacting inM, the sequence of states
visited by the agent behaves like the Markov chain induced by HinM.
An MDPMisergodic if for all possible policies H, the Markov chain induced by Hin
Mis ergodic. For an ergodic MDP, the stationary distribution of states can never depend
on the initial state distribution PS, regardless of the choice of H. We write H
1to denote
the stationary distribution of the resulting Markov chain, which is the limit of the t-step
visitation distribution, dened recursively with H
0=PSand
H
t(s) =X
s02SP(sjs0)H
t1(s0):
Learning from an optimal human
In this chapter, we will assume that Hpursues her goals optimally , that is,His an optimal
policy given her true reward function R. When two or more actions are equally valuable,
we assume Hchooses uniformly at random between these actions. We can formalize this
using the Bellman equations (which are guaranteed to have a unique solution):
H(aHjs;)/ 1"
aH2argmax
aH02AHQ(s;aH0;)#
Q(s;aH;) = E
s0T(js;aH)
R(s;aH;s0) +
V(s;)
V(s;) = max
aH2AHQ(s;aH;)
CHAPTER 4. THEORY: WHAT CAN WE LEARN FROM THE STATE OF THE
WORLD? 25
We denote this policy by H
opt() (read as \the optimal policy for "), and thus its
stationary distribution is H
opt()
1 . Now, recall that our goal is to learn from the state of the
world. For clarity, we will denote the observed state as sobserved . We now want to compute
the posterior P(jsobserved ), which is given by Equation 3.2:
P(jsobserved )/P()P(sobservedj) =P()H
opt()
1 (sobserved ):
In other words, the likelihood ratio is given by H
opt()
1 (sobserved ), and so any information that
we can learn comes from the variation of this quantity with . For this reason, H
opt()
1 (sobserved )
will be the primary object of study in this chapter.
4.2 Preserving the observed state
We argued in Chapter 1 that upon observing some optimized state, our rst instinct would
be to \do nothing" and leave the state as is. There is a fully general argument for this point:
if all we know is that Hoptimized the environment to lead to state sobserved , then clearly H
expects that sobserved is a high-value state, and so perhaps we should try not to change it.
Let us suppose there is reward function that rewards being in the state sobserved , and never
rewards any other state. Intuitively, the \goal" of this reward function would be to visit
sobserved as often as possible. As long as
is suciently large, it will care primarily about
how much we visit sobserved in the stationary distribution. So, the optimal policy for this
reward is the policy that maximizes the visitation of sobserved at the stationary distribution.
We formalize this below:
Proposition 3. Suppose there is some preserve such thatRpreserve (s;a;s0) = 1[s=sobserved ].
Then there exists some
0<1such that if
 >
 0thenpreserve maximizesH
opt()
1 (sobserved ).
Proof Sketch. If there is some other that achieves higher likelihood than preserve , it must visit
sobserved more often in the limit as t!1 . ThenH
opt() must achieve higher reward according
topreserve thanH
opt(preserve ) in the limit as t!1 , despite the fact that H
opt(preserve ) is
optimal for preserve . The only way this can hold is if H
opt(preserve ) visitedsobserved early at
the cost of failing to visit sobserved later. This allows us to bound
.
Proof. Note that for the reward Rpreserve , the expected reward is simply the sum of the
discounted probabilities of being in sobserved , and soERpreserve (H) =1P
t=0
tH
t(sobserved ).
Consider two arbitrary policies H
1andH
2. Assume that they satisfy the following two
conditions:
1.H
1has lower likelihood: H
11(sobserved )<H
21(sobserved ).
2.H
1has higher expected reward: ERpreserve (H
1)ERpreserve (H
2).
CHAPTER 4. THEORY: WHAT CAN WE LEARN FROM THE STATE OF THE
WORLD? 26
Denepto be the lower of the two stationary distribution probabilities, and to be half
the dierence, so that we have:
H
11(sobserved ) =p
H
21(sobserved ) =p+ 2
By the application of the convergence theorem (Proposition 2), we can nd some time T
such that for any tT:
1.H
1
t(sobserved )p+
2
2.H
2
t(sobserved )p+3
2
Putting these together, we get:
8tT:H
2
t(sobserved )H
1
t(sobserved ):
We now use the inequality from the second condition and simplify:
ERpreserve (H
2)ERpreserve (H
1)0
1X
t=0
t(H
2
t(sobserved )H
1
t(sobserved ))0
"T1X
t=0
t(H
2
t(sobserved )H
1
t(sobserved ))#
+
"1X
t=T
t(H
2
t(sobserved )H
1
t(sobserved ))#
0
For the rst terms where t
 -bound (H
1;H
2), then it is not possible for H
1;H
2to satisfy the two
conditions above.
Then, we can dene
0,maxH
1;H
2
-bound (H
1;H
2). Note that every
-bound is less
than 1. SinceSandAHare nite, so too is the space of deterministic policies, and so we
also have
0<1, as required.
Now suppose
 >
 0, and consider an arbitrary policy H. For the pair H
opt(preserve );H,
both conditions cannot be satised. However, since H
opt(preserve ) maximizes ERpreserve , the
second condition must be satised. Thus, the rst condition cannot be satised. Therefore
H
opt(preserve ) maximizes H
opt()
1 (sobserved ).
The above theorem only applies when
is suciently large, eectively because only then
are we suciently condent that Hcared enough about sobserved that we can infer that it
must be a high-value state. However, even when
is small, we would nd a policy that
prioritizes reaching sobserved when possible. Is it possible that despite its focus on the near
term, such a policy would nonetheless maximize the probability of sobserved in the stationary
distribution? It turns out the answer is no: the agent may sacrice the long-term reward
from the stationary distribution in order to get a more enticing short-term reward.
A B
CD
a10:5
0:5a2
a1;a2a1;a2 a1;a2
Figure 4.1: An MDP in which a
greedy goal-seeking agent will sac-
rice long-term reward in pursuit of
short-term reward. The blue Ais the
important state to analyze, and the
redBis the observed state sobserved .Consider the MDP in Figure 4.1, in which the
observed state is sobserved =B. Every time the agent
takes an action, there is an = 0:04 chance that the
next state is chosen uniformly at random from all states.
(This is necessary in order to ensure that the MDP is
ergodic, as we have assumed in this chapter.)
In this MDP, the only relevant decision for a policy
is whether to take action a1ora2in stateA, which we
will refer to as 1and2respectively. The stationary
distribution of 1isfA: 0:01;B: 0:311;C: 0:37;D:
0:309g, while that of 2isfA: 0:01;B: 0:368;C:
0:25;D: 0:372g. The crucial feature is just that 1
visitsCmore while 2visitsBandDmore, as one
would expect looking at Figure 4.1.
We assume the reward space is parameterized such
thatRiassigns 1 to state iand 0 to all other states.
For this MDP, preserve =B. For this reward, we face
a choice: whether to take a1in the hopes of getting to
stateBimmediately, or to take a2to guarantee that
we eventually get to state B.
Intuitively, if
is suciently low, then it is better
to takea1, because the uncertain chance of reaching B
is better than being forced to wait a full timestep before being able to reach B. For example,
CHAPTER 4. THEORY: WHAT CAN WE LEARN FROM THE STATE OF THE
WORLD? 28
when
= 0:1, the value of 1in each state is about fA: 0:0497;B: 1:0106;C: 0:0013;D:
0:0982g, whereas the value of 2in each state is about fA: 0:0105;B: 1:0105;C: 0:0012;D:
0:0981g.1strictly dominates 2(where the dierence is most pronounced in A, since that
is where the two policies dier).
Meanwhile, the optimal policy for Dis clearly2, sincea2goes directly to Dand
preserves the ability to keep going to D, whereasa1does not go to Dand has some
chance of severely curtailing the agent's ability to go to D. Thus at
= 0:1, we have
H
opt(D)
1 (sobserved )> H
opt(preserve )
1 (sobserved ), andpreserve isnota maximizer of
1(sobserved ).
Thus, the requirement in Proposition 3 that
be suciently high cannot be completely
eliminated.
This counterexample relied on our ability to set up the dynamics to oer the agent a
\lottery" (action a1) that could severely curtail its ability to reach sobserved . What if we remove
the ability to set up such lotteries by requiring that the environment be deterministic?
One issue is that deterministic environments are typically not ergodic. However, we can
apply the same trick as in our counterexample above, where to every action we add an
probability that the new state is chosen uniformly at random. Specically, given an MDP
M, dene the -randomized version of Mto be the same as Mexcept that the transition
dynamics are modied to T0(s;aH;s0) = (1)T(s;a;s0) +
jSj.
Then, an MDPMis-deterministic if there exists an MDP M0with a deterministic
transition function Td, such thatMis the-randomized version of M0. We will sometimes
abuse notation and write Td(s;aH) to denote the unique state s0such thatTd(s;aH;s0) = 1.
As long as the reward does not depend on the next state, the optimal policy for an
-deterministic MDP can be found by solving an associated fully deterministic MDP:
Lemma 4. LetM=hS;AH;T;R;PS;
ibe an-deterministic MDP with deterministic
transitionsTd, where the reward does not depend on next state and is written as R(s;aH).
Then a policy His optimal forMi it is optimal for M0=hS;AH;Td;R;PS;
ei, where
the eective discount
e=
(1).
Proof. An optimal policy can be computed from the optimal Q-function, which can be
calculated by value iteration, in which we initialize V0(s) = 0, and then dene the recurrence:
Qt(s;aH) =R(s;aH) +X
s02S
T(s;aH;s0)Vt1(s0)
=
R(s;aH) +
(1)Vt1(Td(s;aH))
+
jSjX
s02SVt1(s0)
Vt(s) = max
aH2AHQt(s;aH):
Note that optimal policies for a Q-function are unchanged if we subtract a constant from
Qt. This allows us to drop the last term from Qt. Using
e=
(1), we have:
CHAPTER 4. THEORY: WHAT CAN WE LEARN FROM THE STATE OF THE
WORLD? 29
Qt(s;aH) =R(s;aH) +
(1)Vt1(Td(s;aH)):
This is equivalent to the Q-value backup for a fully deterministic MDP with dynamics Td
with eective discount rate
e.
In an-deterministic MDP, we we once again nd that the likelihood H
opt()
1 (sobserved ) is
highest for preserve :
Proposition 5. Suppose there is some preserve forMas in Proposition 3, and assume that
Mis-deterministic with 0<< 1. Thenpreserve maximizesH
opt()
1 (sobserved ).
Proof Sketch. In an-deterministic environment, H
opt(preserve ) is always traversing the short-
est path inTdtosobserved , which means that at every timestep tit will be maximizing the
average visitation,1
t+1Pt
t0=0H
opt()
t0(sobserved ), which in the limit converges to the stationary
distribution H
opt()
1 (sobserved ).
Proof. First, we show that H
opt(preserve ) always chooses an action athat is on a shortest
path inTdtosobserved . By Lemma 4, we know that H
opt(preserve ) is an optimal policy for the
Q-function computed by the backups:
Qt(s;aH) = 1[s=sobserved ] +
eVt1(Td(s;aH))
Vt(s) = max
aH2AHQt(s;aH):
This is equivalent to the Q-value backup for a fully deterministic MDP with dynamics Td
with eective discount rate
e. It is clear that the optimal policy in such an MDP is to follow
the shortest path to sobserved .
To elaborate, let L(s) be the length of the shortest path from stosobserved inTd(or1
if no such path exists), with L(sobserved ),0. In addition, dene L0to be the length of the
shortest (non-empty) path from sobserved to itself. Then, we can prove by induction using the
previous recurrence relations that:
Vt(s) =8
<
:
L(s)
e1
dtL(s)
L0eL0
e
1
L0et>L (s)
0 else:
It is easy to see that either all actions are equally good (the else case above), or the best
action is the one which progresses on a shortest path (so that in the next state L(s) has
decreased, or if we are in sobserved then we are following a shortest path back to sobserved ).
We have so far shown that H
opt(preserve ) always chooses an action aHthat is on a shortest
path inTdtosobserved . Since the only other eect of any action besides traversing paths in Td
is to teleport the agent randomly, which the agent cannot control, we only need to analyze
CHAPTER 4. THEORY: WHAT CAN WE LEARN FROM THE STATE OF THE
WORLD? 30
how paths inTdare traversed. Clearly, always traversing the shortest path to sobserved will
maximize the expected number of times that sobserved is visited, and so we have:
H
opt(preserve )2argmax
HtX
t0=0H
t0(sobserved ) for nite t
H
opt(preserve )2argmax
Hlim
t!11
t+ 1tX
t0=0H
t0(sobserved )
H
opt(preserve )2argmax
Hlim
t!1H
t(sobserved ) Limit of average is limit of sequence
H
opt(preserve )2argmax
HH
1(sobserved )
preserve2argmax
H
opt()
1 (sobserved )
4.3 Features as a form of prior information
Overall, it seems quite likely that preserve will have a high likelihood ratio (if it exists). We have
derived two distinct sucient conditions: rst, when His modeled as suciently farsighted
(that is, with suciently high
), and second, when the environment is -deterministic.
This is in some sense a disappointing result: it suggests that it is hard to learn anything
more sophisticated than \preserve the observed state", even though intuitively it seems like
we should be able to infer more from the state of the world. We view this as having similar
implications as the no-free-lunch theorem of machine learning: just as we need to assume
some prior simplicity of the environment in order for machine learning to be feasible at all,
we similarly need to assume some prior information about Rin order to make non-trivial
inferences about the reward that His optimizing.
We will use a particularly simple form of prior information: we will assume that we have
access to a feature function f:S!RFthat describes each state by a vector of real-valued
features of that state. We will use fi:S!Rto denote the ith feature function, so that
fi(s) = [f(s)]i. Just as we have visitation distributions for states, we can have feature counts:
FH
t=tX
t0=0X
s2SH
t0(s)f(s):
The rewardRis then assumed to be of the form R(s;aH;s0) =g(f(s)) for some
simple function g. Unless otherwise specied, we will use rewards that are linear over the
features, that is, R(s;aH;s0) =Tf(s). Note that for a constant c>0,RandRchave
the same optimal policy; as a result we may sometimes assume that is normalized.
CHAPTER 4. THEORY: WHAT CAN WE LEARN FROM THE STATE OF THE
WORLD? 31
Once we use features, we can formalize an important fact about learning from the state of
the world: if there was no way that Hcould have aected a particular feature, then we can't
infer anything about her preferences over that feature. Note that this is not specic to learning
from the state of the world: we could prove a similar theorem for inverse reinforcement
learning as well.
Proposition 6. Assume that for a specic feature fiand any timestep t, the feature count
(FH
t)iis independent of H. ThenH
opt()
1 (sobserved )is invariant to the value of i.
Proof Sketch. Since there is no policy that can aect the value of fi, changing the value of i
has no eect on the optimal policy, and so cannot have an eect on H
opt()
1 (sobserved ).
Proof. Consider some arbitrary , and let0be the same as , exceptihas been increased
by an arbitrary constant c(which may be negative). Then, for any policy H:
ER0(H) =1X
t=0
tX
s2SH
t(s)R0(s)
=1X
t=0
tX
s2S(0)TFH
t
=1X
t=0
tX
s2STFH
t+c(FH
t)i
=K+1X
t=0
tX
s2STFH
t whereKis independent of H
=K+ER(H):
In the fourth line above, we have used the fact that ( FH
t)iis independent of Hto extract
out a constant Kthat is also independent of H.
Thus, for any policy H, the expected reward of the policy under 0only changes by a
constant. This can never change the ordering over policies, and so H
opt() =H
opt(0); that
is, optimal policies are invariant to changes in i. Thus,H
opt()
1 (sobserved ) is also invariant to
changes ini.
4.4 Information from irreversibility
Recall our motivating example of an intact vase: since a vase that is broken can never be
repaired, the fact that it is still intact in the observed state is strong evidence that Hpreferred
that the vase stays intact. This is an instance of a general pattern: if you observe that some
irreversible action was nottaken, this is a strong signal that the consequences of that action
are particularly unwanted. If we have a feature that tracks whether the irreversible transition
CHAPTER 4. THEORY: WHAT CAN WE LEARN FROM THE STATE OF THE
WORLD? 32
has been taken, then presumably the reward weight of that feature should indicate that the
transition is not to be taken. We formalize this below:
Denition 2. Binary feature. A featurefiisbinary if it only takes on the values 0 or 1,
that is,8s;fi(s)2f0;1g.
Denition 3. Irreversible feature. A binary feature fiisirreversible inMif it satises the
following conditions:
1. Initially o:8s2Support(PS);fi(s) = 0 .
2. Can be turned on: 9H;t:H
t(fi)>0.
3. Cannot be turned o: 8s: [fi(s) = 1 =) 8a8s02Support(T(js;a));fi(s0) = 1] .
For example, the \broken vase" feature in the vase environment of Figure 1.2 is an
irreversible feature. Note that in an ergodic MDP, there cannot be irreversible features,
since there is always positive probability of eventually transitioning back to the initial state.
However, we can instead consider non-ergodic MDPs M, which can have irreversible features,
and then ask what we would infer about the reward in an -randomized version of M(which
is always ergodic, if >0).
A
B
Ca1;a2
a1;a2
a1;a2
Figure 4.2: An -deterministic chain
MDP in which the agent has no con-
trol over the irreversible blue feature.
Bis the observed state sobserved .Can we get a general result saying that if we observe
that an irreversible feature is not turned on, then
we can infer that the feature should not be turned
on? Unfortunately the answer is no. Consider the
-deterministic MDP in Figure 4.2, in which there is a
feature that indicates whether or not the state is blue
(that is, whether the state is Cor not). This is a binary,
irreversible feature in the deterministic transitions Td,
and we do observe that the agent is in a state where
the feature is not yet on. However, the agent has no
control over anything, and so by Proposition 6 nothing
can be inferred about the weight on this feature.
The core issue is that we can only make strong
inferences about irreversible events if we believe that
Hcontrolled whether or not the irreversible event hap-
pened. In the chain MDP of Figure 4.2, the irreversible
event is inevitable, and can only be undone by the
chance of randomly transitioning to a randomly chosen
state.
It should be noted that this is not an artifact of -randomization. If we did not have
the ergodic restriction, then we could still have an MDP in which the agent's rst action
randomly sends it into one of two halves of the state space. In one half the agent is forced to
turn on the irreversible feature f1, and in the other half it is forced to turn on the irreversible
CHAPTER 4. THEORY: WHAT CAN WE LEARN FROM THE STATE OF THE
WORLD? 33
Start Food
FireTV
Dousedacook0:9
0:1aTV
adouse
adouseadouse
anoopanoop
anoopanoop
anoop
Figure 4.3: Fire extinguisher MDP. At the start, the agent can choose whether to cook food
or watch TV. Watching TV always succeeds. Cooking usually succeeds in making food, but
also has a 10% chance of starting a re. At any point the agent may use the re extinguisher,
irreversibly covering the house in foam but extinguishing any res present. The agent may
always take a noop, and if it takes an action that is not applicable to the given state, that is
treated equivalently to a noop.
featuref2. The observed state sobserved will have exactly one of the two irreversible features
turned on, but again by Proposition 6 we cannot infer anything about either of the features.
What if we added an assumption that the irreversible transition was \controllable"? For
example, we could assume that in every state swherefi(s) = 0, there is at least one action
that guarantees that the next state s0also satises fi(s0) = 0. Could we then say that the
irreversible feature's weight can be made arbitrarily negative?
It turns out that even with this assumption, we cannot conclude that the irreversible
feature is arbitrarily dispreferred. While we do know that along the trajectory the feature
was never turned on, it is possible that had the environment randomness been dierent then
the optimal policy would have turned the feature on. For example, using a re extinguisher
is irreversible, and we usually observe that re extinguishers have not been used, but there
certainly are situations in which we would use re extinguishers, and so the reward weight
for using a re extinguisher cannot be arbitrarily negative.
Concretely, consider the re extinguisher MDP in Figure 4.3. For ergodicity, we need the
environment to be -randomized, but we ignore this dependence during analysis for the sake
of clarity. In this MDP, the agent starts out in the state Start, and can choose to watch
TV, or to cook, which has a 90% chance of producing Food, and a 10% chance of igniting
Fire. From any of these states, the agent may use the re extinguisher to Douse a re (if
present). We have four features: fFood,fTV,fFireandfDoused , each of which take on value 1
CHAPTER 4. THEORY: WHAT CAN WE LEARN FROM THE STATE OF THE
WORLD? 34
in the corresponding state and 0 everywhere else. Note that fDoused is a binary, controllable,
irreversible feature.
Suppose that we observe sobserved =Food . Intuitively, Hcooked food and was willing to
take the risk of igniting a re. Assuming that
is suciently large, the optimal policy for
the reward = [Food;TV;Fire;Doused ] = [1;0;20;0] is to cook food and then take noop
actions, and to douse the re if one is ignited. This policy leads to H
opt()
1 (sobserved ) = 0:9
(ignoring-randomness), which is the highest possible value (since the maximum probability
of entering the Food state is 0.9). However, Doused cannot be made arbitrarily negative: for
example, if Doused is set to -100, then it is no longer worth cooking. The expected cost of the
re outweighs the expected benet of the food, since we would no longer be willing to douse
the re if it did occur. As a result, the agent would watch TV instead of cooking (or execute
some other 0 reward policy), in which case H
1(sobserved ) = 0.
This type of reasoning is a feature, not a bug: the entire point of a re extinguisher is
to deal with accidental res, and so we want our agent to learn that we are not completely
averse to using re extinguishers, despite the fact that using a re extinguisher is irreversible
and we have never (yet) done it.
4.5 Discussion
In this chapter, we theoretically analyzed solutions to the problem of learning from the state
of the world. We rst analyzed the general argument that the inferred reward would tend to
prioritize preserving the observed state of the world. When a \preservation" reward function
exists, it often does seem to have high likelihood. We identied two dierent sucient
conditions for such a reward function to maximize the likelihood:
1.His suciently farsighted, that is,
is suciently large (Proposition 3), or
2. The dynamics of the MDP are -deterministic (Proposition 5).
These are not guaranteed: there are some cases like Figure 4.1 where with stochastic
dynamics and low
the preservation reward function does not maximize the likelihood.
Nonetheless, it seems likely that in most realistic cases, a preservation reward function would
tend to have high or even maximal likelihood.
We also analyzed another general argument from Chapter 1: if we observe that some
irreversible transition has not happened, then Hprobably did not want that irreversible
transition to happen. While this is certainly sometimes true, as in the vase example of
Figure 1.2, there are often realistic scenarios in which it is not. In particular, just because H
has not yet irreversibly used a re extinguisher, doesn't mean that she doesn't want it to be
used in the event of a re.
Further research could also consider how to best formalize the argument from eort: \if
Hput eort into a particular feature, then that is very informative about the reward weight
for that feature". For example, if the transition dynamics lead to a vase accumulating dust
CHAPTER 4. THEORY: WHAT CAN WE LEARN FROM THE STATE OF THE
WORLD? 35
over time, that Hmust wipe o in order to make it clean, and we then observe a clean vase
insobserved , that is informative about H's preferences over dusty vases.
Having claried what we can hope to do by learning from the state of the world, we now
turn to the design of algorithms that can eciently solve such problems. We will leverage the
assumption that the reward is linear in features, as identied in Section 4.3. We will then
use these algorithms to further explore conceptually what can be done by learning from the
state of the world, in Chapter 7.
36
Chapter 5
An exact algorithm for tabular
environments
Having discussed the inferences that could be made in theory, we now turn to the task
of creating an algorithm that can make these inferences in practice. In this chapter, we
will consider nite-horizon tabular environments, that is, in our state of the world problem
hM;s0;H;T;h;R;Pii, we will require that T6=1, and thatMis suciently small
that we can iterate over it, analogously to value iteration.
In Chapter 4 we saw that allowing for arbitrary reward functions often leads to degenerate
solutions. So, we assume that Mis equipped with a feature function fthat identies the
relevant features, and the the reward function is linear in features, so that R(s) =Tf(s).
Equation 3.1 species how to compute the likelihood of a particular reward R. We
reproduce it here:
P(s0j) =X
sT;s2;s12S
aH
T;aH
2;aH
12AHPS(sT)1Y
t=TH(aH
tjst;)T(st+1jst;aH
t): (5.1)
Our goal is to compute the posterior P(js0). However, typically will be drawn from
a continuous space, and the transition dynamics Tcan be complex and nonlinear, making
it very dicult if not impossible to compute an exact form for the posterior that works for
arbitrary. This remains true even if we do not require the distribution to be normalized.
We thus need to use approximate inference methods in order to estimate the posterior.
5.1 MCMC sampling
One standard way to address the computational challenges involved with the continuous and
high-dimensional nature of is to use MCMC sampling to sample from p(js0)/p(s0j)p().
We apply this sampling in a standard manner to derive an algorithm that we present in
Algorithm 1.
CHAPTER 5. AN EXACT ALGORITHM FOR TABULAR ENVIRONMENTS 37
Algorithm 1 MCMC sampling to estimate the state of the world posterior
Require: State of the world problem hM;s0;H;T;h;R;Pii, step size
1: random sample( P)
2:Compute the last-step occupancy measure p(s0j)
3:p P(s0j)P()
4:repeat
5:0 random sample(N(;))
6: Compute the last-step occupancy measure p(s0j0)
7:p0 p(s0j0)p(0)
8: ifrandom sample(Unif(0 ;1))min(1;p0
p)then
9: 0
10: end if
11: Appendto the list of samples
12:until have generated the desired number of samples
While this Algorithm 1 gives us an estimate of the full posterior distribution, even for
relatively simple environments it can take quite a long time to converge, and so we seek a
better solution.
5.2 Reward Learning by Simulating the Past
To obtain a more ecient algorithm, we take inspiration from the research literature on
inverse reinforcement learning (IRL). The problem formulation in IRL is similar to ours: the
primary dierence is that Rcan observe a set of trajectories figrather than a single state
s0. If we apply MCMC sampling to IRL as we did in the previous section, then we recover
the Bayesian IRL algorithm (Ramachandran and Amir, 2007).
Instead of sampling with Bayesian IRL, it is common to use the Maximum Causal Entropy
IRL framework (MCEIRL) (Ziebart et al., 2010). As discussed in Section 2.5, MCEIRL
computes a point estimate of the true reward Rrather than the full posterior P(js0). It
also assumes that His the Boltzmann-rational policy, which can be computed using soft
value iteration.
We adopt both of these assumptions in our setting to derive a new algorithm analogous
to MCEIRL for the state of the world setting.
Deriving the gradient
Our goal is now to nd the maximum a posteriori estimate of the reward:
= argmaxlnp(js0) = argmaxlnp(s0j) + lnP(): (5.2)
CHAPTER 5. AN EXACT ALGORITHM FOR TABULAR ENVIRONMENTS 38
MCEIRL uses gradient ascent to solve the problem, since it is convex in the reward
parameters . We adopt the same approach. We assume that the gradient of the log prior
rlnP() is easy to compute, and so we focus on computing the gradient of the log likelihood
rlnp(s0j).
First, we note that the likelihood in Equation 3.1 can be rewritten in terms of the
likelihoods of individual trajectories:
p(s0j) =X
sT;:::s2;s12S
aH
T;:::aH
2;aH
12AHp(j); (5.3)
where=sTaH
T:::s1aH
1s0is the (hypothesized) trajectory that Htook in the
environment.
We can then rewrite our desired gradient in terms of the gradients of individual trajectories
as follows:
rlnp(s0j) =1
p(s0j)rp(s0j)
=1
p(s0j)X
sT:12S
aH
T:12AHrp(j)
=1
p(s0j)X
sT:12S
aH
T:12AHp(j)rlnp(j):
This has a nice interpretation: compute the MCEIRL gradients for each trajectory, and
then take their weighted sum, where each weight is the probability of the trajectory given
the evidence s0and current reward . In Section 2.5 we derived the exact gradient for a
trajectory under the MCEIRL assumption of Boltzmann-rationality. We now substitute it in
to get:
rlnp(s0j) =1
p(s0j)X
sT:12S
aH
T:12AH"
p(j)1X
t=Tg(st;aH
t;)#
; (5.4)
where we have used the denitions from Section 2.5:
g(st;aH
t;),f(st) +E
s0
t+1
Ft+1(s0
t+1;)
Ft(st;)
Ft(st;),f(st) + E
aH0
t:T1;s0
t+1:T"TX
t0=t+1f(s0
t0)#
:
CHAPTER 5. AN EXACT ALGORITHM FOR TABULAR ENVIRONMENTS 39
Computing the gradient with dynamic programming
The gradient in Equation 5.4 still involves a combinatorially large summation over all
possible past trajectories. In order to solve even simple environments, we need to avoid this
combinatorial explosion. Our approach is to use dynamic programming.
We rst express the gradient in Equation 5.4 asG0(s0;)
p(s0j), which can be done if we dene
Gt(st;),X
sT:t1;aH
T:t1"
p(T:t1;stj)t1X
t0=Tg(st0;aH
t0;)#
: (5.5)
Thus, to compute the gradient, we only need to compute G0(s0;) andp(s0j) eciently.
The latter is particularly easy, as it is just the probability of observing a state at a specic
timestep:
p(sTj) =PS(sT)
p(st+1j) =X
st2S
aH
t2AHp(stj)soft
t(aH
tjst;)T(st+1jst;aH
t):
Note thatsoft
tcan be computed using soft value iteration, as detailed in Section 2.5.
Before we tackle G, we rst derive a recursive rule for F:
F0(s0;) =f(s0)
Ft1(st1;) =f(st1) + E
aH0
t1:1;s0
t:0"0X
t0=tf(s0
t0)#
=f(st1) +E
aH0
t1;s0
t"
f(s0
t) + E
aH0
t:1;s0
t+1:0"0X
t0=t+1f(s0
t0)##
=f(st1) +E
aH0
t1;s0
t[Ft(s0
t;)]
=f(st1) +X
aH0
t1;s0
tsoft
t1(aH0
t1jst1;)T(s0
tjst1;aH0
t1)Ft(s0
t):
Note thatg(st;aH
t;) can easily be computed using its denition given F. We are now
ready to derive a recursive relation for G:
CHAPTER 5. AN EXACT ALGORITHM FOR TABULAR ENVIRONMENTS 40
Gt+1(st+1;)
=X
sT:t2S
aH
T:t2AH"
p(T:t;st+1j)tX
t0=Tg(st0;aH
t0;)#
=X
st2S
aH
t2AHX
sT:t12S
aH
T:t12AHT(st+1jst;aH
t)soft
t(aH
tjst;)p(T:t1;stj)
g(st;aH
t;) +t1X
t0=Tg(st0;aH
t0;)!
=X
st2S
aH
t2AH2
6664T(st+1jst;aH
t)soft
t(aH
tjst;)0
BBB@X
sT:t12S
aH
T:t12AHp(T:t1;stj)1
CCCAg(st;aH
t;)3
7775
+X
st2S
aH
t2AH2
6664T(st+1jst;aH
t)soft
t(aH
tjst;)X
sT:t12S
aH
T:t12AH
p(T:t1;stj)t1X
t0=Tg(st0;aH
t0;)!3
7775
=X
st2S
aH
t2AHT(st+1jst;aH
t)soft
t(aH
tjst;)
p(stj)g(st;aH
t;) +Gt(st;)
:
For the base case, note that
GT+1(sT+1) =X
sT;aH
T
p(sT;aH
T;sT+1)g(sT;aH
T;sT+1)
=X
sT;aH
TT(sT+1jsT;aH
T)soft
T(aH
TjsT)
p(sT)g(sT;aH
T;sT+1)
:
Comparing this to the recursive rule, for the base case we can set GT(sT) = 0.
Overall algorithm
Combining all of these ingredients together gives us our nal algorithm, presented in Algo-
rithm 2. Since we combine gradients from simulated past trajectories, we name the algorithm
Reward Learning by Simulating the Past (RLSP).
CHAPTER 5. AN EXACT ALGORITHM FOR TABULAR ENVIRONMENTS 41
Algorithm 2 Reward Learning by Simulating the Past.
Require: State of the world problem hM;s0;H;T;h;R;Pii, learning rate
1: random sample( P)
2:repeat
3:8sT2S:p(sTj) PS(sT)
4:8s02S:F0(s0;) f(s0)
5:8sT2S:GT(sT;) 0
6:soft soft value iteration( M;;T )
7: // Compute probabilities of states
8: fortin [T;:::;2;1]do
9:8st+12S:p(st+1j) P
st;aH
tp(stj)soft
t(aH
tjst;)T(st+1jst;aH
t)
10: end for
11: // Compute expected feature counts
12: fortin [1;2;:::;T]do
13:8st2S:Ft(st;) f(st) +P
aH0
t;s0
t+1soft
t(aH0
tjst;)T(s0
t+1jst;aH0
t)Ft+1(st+1;)
14: end for
15: // Compute G
16: fortin [T;:::;2;1]do
17:8st+12S:Gt+1(st+1;) 0
18: forst;aH
tinSAHdo
19: g(st;aH
t;) f(st) +E
s0
t+1
Ft+1(s0
t+1;)
Ft(st;)
20: target soft
t(aH
tjst;)
p(stj)g(st;aH
t;) +Gt(st;)
21:8st+12S:Gt+1(st+1;) Gt+1(st+1;) +T(st+1jst;aH
t)target
22: end for
23: end for
24: // Gradient ascent
25: +h
G0(s0;)
p(s0j)+rlnP()i
26:until convergence
CHAPTER 5. AN EXACT ALGORITHM FOR TABULAR ENVIRONMENTS 42
5.3 Requirements
The algorithms that we have presented here can infer information about human preferences,
but require fairly strong assumptions. In particular, in order to apply either RLSP or the
sampling algorithm, we need to have:
1.Small state and action spaces, so that we can enumerate all possible state-action pairs,
2. Perfect knowledge of the transition dynamics Tof the environments, and
3. A feature function fthat identies relevant features of interest.
In the next chapter, we relax these restrictions.
43
Chapter 6
Function approximation for
high-dimensional environments
Our goal now is to scale RLSP to realistic environments, where we cannot enumerate states,
the dynamics of the environment are not fully known in advance, and a feature function may
not be available. Consider for example the balanced cheetah in Figure 6.1. Just by looking
at this single balanced state, it should be possible to infer the \goal" of balancing. However,
we cannot apply the RLSP algorithm from the previous chapter for several reasons: the state
space is continuous, we cannot simply enumerate the transition matrix, and there is no clear
way to obtain a feature function.
Once again, we can take inspiration from corresponding research in inverse reinforcement
Figure 6.1: Suppose we observe a Cheetah balancing on its front leg (left). The state contains
joint velocities in addition to positions, which are fairly low, showing that the cheetah is
indeed balancing rather than, say, falling over. Intuitively, just by observing this balanced
state, we should be able to reason that the Cheetah is \trying" to balance { other plausible
goals, such as running forward, falling over, hopping on a single leg, and so on would not
have led to this kind of state. We would like to have an algorithm that can make these sorts
of inferences, despite the continuous state space, the lack of a feature function, and only
having access to a simulator (rather than a full transition matrix as in the previous chapter).
CHAPTER 6. FUNCTION APPROXIMATION FOR HIGH-DIMENSIONAL
ENVIRONMENTS 44
learning. Recent IRL algorithms (Fu et al., 2017; Finn et al., 2016) have shown how to
generalize tabular IRL algorithms to work in high-dimensional environments. The key idea
is to use a neural net function approximator to represent the reward function and learn a
policy in tandem with the reward function. We can then compute gradients by comparing
rollouts from the policy with the provided demonstrations, as in Equation 2.3.
This is a useful starting point for us, but does not solve everything. Our rst challenge is
that unlike the MCEIRL gradient of Equation 2.3, the RLSP gradient in Equation 5.4 cannot
be easily expressed as a comparison between policy rollouts and the observed state s0, and so
it is unclear how the gradient can be computed. Our key insight here is that to sample from
the distribution over past trajectories p(T:1js0;), we can start at the observed state s0
and simulate backwards in time . To enable this, we derive a gradient that is amenable to
estimation through backwards simulation, and learn an inverse policy and inverse dynamics
model using supervised learning to perform the backwards rollouts.
Our second challenge is that we cannot use a deep neural net as our reward function,
because this is too expressive: we would start learning the degenerate reward functions
identied in Section 4.2 that only assign reward to the observed state and ignore everything
else. We need a reward representation that can be meaningfully updated from a single
state observation, yet also can capture complex functions of the input in order to express
concepts like the \balanced Cheetah". Our solution here is to represent the reward as a linear
combination of features, where the features are learned through self-supervised representation
learning techniques in a pretraining step.
6.1 Gradient as backwards-forwards consistency
Here we tackle the rst challenge: expressing the RLSP gradient in a form that we can work
with when we only have access to a simulator for the environment. We start with the gradient
from Section 5.2:
rlnp(s0j) =1
p(s0j)X
sT:12S
aH
T:12AHp(j)rlnp(j)
=X
sT:12S
aH
T:12AHp(T:1js0;)rlnp(j)
= E
T:1p(js0;)[rlnp(j)]:
Approximating the expectation
For higher-dimensional environments, we must approximate the expectation over past trajec-
toriesp(T:1js0;). We would like to sample from the distribution, but it is not clear how
CHAPTER 6. FUNCTION APPROXIMATION FOR HIGH-DIMENSIONAL
ENVIRONMENTS 45
to sample the past conditioned on the present. Our key idea is that just as we can sample
the future by rolling out forwards in time, we should be able to sample the past by rolling out
backwards in time . Note that by the Markov property we have:
p(T:1js0;) =1Y
t=Tp(stjaH
t;st+1;:::s 0;)p(aH
tjst+1;aH
t+1;:::s 0;)
=1Y
t=Tp(stjaH
t;st+1;)p(aH
tjst+1;)
Thus, given the inverse policy (H)1
t(aH
tjst+1;) as well as the inverse dynamics
T1
t(stjaH
t;st+1;), we can sample a past trajectory T:1p(T:1js0;) by iteratively
applying (H)1andT1, starting from s0. Thus, our gradient can be written as
E
aH
1:T(H)1
s1:TT1[rlnp(j)]:
In order to learn ( H)1, we must rst have H. We assumed that the human was
Boltzmann-rational, which corresponds to the maximum entropy reinforcement learning
objective (Levine, 2018). We use Soft Actor-Critic (SAC; Haarnoja et al., 2018) to estimate
the policyH(ajs;), since it explicitly optimizes the maximum entropy RL objective.
Given the forward policy H(aHjs;) and simulatorT, we can construct a dataset of
sampled forward trajectories, and learn the inverse policy ( H)1and the inverse dynamics
T1using supervised learning. Given these, we can then sample T:1, allowing us to
approximate the expectation in the gradient. In general, both ( H)1andT1could be
stochastic and time-dependent, even if HandTare themselves deterministic and time-
independent.
Estimating the gradient for a trajectory
We now turn to the term within the expectation, which is the inverse reinforcement learning
gradient given a demonstration trajectory =sTaH
T:::s 0. Assuming that the user is
Boltzmann-rational, this is the MCEIRL gradient from Section 2.5:
rlnp(j) = 0X
t=Tf(st)!
FT(sT;)+1X
t=T
E
s0
t+1T(jst;aH
t)
Ft+1(s0
t+1;)
Ft+1(st+1;)!
(6.1)
(Recall thatFt(st;) is the expected feature count when continuing to act from state st
according to policy H, that is,Ft(st;), E
aH
t:1H
st:0TP0
t0=tf(st0)
.)
CHAPTER 6. FUNCTION APPROXIMATION FOR HIGH-DIMENSIONAL
ENVIRONMENTS 46
The rst term computes the feature counts of the demonstrated trajectory , while the
second term computes the feature counts obtained by the policy for the current reward
function(starting from the initial state sT). Sincer(s) =Tf(s), these terms increase the
reward of features present in the demonstration and decrease the reward of features under
the current policy. Thus, the gradient incentivizes consistency between the demonstration
and rollouts from the learned policy.
The last term is essentially a correction for the observed dynamics: if we see that st;aH
t
led tost+1, it corrects for the fact that we \could have" seen some other state s0
t+1. Since this
correction is zero in expectation (and expensive to compute), we drop it for our estimator.
Gradient estimator
After dropping the last term in Equation 6.1, expanding the denition of F, and substituting
in to our earlier gradient expression, our nal gradient estimator is:
rlnp(s0j) = E
aH
1:T(H)1
s1:TT12
66664 0X
t=Tf(st)!
 E
s0
T=sT
aH
T:1H
sT+1:0T" 0X
t=Tf(s0
t)!#3
77775(6.2)
Thus, given s0,,H,T, (H)1, andT1, computing the gradient consists of three
steps:
1.Simulate backwards from s0, and compute the feature counts of the resulting trajectories.
2. Simulate forwards from sTof these trajectories, and compute their feature counts.
3. Take the dierence between these two quantities.
This again incentivizes consistency, this time between the backwards and forwards trajec-
tories: the gradient leads to movement towards \what the human must have done" and away
from \what the human would do if they had this reward". The gradient becomes zero when
they are identical.
It may seem like the backwards and forwards trajectories should always be consistent
with each other, since ( H)1andT1are inverses of HandT. The key dierence is that
s0imposes constraints on the backwards trajectories, but not on the forward trajectories.
For example, suppose we observe s0in which a vase is unbroken, and our current hypothesis
is that the user wants to break the vase. When we simulate backwards, our trajectory will
contain an unbroken vase (even though the policy usually breaks vases), and when we simulate
forwards from sT,Hwill break the vase. The gradient would then reduce the reward for a
broken vase and increase the reward for an unbroken vase.
CHAPTER 6. FUNCTION APPROXIMATION FOR HIGH-DIMENSIONAL
ENVIRONMENTS 47
6.2 Learning a latent MDP
Our gradient still relies on a feature function f, with the reward parameterized as r(s) =
Tf(s). A natural way to remove this assumption would be to instead allow to parameterize
a neural network, which can then learn whatever features are relevant to the reward from the
RLSP gradient.
However, this approach will not work. The information contained in the RLSP gradient
is insucient to identify the appropriate features to construct: after all, it is derived from a
single state. If we were to learn a single unied reward using the same gradient, the resulting
reward would likely be degenerate: for example, it may simply identify the observed state,
that isR(s) = 1[s=s0]. Thus, we continue to assume that the reward is linear in features,
and instead learn the feature function using self-supervised representation learning. Such
techniques dene an auxiliary task and train the neural net to perform well on the task, in
the hopes that the neural net learns generally useful features that can then be used for the
task of interest. There are many potential auxiliary tasks that we could use:
Contrastive tasks. Contrastive approaches (Oord et al., 2018) give the model a discrim-
ination task: given a context , select which of a large set of targets is \most related" to
the context. The notion of relatedness can change: in image classication, two images are
\related" if they are dierent augmentations of the same base image (He et al., 2020; Chen
et al., 2020); while in reinforcement learning, two states could be related if they come from
the same trajectory (Lee et al., 2020; Stooke et al., 2020).
Reconstruction. In reconstructive approaches, the input state must be encoded into a
low-dimensional representation vector, in such a way that the original state can then be
decoded back from this vector (Kingma and Welling, 2014). In sequential environments, the
decoder can instead predict past or future states. This has been applied in reinforcement
learning (Stooke et al., 2020) as well as natural language processing (Radford et al., 2019).
Alternatively, tasks can be dened over the sequence as a whole, for example to predict a
missing word in a sentence (Devlin et al., 2019; Clark et al., 2020).
State space models. For partially observable environments recurrent state space models
(RSSMs; Karl et al., 2017; Doerr et al., 2018; Hafner et al., 2019, 2020; Buesing et al., 2018;
Kurutach et al., 2018) could be used instead: such methods aim to learn a latent MDP . They
do so by allowing the latent states to be computed by a recurrent model over the observations,
thus allowing the states to encode the history. For such a model, we can imagine that the
underlying POMDP has been converted into a latent MDP whose feature function fis the
identity. We can then compute RLSP gradients directly in this latent MDP, where Tandf
are known.
Goal-based exploration. By default, in the absence of a dataset of environment behavior,
the methods above would have to be applied to a dataset of environment interaction using
CHAPTER 6. FUNCTION APPROXIMATION FOR HIGH-DIMENSIONAL
ENVIRONMENTS 48
rollouts of a random policy. However, in many environments, a random policy will stay
within a small subset of the state space with high probability. To cover more of the state
space, we can extent any of the methods above with methods that increase the diversity
of the data on which the features are trained. These methods include unsupervised skill
learning (Achiam et al., 2018; Eysenbach et al., 2018; Nair et al., 2018; Sharma et al., 2020),
curiosity-driven exploration (Burda et al., 2018), and goal-conditioned policies with a diverse
goal space (Schaul et al., 2015; Andrychowicz et al., 2017).
Experimental choices. In our experiments, we use a variational autoencoder (VAE; Kingma
and Welling, 2014) to learn the feature function. The VAE encodes the states into a latent
feature representation, which we can use to learn a reward function if the environment is
fully observable, i.e., the states contain all relevant information. However, we expect that
better results could be obtained using other, more recent methods, if tuned well.
6.3 Deep RLSP
Putting these components together gives us the Deep RLSP algorithm (Algorithm 3). We rst
learn a feature function fusing self-supervised learning, and then train an inverse dynamics
modelT1, all using a dataset of environment interactions (such as random rollouts). Then,
we update using Equation 6.2, and continually train H, and (H)1alongsideto keep
them up to date. The full algorithm also adds a few bells and whistles that we describe
below.
Initial state distribution PS.The attentive reader may wonder why our gradient appears
to be independent of PS. This is actually not the case: while HandTare independent
ofPS, (H)1andT1dodepend on it. For example, if we observe Alice exiting the San
Francisco airport, the corresponding ( H)1should hypothesize dierent
ights if she started
from New York than if she started from Tokyo.
However, in order to actually produce such explanations, we must train ( H)1andT1
solely on trajectories of length Tstarting from sTPS. We instead train ( H)1andT1
on a variety of trajectory data, which loses the useful information in PS, but leads to several
benets. First, we can train the models on exactly the distributions that they will be used
on, allowing us to avoid failures due to distribution shift. Second, the horizon Tis no longer
critical: previously, Tencoded the separation in time between sTands0, and as a result
misspecication of Tcould cause bad results. Since we now only have information about s0,
it doesn't matter much what we set Tto, and as a result we can use it to set a curriculum
(discussed next). Finally, this allows Deep RLSP to be used in domains where an initial state
distribution is not available.
Since we are no longer able to use the information from PSthrough (H)1andT1, we
add in a heuristic to incorporate the information elsewhere. Specically, we weight every
CHAPTER 6. FUNCTION APPROXIMATION FOR HIGH-DIMENSIONAL
ENVIRONMENTS 49
Algorithm 3 TheDeep RLSP algorithm. The initial dataset of environment interactions D
can be constructed in many dierent ways: random rollouts, human play data, curiosity-driven
exploration, etc. The specic method will determine the quality of the learned features.
procedure Deep RLSP (fs0g,T)
D dataset of environment interactions
Initializef;H;(H)1;T1;randomly.
f SelfSupervisedLearning( D).Train encoder and decoder for latent MDP
T1 SupervisedLearning( D) .Train inverse dynamics
T 1 .Start horizon at 1
foriin [1::num epochs] do
H SAC() .Train policy
fg Rollout(H;T) .Collect dataset by rolling out H
(H)1 SupervisedLearning( f;fg) .Train inverse policy
 +ComputeGrad(fs0g,H,T, (H)1,T1,T,f).Update
ifgradient magnitudes are suciently low then
T T+ 1 .Advance horizon
end if
end for
return,f
end procedure
procedure ComputeGrad (fs0g,H,T, (H)1,T1,T,f)
fbackwardg Rollout(fs0g;(H)1;T1;T) .Simulate backwards from s0
fbackward AverageFeatureCounts( f;fbackwardg).Compute backward feature counts
fsTg FinalStates(fbackwardg)
fforwardg Rollout(fsTg;H;T;T) .Simulate forwards from sT
fforward AverageFeatureCounts( f;fforwardg).Compute forward feature counts
returnfbackwardfforward
end procedure
backwards trajectory by the cosine similarity between the nal state sT, and a sample
^sTPS.
Curriculum. Since the horizon Tis no longer crucial, we can use it to provide a curriculum.
We initially calculate gradients with low values of T, to prevent compounding errors in our
learned models, and making it easier to enforce backwards-forwards consistency, and then
slowly grow T, making the problem harder. In practice, we found this crucial for performance:
intuitively, it is much easier to make short backwards and forwards trajectories consistent
than with longer trajectories; the latter would likely have much higher variance.
Multiple input states. If we get multiple independent s0as input, we average their
gradients.
CHAPTER 6. FUNCTION APPROXIMATION FOR HIGH-DIMENSIONAL
ENVIRONMENTS 50
Replay buer. We also maintain a replay buer that stores previously collected ( s;a;s0)
transitions. This buer persists across policy training steps. When training the policy , we
sample transitions from both the replay buer as well as from the simulator T.
Having now dened both the RLSP and Deep RLSP algorithms, we turn to demonstrating
and evaluating them on a variety of environments and tasks.
51
Chapter 7
Evaluation: Correcting misspecied
rewards and imitating skills
To evaluate the importance of learning from the state of the world, we would like to check
whether the inferred preferences enable Rto be more helpful to Hin a variety of contexts.
However, it is not very clear how exactly to do so. The inferred reward is very likely to assign
states0maximal reward, since by assumption, when Alice optimized Rshe ended up at s0.
If the robot then starts in state s0, if a no-op action is available (as it often is), the inferred
reward is likely to incentivize no-ops, which is not very interesting or helpful.
Ultimately, our hope is that the information learned from the state of the world can be
combined with other sources of preference information in order for Rto learn both what it
should do, and what it should not do. So, in Section 7.2, we create a suite of environments
with a true reward, R, a specied reward, Rspec, the initial state of the environment, sT,
and the state observed by R,s0. Here,Rspecis a stand-in for some information about what
Rshould do, and will ignore some aspect(s) of R.
To evaluate an algorithm in an environment, we use it to infer a reward Hfroms0, which
is then combined with the specied reward to get a nal reward nal=H+spec. (We later
consider another heuristic method for combining these two rewards in Section 7.6, as well as
a more principled approach in Chapter 8.) We inspect the inferred reward qualitatively and
report the expected amount of true reward obtained when planning with nal, as a fraction
of the expected true reward from the optimal policy.
Section 7.3 identies a set of environments in which inferred rewards cancapture what R
should do: robot locomotion. Intuitively, if a quadrupedal robot is in mid-jump, a \no-op"
action is not available: the state of the environment will change no matter what the Rdoes,
simply because of gravity. As a result, the inferred reward cannot simply lead to behavior
that stays in s0forever, and so the inferred reward could be sucient to learn good behavior.
Thus, in these environments we use each algorithm to infer a reward function from the
state of the world, given a state sampled from rollouts of a policy that is performing a
specic type of movement behavior. We then optimize a new policy using the inferred reward
function, and evaluate how well the new policy imitates the original skill.
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 52
Having done these basic tests of algorithm functionality, we turn to investigating partic-
ular details of the algorithm implementations, including the importance of priors over sT
(Section 7.4), the robustness to the planning horizon (Section 7.5), and alternative methods
for combining reward functions (Section 7.6).
7.1 Algorithm details
RLSP is almost entirely specied by Algorithm 2. In our experiments, we set the learning
rate to 0:1 and the temperature for soft value iteration to 1 :0.
Deep RLSP on the other hand has several implementation details beyond what is specied
in Algorithm 3. In this section we describe the hyperparameters and architecture choices for
all models used in Deep RLSP in our experiments. All models are implemented using the
TensorFlow framework.
Feature function. We use a variational autoencoder (VAE) (Kingma and Welling, 2014) to
learn the feature function f. The encoder and decoder consist of 3 feed-forward layers of size
512. The latent space has dimension 30. The model is trained for 100 epochs on 100 rollouts
of a random policy in the environment. During training we use a batch size of 500 and a
learning rate of 105. We use the standard VAE loss function, but weight the KL-divergence
term with a factor c= 0:001, which reduces the regularization and empirically improved
the reconstruction of the model signicantly in our experiments. We hypothesize that the
standard VAE regularizes too much in our setting, because the latent space has a higher
dimension than the input space, which is not the case in typical dimensionality reduction
settings.
Inverse dynamics model. Our inverse dynamics model T1is a feed-forward neural
network with 5 layers of size 1024 with ReLU activations. We train it on 1000 rollouts of a
random policy in the environment for 100 epochs, with a batch size of 500 and learning rate
of 105.
Note that the model predicts the previous observation given the current observation and
action; it does not use the feature representation. We found the model to perform better if it
predicts the residual ot1otgivenotandatinstead of directly predicting ot1.
We normalize all inputs to the model to have zero mean and unit variance. To increase
robustness, we also add zero-mean Gaussian noise with standard deviation 0 :001 to the inputs
and labels during training and clip the outputs of the model to the range of values observed
during training.
Policy. For learning the policy Hwe use the stable-baselines implementation of Soft Actor-
Critic (SAC) with its default parameters for the MuJoCo environments (Haarnoja et al.,
2018; Hill et al., 2018). Each policy update during Deep RLSP uses 104total timesteps
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 53
(except Hopper, where we use 2 104) and we evaluate the nal reward function using 2 106
timesteps (except Pendulum, where we use 64, which is the default of Hill et al. (2018)).
Inverse policy. Because the inverse policy ( H)1is not a deterministic function, we
represent it with a mixture density network , a feed-forward neural network that outputs a
mixture of Gaussian distributions (Bishop, 1994).
The network has 3 layers of size 512 with ReLU activations and outputs a mixture of 5
Gaussians with a xed diagonal covariance matrix 0 :05I.
During Deep RLSP, we maintain an experience replay that is initialized with random
rollouts. In every iteration all states encountered during the algorithm are added to the
experience replay.
To update the inverse policy we sample batches with batch size 500 from the experience
replay, apply the forward policy and the forward transition model on the states to label the
data. We then train the model with a learning rate of 104.
Feature learning dataset. By default, we use random rollouts to generate the initial
datasetDthat is used to train the features and the inverse model T1.
Additional hyperparameters. We run Deep RLSP with a learning rate of 0 :01, and use
200 forward and backward trajectories to estimate the gradients. Starting with T= 1 we
increment the horizon when the gradient norm drops below 2 :0 or after 10 steps, whichever
comes rst. We run the algorithm until T= 10.
Computational cost. It is important to note that Deep RLSP can be signicantly more
computationally costly than the baselines that we compare against. Imitation learning with
full demonstrations can already be quite computationally expensive. Deep RLSP takes
this much further by learning several distinct deep neural net models, simulating potential
demonstrations (which are likely much noisier than real demonstrations), and nally imitating
them.
7.2 Conceptual environments
In our rst evaluation, we design a suite of environments to test whether algorithms that
learn from the state of the world can extract various dierent types of information from the
state of the environment at deployment. In each environment, a candidate algorithm must
infer a reward function given only the MDP without reward MnR, the observed state
s0, and (depending on the experiment) the initial state sT. This is then combined with a
specied rewardRspec, to give a nal reward nal=H+specthat we then evaluate. The
hyperparameter is tune for all algorithms (including baselines) to give the best results.
It should be noted that these environments were designed to illustrate properties of learning
from the state of the world, and thus should not be considered an outside independent check
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 54
on the value of learning from the state of the world. Nonetheless, we nd the qualitative
behaviors displayed below to be good indications of the value of such an approach.
We rst describe our baselines, and then explain our suite of environments along with the
results.
Baselines
To our knowledge, there are no existing algorithms for state of the world problems, and so
we design some simple algorithms ourselves to serve as baselines.
Specied reward policy spec.This baseline corresponds to not doing any learning from
the state of the world. We don't use s0at all and just optimize the specied reward, that is,
nal=spec.
Policy that penalizes deviations deviation .One of the core intuitions behind learning
from the state of the world is that the state s0has already been optimized for human
preferences, and so random changes to s0are likely to be bad. Thus, one possible baseline
is to simply minimize changes to the observed state. We accomplish this by penalizing
deviations from the observed features f(s0), givingRnal(s) =T
specf(s) +jjf(s)f(s0)jj.
Note that for this baseline, there is no nal; the algorithm directly species a nal reward
Rnal, rather than rst specifying an inferred reward that is then added to Rspec.
Relative reachability policy reachability .Rather than just penalizing any deviations, we
could instead only penalize those deviations that are impactful . For this, we look to the
literature on low-impact AI systems for baselines. We use relative reachability (Krakovna
et al., 2018) as our baseline. This is an approach low-impact AI systems that considers a
change to be negative when it decreases coverage , relative to what would have happened had
the agent done nothing. Here, coverage is a measure of how easily states can be reached from
the current state.
We compare against the variant of relative reachability that uses undiscounted coverage
and a baseline policy where the agent takes no-op actions, as in the original paper. Relative
reachability requires known dynamics but not a handcoded featurization. A version of relative
reachability that does make use of handcoded features instead of operating directly on states
would behave similarly on our suite of environments. Like the deviation baseline, relative
reachability directly species a nal reward Rnal.
Average features policy features .One way to think about s0is that it is a sample from
the rollout of an optimal (or at least good) policy for R. As a result, it is likely that the
features ofs0are generally good, and so should be incentivized. Since the reward takes on the
formR(s) =Tf(s), this suggests that we should set H=f(s0)
jjf(s0)jj, where the normalization
is simply to put the reward on a common scale. We call this the AverageFeatures baseline.
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 55
Analysis
We compare RLSP and Deep RLSP to our baselines with the assumption of known sT,
because it makes it easier to analyze their properties. We consider the case of unknown sT
in Section 7.4.
For Deep RLSP, in these stylized gridworlds, self-supervised learning should not be
expected to learn the necessary features. For example, in the room with vase environment,
the two door features are just particular locations, with no distinguishing features in the
state that would allow self-supervised learning to identify these locations as important. So,
when evaluating Deep RLSP, we run Algorithm 3 without the feature learning and instead
use the pre-dened feature function fof the environments. We also use a Mixture Density
Network (MDN) (Bishop, 1994) for the inverse dynamics T1, in order to allow the model
to output probability distributions. Once we do this, Deep RLSP has the same behavior as
RLSP, and so we only report results with RLSP below.
We summarize the results in Table 7.1, and show the environments and trajectories in
Figure 7.1.
Table 7.1: Performance of algorithms on environments designed to test particular properties.
Side eect Env eect Implicit reward Desirable act Unseen eect
Room Toy train Apple collection Batteries Far away vase
Easy Hard
spec 7 7 7 3 7 7
deviation 3 7 7 7 3
reachability 3 3 7 7 3
features 3 7
RLSP 3 3 3 3 3 7
Side eects: Room with vase (Figure 7.1a) . The room tests whether the robot can avoid
breaking a vase as a side eect of going to the purple door. There are features for the number
of broken vases, standing on a carpet, and each door location. Rspechas weight 1 for the
purple door feature, and 0 for all other weights. As a result, it causes the agent to walk over
the base, breaking it. Radditionally has weight -1 for the broken vases feature.
Since Alice didn't walk over the vase, RLSP infers a negative reward on broken vases,
and a small positive reward on carpets (since paths to the top door usually involve carpets).
So,RLSP successfully avoids breaking the vase.
The penalties also achieve the desired behavior: deviation avoids breaking the vase since it
would change the \number of broken vases" feature, while relative reachability avoids breaking
the vase since doing so would result in all states with intact vases becoming unreachable.
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 56sTands0
 spectrajectories
 RLSP trajectories
(a)
 (b)
 (c)
 (d)
 (e)
Figure 7.1: Evaluation of RLSP on our environments. Silhouettes indicate the initial position
of an object or agent, while lled in versions indicate their positions after an agent has acted.
The rst row depicts the information given to RLSP. The second row shows the trajectory
taken by the robot when following the policy specthat is optimal for Rspec, whereHis
the reward inferred using RLSP. The third row shows the trajectory taken when following
the policyRLSP that is optimal for nal=H+spec. (a) Side eects: Room with vase
(b) Distinguishing environment eects: Toy train (c) Implicit reward: Apple collection (d)
Desirable side eect: Batteries (e) \Unseen" side eect: Room with far away vase.
features fails to infer that breaking vases is bad, though this is due to a quirk in the
feature encoding. In particular, the feature counts the number of broken vases, and so the
inferred reward Hhas a value of zero for this feature, eectively ignoring it. If we change
the featurization to instead count the number of unbroken vases, then features would likely
get the right behavior.
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 57
Distinguishing environment eects: Toy train (Figure 7.1b) . To test whether al-
gorithms can distinguish between eects caused by the agent and eects caused by the
environment, as suggested in Krakovna et al. (2018), we add a toy train that moves along a
predened track. The train breaks if the agent steps on it. We add a new feature indicating
whether the train is broken and new features for each possible train location. Once again,
Rspecjust has weight 1 on the purple door feature, and so the resulting policy ends up
breaking the train. Radditionally has weight -1 on broken vases and trains.
RLSP infers a negative reward on broken vases and broken trains, for the same reason
as in the previous environment. It also infers not to put any weight on any particular train
location, even though it changes frequently, because it doesn't help explain s0. As a result,
RLSP walks over a carpet, but not a vase or a train.
For the penalty-based algorithms, deviation immediately breaks the train in order to
prevent the train location features from changing. reachability on the other hand deduces that
breaking the train is irreversible, and so correctly follows the same trajectory as RLSP.
features fails to infer that trains should not be broken, due to the same quirk in feature
encoding as in the previous environment, and as a result breaks the train.
Implicit reward: Apple collection (Figure 7.1d) . This environment tests whether the
algorithms can learn tasks implicit in s0. There are three trees that grow apples, as well as a
basket for collecting apples, and the goal is for the robot to harvest apples. We have features
for the number of apples in baskets, the number of apples on trees, whether the robot is
carrying an apple, and each location that the agent could be in. s0has two apples in the
basket, while sThas none. The specied reward Rspecis zero: the robot must infer the task
from the observed state. The true task is to collect apples, and so Rhas weight 1 on the
number of apples in the basket, and no weight on any other feature.
specis arbitrary since every policy is optimal for the zero reward. A penalty-based
algorithm can never get positive reward, because Rspecis always zero and penalties can only
decrease reward. As a result, the optimal policy for both deviation andreachability is to do
nothing, which achieves the maximum of zero reward.
RLSP infers a positive reward on apples in baskets, a negative reward for apples on
trees, and a small positive reward for carrying apples. Despite the spurious weights on other
features,RLSP harvests apples as desired, achieving the maximum possible true reward.
features also correctly infers that it is good to place apples in the basket, but it also
rewards the agent for staying in the original location. As a result, it avoids picking apples
from the tree that is furthest away, and so it does not pick apples as eectively as RLSP.
Desirable side eect: Batteries (Figure 7.1c) . This environment tests whether the
algorithms can tell when a side eect is allowed. We take the toy train environment, remove
vases and carpets, and add batteries. The robot can pick up batteries and put them into the
(now unbreakable) toy train, but the batteries are never replenished. If the train runs for
10 timesteps without a new battery, it stops operating. There are features for the number
of batteries, whether the train is operational, each train location, and each door location.
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 58
There are two batteries in sTbut only one in s0. The true reward Rplaces weight 1 on
being at the purple door, as well as weight -1 on the train being out of power.
We consider two variants for the specie reward Rspec{ an \easy" case, where is identical
to the true reward R, and a \hard" case, where it only has weight 1 on being at the purple
door (and no weight on keeping the train operational).
Unsurprisingly, specsucceeds at the easy case (where Rspec=R), and fails on the hard
case by allowing the train to run out of power. Both deviation andreachability see the action
of putting a battery in the train as a side eect to be penalized, and so neither can solve the
hard case. In the easy case, they still penalize picking up the batteries, and so only solve the
easy case if the penalty weight is small.
RLSP sees that one battery is gone and that the train is operational, and infers that
Alice wants the train to be operational and doesn't want batteries (since a preference against
batteries and a preference for an operational train are nearly indistinguishable). So, it solves
both the easy and the hard case, with RLSP picking up the battery, then staying at the
purple door except to deliver the battery to the train.
features incorrectly infers that batteries should notbe used up, since the number of
batteries in the environment is positive and so we infer that batteries are good. It thus fails
to solve the hard case, and only solves the easy case if is suciently small. Part of the
issue here is that features is not invariant to the addition of a constant to any feature. We
could x this by instead setting features =f(s0)f(sT), treatingf(sT) as a baseline to
whichf(s0) should be compared, but this would make features reliant on knowledge of sT
(which we will not have in future sections) and would cause it to always fail the vase and toy
train environments (whereas currently it depends on how the features are constructed).
\Unseen" side eect: Room with far away vase (Figure 7.1e) . This environment
demonstrates a limitation of our algorithm: it cannot identify side eects that Alice would
never have triggered. In this room, the vase is nowhere close to the shortest path from the
Alice's original position to her goal, but is on the path to the robot's goal. The features,
specied reward and true reward are all the same as in the room with vase environment
(Figure 7.1a).
Since our baselines don't care about the trajectory the human takes, they all perform as
before:specwalks over the vase, deviation andreachability both avoid it, and features fails to
avoid it but only due to a quirk of feature construction.
RLSP infers a near zero weight on the broken vase feature, since it is not present on any
reasonable trajectory to the goal, and so breaks it when moving to the goal. Note that this
only applies when Alice is known to be at the bottom left corner at sT: if we have a uniform
prior oversT(considered in Section 7.4) then we do consider trajectories where vases are
broken.
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 59
7.3 Skill learning with Deep RLSP
The apples collection environment (Figure 7.1d) demonstrates that at least in some cases
it is possible to learn what to do from the state of the world, without needing a specied
rewardRspec. In the case of that environment, given our choice of feature function, there was
no possible reward function that would view s0as uniquely optimal, which forced RLSP to
infer a reward that would incentivize changes to the environment. In this section, we identify
another domain in which changes to the environment are necessary: robot locomotion. In
this domain, due to physical laws (particularly gravity), Rusually does not have the ability
to preserve the state as it is. As a result, any reward inferred from the state of the world
must take into account that the state can and will change, suggesting that we may be able to
infer whatRshould do, rather than just what it should not do.
To test learning from the state of the world in this domain, we use the MuJoCo physics
simulator (Todorov et al., 2012). We consider the Inverted Pendulum ,Half-Cheetah and
Hopper environments implemented in Open AI Gym (Brockman et al., 2016). Since these
environments are continuous and high-dimensional, only Deep RLSP can be evaluated in this
domain.
In the inverted pendulum environment, the pendulum falls very quickly in random rollouts,
andT1never learns what a balanced pendulum looks like. So, for this environment only,
when constructing the initial dataset of environment interactions D, we combine random
rollouts with rollouts from an expert policy that balances the pendulum.
Baselines
To our knowledge, this is the rst work to train policies using a single state as input. Due to
lack of alternatives, we compare against GAIL (Ho and Ermon, 2016) using the implementation
from the imitation library (Wang et al., 2020). GAIL is an imitation learning algorithm,
and thus requires transitions as inputs, rather than single states. For each state we provide to
Deep RLSP, we provide a transition ( s;a;s0) to GAIL. GAIL is thus given more information
than Deep RLSP.
Most of our previous baselines in Section 7.2 do not apply in this setting, either because
they do not scale to high-dimensional environments, or because they depend on the presence
ofRspec. In particular, the penalty-based deviation and relative reachability methods do not
make sense as baselines, because the idea is to pursue some already specied behavior given
byRspec, but avoid having side eects via the use of a penalty.
The only baseline that can still be applied is the AverageFeatures baseline, in which we
setH=f(s0)
jjf(s0)jj. Here, the function fmust still be learned through self-supervised learning,
but we can avoid learning ( H)1andT1entirely. This can be thought of as an ablation of
Deep RLSP, in which we ignore all temporal information.
In our experiments, we sometimes sample multiple states from rollouts of an expert policy,
instead of just a single state. In this case, for Deep RLSP, we simply average the gradients
for each of the states individually (which can be thought of as a lower-variance estimator of
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 60
Environment SAC # states Deep RLSP AverageFeatures Waypoints GAIL
Inverted
Pendulum10001 262 (246) 6 (2) N/A 1000 (0)
10 605 (242) 3 (1) 4 (1) 1000 (0)
50 258 (198) 6 (4) 3.7 (0.3) 1000 (0)
Cheetah
(forward)132361 4833 (2975) 6466 (3343) N/A -288 (55)
10 6299 (559) 6245 (2352) -10 (23) -296 (172)
50 7657 (177) 4504 (2970) -126 (38) -54 (295)
Cheetah
(backward)133611 5694 (2513) 12443 (645) N/A -335 (46)
10 8102 (624) 12829 (651) -80 (388) -283 (45)
50 7795 (551) 11616 (178) -509 (87) 2113 (1015)
Hopper
(terminate)32741 80 (21) 99 (45) N/A 991 (9)
10 168 (58) 159 (126) 58 (7) 813 (200)
50 130 (12) 65 (36) 14 (4) 501 (227)
Hopper
(penalty)33631 1964 (545) 2537 (363) N/A 990 (9)
10 564 (75) 3103 (64) 709 (133) 784 (229)
50 2130 (744) 2078 (581) 1612 (785) 508 (259)
Table 7.2: Average returns achieved by the policies learned through various methods, for
dierent numbers of input states. The states are sampled from a policy trained using SAC
on the true reward function; the return of that policy is given as a comparison. Besides the
SAC policy return, all values are averaged over 3 seeds and the standard error is given in
parentheses. We don't report Waypoints on 1 state as it is identical to AverageFeatures on 1
state.
the gradient), and we similarly use an average for the AverageFeatures baseline (which is
why we call it the Average Features baseline).
An alternative to averaging the features is to view each of the observed states sias a
potential waypoint of the expert policy, and reward Rfor being near any one of them. We
implement this Waypoints method as R(s) =maxif(si
0)
jjf(si
0)jjf(s). Note that when we only
have a single state as input, this is equivalent to AverageFeatures, and so we do not report
results for Waypoints with one state.
Solving the environments without access to the reward function
First we look at the typical target behavior in each environment: balancing the inverted
pendulum, and making the half-cheetah and the hopper move forwards. Additionally we
consider the goal of making the cheetah run backwards (that is, the negative of its usual
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 61
reward function). We aim to use Deep RLSP to learn these behaviors without having access
to the reward function .
We train a policy using soft actor critic (SAC) (Haarnoja et al., 2018) to optimize for the
true reward function, and sample either 1, 10 or 50 states from rollouts of this policy to use
as input. We then use Deep RLSP to infer a reward and policy. Ideally we would evaluate
this learned policy rather than reoptimizing the learned reward, since learned reward models
can often be gamed (Stiennon et al., 2020), but it would be too computationally expensive to
run the required number of SAC steps during each policy learning step. As a result, we run
SAC for many more iterations on the inferred reward function, and evaluate the resulting
policy on the true reward function (which Deep RLSP does not have access to).
Results are shown in Table 7.2. In Hopper, we noticed that videos of the policies learned
by Deep RLSP looked okay, but the quantitative evaluation said otherwise. It turns out that
the policies learned by Deep RLSP do jump, as we might want, but they often fall down,
terminating the episode; in contrast GAIL policies stand still or fall over slowly, leading to
later termination and explaining their better quantitative performance. We wanted to also
evaluate the policies without this termination bias, and so we evaluate the same policies in an
environment that does not terminate the episode, but provides a negative reward instead; in
this evaluation both Deep RLSP and AverageFeatures perform much better. We also provide
videos of the learned policies at https://sites.google.com/view/deep-rlsp , which show
that the policies learned by Deep RLSP do exhibit hopping behavior (though with a strong
tendency to fall down).
GAIL is only able to learn a truly good policy for the (very simple) inverted pendulum,
even though it gets states and actions as input. Deep RLSP on the other hand achieves
reasonable behavior (though clearly not expert behavior) in all of the environments, using
only a few states as input. Surprisingly, the AverageFeatures method also performs quite
well, even beating the full algorithm on some tasks, though failing quite badly on Pendulum.
It seems that the task of running forward or backward is very well specied by a single state,
since it can be inferred even without any information about the dynamics (except that which
is encoded in the features learned from the initial dataset).
Learning skills from a single state
We investigate to what extent Deep RLSP can learn other skills where the reward is not
clear. Evaluation on these tasks is much harder, because there is no ground truth reward.
Therefore we evaluate qualitatively how similar the policies learned by Deep RLSP are to the
original skill. Unlike the previous case, we do not reoptimize the learned reward and only
look at the policies learned by Deep RLSP.
We consider skills learned by running Dynamics-Aware Unsupervised Discovery of Skills
(DADS; Sharma et al., 2020). Since we are not interested in navigation, we remove the \x-y
prior" used to get directional skills in DADS. We run DADS on the half-cheetah environment
and select all skills that are not some form of running. This resulted in two skills: one
in which the cheetah is moving forward making big leaps ( \jumping" ) and one in which
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 62
Figure 7.2: We sample a few states from a policy performing a specic skill to provide as
input. Here, Deep RLSP learns to balance the cheetah on the front leg from a single state .
We provide videos of the original skills and learned policies at: https://sites.google.com/
view/deep-rlsp .
it is slowly moving forward on one leg ( \balancing" ). As before we roll out these policies
and sample individual states from the trajectories to provide as an input for Deep RLSP.
We then evaluate the policy learned by Deep RLSP. Since the best evaluation here is to
simply watch what the learned policy does, we provide videos of the learned policies at
https://sites.google.com/view/deep-rlsp .
The rst thing to notice is that relative to the ablations, only Deep RLSP is even close
to imitating the skill. None of the other policies resemble the original skills at all. While
AverageFeatures could perform well on simple tasks such as running, the full algorithm is
crucial to imitate more complex behavior.
Between Deep RLSP and GAIL the comparison is less clear. Deep RLSP can learn
the balancing skill fairly well from a single state, which we visualize in Figure 7.2 (though
we emphasize that the videos are much clearer). Like the original skill, the learned policy
balances on one leg and slowly moves forward by jumping, though with slightly more erratic
behavior. However, the learned policy sometimes drops back to its feet or falls over on its
back. We suspect this is an artifact of the short horizon ( T10) used for simulating the
past in our algorithm. A small horizon is necessary to avoid compounding errors in the
learned inverse dynamics model, but can cause the resulting behavior to be more unstable on
timescales greater than T. We see similar behavior when given 10 or 50 states. GAIL leads
to a good policy given a single transition, where the cheetah balances on its front leg and
head (rather than just the front leg), but does not move forward very much. However, with
10 or 50 transition, the policies learned by GAIL do not look at all like balancing.
However, the jumping behavior is harder to learn, especially from a single state. We
speculate that here a single state is less informative than the balancing state. In the balancing
state, the low joint velocities tell us that the cheetah is not performing a
ip, suggesting that
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 63
we had optimized for this specic balancing state. On the other hand, with the jumping
behavior, we only get a single state of the cheetah in the air with high velocity, which is likely
not sucient to determine what the jump looked like exactly. In line with this hypothesis, at
1 state Deep RLSP learns to erratically hop, at 10 states it executes a jump too vigorously
and falls over, and at 50 states it makes larger hops (but still sometimes falls over).
The GAIL policies for jumping are also reasonable, though in a dierent way that makes
it hard to compare. Using 1 or 10 transitions, the policy doesn't move very much, staying in
contact with the ground most of the time. However, at 50 transitions, it performs behaviors
that are noticeably forward hops, without falling over, leading to a policy that looks better
than that learned by Deep RLSP.
7.4 Investigating the prior distribution
In Section 7.2, we considered the setting where the robot knows sT, since it is easier to
understand and analyze what happens. However, typically we will not know sT, and will
instead have some prior over it. Here, we compare RLSP in two settings: perfect knowledge
ofsT(as in Section 7.2), and no knowledge of sT, which we represent by using a uniform
prior distribution over all states.
Side eects: Room with vase (Figure 7.1a) and toy train (Figure 7.1b) . In both room
with vase and toy train, RLSP learns a smaller negative reward on broken vases when using
a uniform prior. This is because RLSP considers many more feasible trajectories when using
a uniform prior, many of which do not give Alice a chance to break the vase, as in Room
with far away vase in Section 7.2. In room with vase, the small positive reward on carpets
changes to a near-zero negative reward on carpets. With known sT, RLSP overts to the
few consistent trajectories, which usually go over carpets, whereas with a uniform prior it
considers many more trajectories that often don't go over carpets, and so it correctly infers a
near-zero weight. In toy train, the negative reward on broken trains becomes slightly more
negative, while other features remain approximately the same. This may be because when
Alice starts out closer to the toy train, she has more of an opportunity to break it, compared
to the case where sTis known.
Implicit preference: Apple collection (Figure 7.1d) . Here, a uniform prior leads to a
smaller positive weight on the number of apples in baskets compared to the case with known
sT. Intuitively, this is because RLSP is considering cases where sTalready has one or two
apples in the basket, which implies that Alice has collected fewer apples and so must have
been less interested in them. States where the basket starts with three or more apples are
inconsistent with the observed s0and so do not have an eect on the gradient. Following the
inferred reward still leads to good apple harvesting behavior.
Desirable side eects: Batteries (Figure 7.1c) . With the uniform prior, we see the same
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 64
/uni00000014 /uni00000016 /uni00000014/uni00000013 /uni00000016/uni00000013 /uni00000014/uni00000013/uni00000013
/uni0000002b/uni00000052/uni00000055/uni0000004c/uni0000005d/uni00000052/uni00000051/uni00000013/uni00000011/uni00000019/uni00000013/uni00000011/uni0000001b/uni00000014/uni00000011/uni00000013/uni00000029/uni00000055/uni00000044/uni00000046/uni00000057/uni0000004c/uni00000052/uni00000051/uni00000003/uni00000052/uni00000049/uni00000003/uni00000050/uni00000044/uni0000005b/uni00000003/uni00000035/uni00000057/uni00000055/uni00000044/uni0000004c/uni00000051
/uni00000055/uni00000052/uni00000052/uni00000050
/uni00000045/uni00000044/uni00000057/uni00000057/uni00000048/uni00000055/uni0000004c/uni00000048/uni00000056
/uni00000044/uni00000053/uni00000053/uni0000004f/uni00000048/uni00000056
Figure 7.3: Reward achieved by RLSP, as a fraction of the expected reward of the optimal
policy, for dierent values of Alice's planning horizon T.
behavior as in Apple collection, where RLSP with a uniform prior learns a slightly smaller
negative reward on the batteries, since it considers states sTwhere the battery was already
gone. In addition, due to the particular setup the battery must have been given to the train
two timesteps prior, which means that in any state where the train started with very little
charge, it was allowed to die even though a battery could have been provided before, leading
to a near-zero positive weight on the train losing charge. Despite this, RLSP successfully
delivers the battery to the train in both easy and hard cases.
\Unseen" side eect: Room with far away vase (Figure 7.1e) . With a uniform prior,
we \see" the side eect: if Alice started at the purple door, then the shortest trajectory to
the black door would break a vase. As a result, RLSP successfully avoids the vase (whereas it
previously did not). Here, uncertainty over the initial state sTcan counterintuitively improve
the results, because it increases the diversity of trajectories considered, which prevents RLSP
from \overtting" to the few trajectories consistent with a known sTands0.
Overall, RLSP appears to be quite robust to the use of a uniform prior over sT, suggesting
that we do not need to be particularly careful in the design of that prior.
7.5 Robustness to H's planning horizon
We investigate how RLSP performs when assuming the wrong value of H's planning horizon
T. We vary the value of Tassumed by RLSP, and report the true return achieved by
RLSP obtained using the inferred reward and a xed horizon for the robot to act. For this
experiment, we used a uniform prior over sT, since with known sT, RLSP often detects that
the givensTands0are incompatible (when Tis misspecied). The results are presented in
Figure 7.3.
The performance worsens when RLSP assumes that Alice had a smaller planning horizon
than she actually had. Intuitively, if we assume that Alice has only taken one or two actions
ever, then even if we know the actions, they could have been in service of many goals, and
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 65
so we end up quite uncertain about Alice's reward. For the Apple collection environment
the underestimated horizon prevents Rfrom learning anything at all, since all very short
trajectories consistent with s0do not involve any apple collection.
When the assumed Tis larger than the true horizon, RLSP correctly infers things the robot
should notdo. Knowing that the vase was not broken for longer than Ttimesteps is more
evidence to suspect that Alice cared about not breaking the vase. However, overestimated T
leads to worse performance at inferring implicit preferences, as in the Apples environment.
If we assume Alice has only collected two apples in 100 timesteps, she must not have cared
about them much, since she could have collected many more. The batteries environment is
unusual { assuming that Alice has been acting for 100 timesteps, the only explanation for
the observed s0is that Alice waited until the 98th timestep to put the battery into the train.
This is not particularly consistent with any reward function, and performance degrades.
Overall,Tis an important parameter and needs to be set appropriately. However, even
whenTis misspecied, performance tends to degrade gracefully to what would have happened
if we optimizedRspecby itself, so RLSP does not hurt. In addition, if Tis larger than it
should be, then RLSP still tends to accurately infer parts of the reward that specify what
not to do.
While this evaluation showed that RLSP is reasonably robust to the choice of planning
horizonTand prior over sT, this may be specic to our gridworlds. In the real world, we
often make long term hierarchical plans, and if we don't observe the entire plan (corresponding
to a choice of T that is too small) it seems possible that we infer bad rewards, especially if
we have an uninformative prior over sT. We do not know whether this will be a problem,
and if so how bad it will be, and hope to investigate it in future work with more realistic
environments.
7.6 Con
icts between the inferred reward and H's
desires
In Section 7.2, we evaluated RLSP by combining the reward it infers with a specied reward
to get a nal reward nal=H+spec. The problem of combining Handspecis dicult,
since the two rewards incentivize dierent behaviors and will con
ict. The Additive method
above is a simple way of trading o between the two.
Both RLSP and the sampling algorithm of Appendix 5.1 can incorporate a prior over .
Another way to combine the two rewards is to condition the prior on specbefore running the
algorithms. In particular, we could replace our prior P(H) with a new prior P(Hjspec),
such as a Gaussian distribution centered at spec. When we use this prior, the reward returned
by RLSP can be used as the nal reward nal.
It might seem like this is a principled Bayesian method that allows us to combine the
two rewards. However, the con
ict between the two reward functions still exists. In this
formulation, it arises in the new prior P(Hjspec). Modeling this as a Gaussian centered at
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 66
Comparison of the methods for combining specandH
/uni00000013/uni00000011/uni00000013/uni00000018 /uni00000013/uni00000011/uni00000015 /uni00000013/uni00000011/uni00000018 /uni00000015 /uni00000018/uni00000013/uni00000011/uni00000019/uni00000013/uni00000011/uni0000001b/uni00000014/uni00000011/uni00000013/uni00000029/uni00000055/uni00000044/uni00000046/uni00000057/uni0000004c/uni00000052/uni00000051/uni00000003/uni00000052/uni00000049/uni00000003/uni00000050/uni00000044/uni0000005b/uni00000003/uni00000035/uni00000057/uni00000048/uni00000050/uni00000053/uni00000048/uni00000055/uni00000044/uni00000057/uni00000058/uni00000055/uni00000048/uni00000003/uni00000020/uni00000003/uni00000013
/uni00000024/uni00000047/uni00000047/uni0000004c/uni00000057/uni0000004c/uni00000059/uni00000048/uni0000000f/uni00000003/uni00000045/uni00000044/uni00000057/uni00000057/uni00000048/uni00000055/uni0000004c/uni00000048/uni00000056
/uni00000024/uni00000047/uni00000047/uni0000004c/uni00000057/uni0000004c/uni00000059/uni00000048/uni0000000f/uni00000003/uni00000055/uni00000052/uni00000052/uni00000050
/uni00000024/uni00000047/uni00000047/uni0000004c/uni00000057/uni0000004c/uni00000059/uni00000048/uni0000000f/uni00000003/uni00000057/uni00000055/uni00000044/uni0000004c/uni00000051
/uni00000025/uni00000044/uni0000005c/uni00000048/uni00000056/uni0000004c/uni00000044/uni00000051/uni0000000f/uni00000003/uni00000045/uni00000044/uni00000057/uni00000057/uni00000048/uni00000055/uni0000004c/uni00000048/uni00000056
/uni00000025/uni00000044/uni0000005c/uni00000048/uni00000056/uni0000004c/uni00000044/uni00000051/uni0000000f/uni00000003/uni00000055/uni00000052/uni00000052/uni00000050
/uni00000025/uni00000044/uni0000005c/uni00000048/uni00000056/uni0000004c/uni00000044/uni00000051/uni0000000f/uni00000003/uni00000057/uni00000055/uni00000044/uni0000004c/uni00000051
/uni00000013/uni00000011/uni00000013/uni00000018 /uni00000013/uni00000011/uni00000015 /uni00000013/uni00000011/uni00000018 /uni00000015 /uni00000018
/uni00000036/uni00000057/uni00000044/uni00000051/uni00000047/uni00000044/uni00000055/uni00000047/uni00000003/uni00000047/uni00000048/uni00000059/uni0000004c/uni00000044/uni00000057/uni0000004c/uni00000052/uni00000051/uni00000013/uni00000011/uni00000019/uni00000013/uni00000011/uni0000001b/uni00000014/uni00000011/uni00000013/uni00000057/uni00000048/uni00000050/uni00000053/uni00000048/uni00000055/uni00000044/uni00000057/uni00000058/uni00000055/uni00000048/uni00000003/uni00000020/uni00000003/uni00000013/uni00000011/uni00000014
/uni00000013/uni00000011/uni00000013/uni00000018 /uni00000013/uni00000011/uni00000015 /uni00000013/uni00000011/uni00000018 /uni00000015 /uni00000018/uni00000013/uni00000011/uni00000019/uni00000013/uni00000011/uni0000001b/uni00000014/uni00000011/uni00000013/uni00000057/uni00000048/uni00000050/uni00000053/uni00000048/uni00000055/uni00000044/uni00000057/uni00000058/uni00000055/uni00000048/uni00000003/uni00000020/uni00000003/uni00000014
Figure 7.4: Comparison of the Additive and Bayesian methods. We show how the percentage
of true reward obtained by RLSP varies as we change the tradeo between the inferred reward
RHand the specied reward Rspec. The zero temperature case corresponds to traditional
value iteration; this often leads to identical behavior and so the lines overlap. So, we also
show the results when planning with soft value iteration, varying the softmax temperature,
to introduce some noise into the policy. Overall, there is not much dierence between the
two methods. We did not include the Apples environment because Rspecis uniformly zero
and the Additive and Bayesian methods do exactly the same thing.
specsuggests that before knowing s0, it seems likely that His very similar to spec. However,
this is not true { Alice is probably providing the reward specto the robot so that it causes
some change to the state that she has optimized, and so it will be predictably dierent from
spec. On the other hand, we do need to put high probability on spec, since otherwise nal
will not incentivize any of the behaviors that specdid.
Nonetheless, this is another simple heuristic for how we might combine the two rewards,
that manages the tradeo between RspecandRH. We compared the Additive and Bayesian
methods by evaluating their robustness. We vary the parameter that controls the tradeo
and report the true reward obtained by RLSP, as a fraction of the expected true reward
under the optimal policy. For the Bayesian method, we vary the standard deviation of the
Gaussian prior over RHthat is centered at Rspec. For the Additive method, the natural
choice would be to vary ; however, in order to make the results more comparable, we instead
set= 1 and vary the standard deviation of the Gaussian prior used while inferring RH,
which is centered at zero instead of at Rspec. A larger standard deviation allows RHto
become larger in magnitude (since it is penalized less for deviating from the mean of zero
reward), which eectively corresponds to a smaller .
While we typically create RLSP using value iteration, this leads to deterministic policies
with very sharp changes in behavior that make it hard to see dierences between methods,
and so we also show results with soft value iteration, which creates stochastic policies that
vary more continuously. As demonstrated in Figure 7.4, our experiments show that overall
the two methods perform very similarly, with some evidence that the Additive method is
slightly more robust. The Additive method also has the benet that it can be applied in
situations where the inferred reward and specied reward are over dierent feature spaces, by
CHAPTER 7. EVALUATION: CORRECTING MISSPECIFIED REWARDS AND
IMITATING SKILLS 67
creating the nal reward Rnal(s) =HTfH(s) +Rspec(s).
However, both of these methods are quite unprincipled, and do not resolve the con
ict
between the two goals. The presence of this con
ict is worrying: it should not be the
case that two dierent sources of information about the same underlying reward function
predictably con
ict; the fact that this is happening suggests that we have a poorly specied
model somewhere. The problem is that we are implicitly treating Rspecincorrectly: both the
Additive and Bayesian methods are implicitly suggesting that the true reward must be \near"
Rspec, but this is not actually the case: we believe that Rspecwill capture some aspects of
good behavior, but completely neglect other aspects; this need not correspond to low L2
distance in parameter space.
We could try to improve upon our assumptions about what information Rspecprovides,
in order to resolve the con
ict. For example, rather than interpreting specas the parameters
of a reward function , we could interpret it as a changelist : that is, it is a specication of how
the environment should be changed, relative to the current state of the environment . In this
case, in our vase environment, specwould be interpreted as saying \all else equal, it is better
forRto be at the purple door then at its current location", without making any claims about
what is preferred when all else is not equal (Boutilier et al., 2004). This information could
then be integrated with the inferred reward to provide a full reward function for R.
Taking a step back, we can see that in general, the inferred reward does not have enough
information about what Rshould do, and so it must be combined with some information
fromH. In our evaluation so far, we have considered the additional information to come from
spec, but this is not necessary { it could just as well come from preferences, demonstrations,
corrections, or natural language, for example. Thus, what we really want is a general
formalism that allows us to infer rewards from the state of the world, combine them with
information provided by Honline, and then use the resulting information to act in the
environment. We tackle this problem in the next chapter.
68
Chapter 8
Using assistance instead of reward
learning to integrate information
A central premise of this dissertation is that we cannot specify a perfect reward function
for powerful agents. A natural solution is to have agents be uncertain about the objective
and infer it from observations. While we have shown that some of this information can be
extracted from the state of the world, it also seems likely that some information will need to
come from human feedback.
So far we have implicitly been using the reward learning paradigm to combine reward
information from multiple sources, as in Chapter 7. In this chapter, we recast the entire
reward learning paradigm as a special, constrained case of the assistance paradigm, and
demonstrate that by removing these constraints we can gain signicant qualitative benets.
We then show how to recast learning from the state of the world in the assistance paradigm
to gain these benets.
In the reward learning paradigm (Leike et al., 2018; Jeon et al., 2020; Christiano et al.,
2017; Ziebart et al., 2010), a reward model is learned from human feedback, and then used
by a control algorithm to select actions in the environment. Crucially, the control algorithm
and the reward learning process do not interact: they are separate processes that optimize
separate objectives. When learning the reward model, multiple forms of information can be
integrated into a single posterior over reward functions.
In contrast, in the assistance paradigm (Hadeld-Menell et al., 2016; Fern et al., 2014),
the human His modeled as part of the environment and as having some latent goal that the
agentR(for robot) does not know. R's goal is to maximize this (unknown) human goal. In
this formulation, Rmust balance between actions that help learn about the unknown goal,
and control actions that lead to high reward. There is a single policy is responsible for both
learning about the goal (to the extent necessary) as well as acting in pursuit of the goal.
We claim that the assistance paradigm has several qualitative advantages over the reward
learning paradigm. The goal of this chapter is to clarify and illustrate these advantages. We
then show how to formalize state of the world learning within the assistance paradigm in
Section 8.6.
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 69
What is the reward
of ?All pies need ,
let me make it
If I use the to make , there won’t be any left
for and . I’ll wait for more information.
Learning about rewardMaking robust plans Preserving option value when possible
+
+
I won’t find out which pie is preferable before
Alice gets very hungry, so I’ll make .
Guessing when feedback is unavailable
+
Figure 8.1: Rmust cook a pie for H, by placing
our to make the pie dough, lling it with
Apple, Blueberry, or Cherry lling, and nally baking it. However, Rdoes not know which
llingHprefers, and His not yet available for questions. What should Rdo in this situation?
On the right, we show what qualitative reasoning we might want Rto use to handle the
situation.
Our key insight is that by integrating reward learning and control using the learned reward
into a single policy, the assistance formulation allows for reward learning to depend on how
the reward will be used, and vice versa . This integration enables new, desirable qualitative
behaviors:
1.Decision-making for \control" can take into account the fact that the agent can do
more \reward learning" in the future. This allows the agent to take actions that are
robustly instrumentally useful now, knowing that the intermediate results can be used
later once the reward is better understood (Section 8.4).
2.Decision-making for \reward learning" (e.g. which questions to ask the human) can take
into account how the resulting updates to the reward will be used for \control". This
allows the agent to only ask questions whose answers are decision-relevant (Section 8.4).
Consider for example the kitchen environment illustrated in Figure 8.1, in which Rmust
bake a pie for H.Ris uncertain about which type of pie Hprefers to have, and currently H
is at work and cannot answer R's questions. An assistive Rcan make the pie crust while
waiting for Hto return, after which Rcan ask her about her preferences over the lling
(Section 8.4). Rmay never clarify all of H's preferences: for example, Ronly needs to know
how to dispose of food if it turns out that the ingredients have gone bad (Section 8.4). If H
will help with making the pie, Rcan allowHto disambiguate her desired pie by watching
what lling she chooses (Section 8.4). These behaviors rely crucially on the integration of
reward learning and control into a single policy, and as such vanilla reward learning agents
do not show these behaviors.
To clarify and illustrate the advantages of the assistance paradigm, we rst precisely
characterize the dierences between reward learning and assistance, by showing that two
phase, communicative assistance is equivalent to reward learning (Section 8.3). We then give
qualitative examples of desirable behaviors that can only be expressed once these restrictions
are lifted, and thus are only exhibited by assistive agents (Section 8.4).
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 70
We do not mean to suggest that work on reward learning should be replaced by research on
assistance. Amongst other limitations, assistive agents are very computationally complex. Our
goal is simply to clarify what qualitative benets an assistive formulation could theoretically
provide. Further research is needed to develop ecient algorithms that can capture these
benets. Such algorithms may look like algorithms designed to solve assistance problems
as we have formalized them here, but they may also look like modied variants of reward
learning, where the modications are designed to provide the qualitative benets we identify.
8.1 Reward learning
We consider two variants of reward learning: non-active reward learning, in which Rmust
infer the reward by observing H's behavior, and active reward learning, in which Rmay
choose particular questions to ask Hin order to get particular feedback.
Anon-active reward learning problem P=hMnr;C;h;r;Pi;H;kicontains a
POMDP without reward Mnr=hS;AR;
R;OR;T;P0;
i, and instead Rhas access to a
parameterized reward space h;r;Pi.Ris able to learn about by observing Hmake
kdierent choices c, each chosen from a set of potential choices C. In order for Rto learn
from the human's choices, it also assumes access to the human decision function H(cj)
that determines how the human makes choices for dierent possible reward functions r.
Common decision functions include perfect optimality (Ng and Russell, 2000) and Boltzmann
rationality (Ziebart et al., 2010). As discussed in Section 2.4, there are many types of
choices (Jeon et al., 2020), including demonstrations (Argall et al., 2009; Ng and Russell,
2000; Ziebart et al., 2010; Fu et al., 2017; Gao et al., 2012), comparisons (Zhang et al., 2017;
Wirth et al., 2017; Christiano et al., 2017; Sadigh et al., 2017), corrections (Bajcsy et al.,
2017), proxy rewards (Hadeld-Menell et al., 2017b), natural language (Fu et al., 2019), etc.
A policy decision function f(c0:k1) produces a policy Rafter observing H's choices.fis
asolution if it maximizes expected reward EP;c0:k1H[ER(f(c0:k1))]. SinceH's choices
c0:k1do not aect the state of the environment that Ris acting in, this is equivalent to
choosingRthat maximizes expected reward given the posterior over reward functions, that
isEP(jc0:k1)
ER(R)
.
Anactive reward learning problem P=hMnr;Q;C;h;r;Pi;H;kiadds the
ability forRto askHparticular questions q2Qin order to get more targeted feedback about
. The human decision function H(cjq;) now depends on the question asked. A solution
consists of a question policy R
Q(qijq0:i1;c0:i1) and a policy decision function f(q0:k1;c0:k1)
that maximize expected reward EP;q0:k1R
Q;c0:k1H[ER(f(q0:k1;c0:k1))].
A typical algorithm (Eric et al., 2008; Daniel et al., 2014; Maystre and Grossglauser,
2017; Christiano et al., 2017; Sadigh et al., 2017; Zhang et al., 2017; Wilde et al., 2020) will
compute and ask q2Qthat maximizes an active learning criterion such as information
gain (Byk et al., 2019) or volume removal (Sadigh et al., 2017). Best results are achieved
by selecting questions with the highest value of information (Cohn, Robert W, 2016; Zhang
et al., 2017; Mindermann et al., 2018; Wilde et al., 2020), but these are usually much more
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 71
computationally expensive. Rthen nds a policy that maximizes expected reward under the
inferred distribution over , in order to approximately solve the original POMDP.
Note that a non-active reward learning problem is equivalent to an active reward learning
problem with only one question, since having just a single question means that Rhas no
choice in what feedback to get:
Proposition 7. Every non-active reward learning problem hMnr;C;h;r;Pi;H;kican
be reduced to an active reward learning problem.
Proof. We construct the active reward learning problem as hMnr;Q0;C;h;r;Pi;H0;ki,
whereQ0,fqgwhereqis some dummy question, and H0(cjq;),H(cj).
Suppose the solution to the new problem is hR0
Q;f0i. Sincef0is a solution, we have:
f0= argmax
^fE
P;q0:k1R0
Q;c0:k1H0(jqi;)h
ER(^f(q0:k1;c0:k1))i
= argmax
^fE
P;q0:k1=q;c0:k1H0(jq;)h
ER(^f(q0:k1=q;c0:k1))i
allqareq
= argmax
^fE
P;c0:k1H(j)h
ER(^f(q0:k1=q;c0:k1))i
:
Thusf(c0:k1) =f0(q0:k1=q;c0:k1) is a maximizer of EP;c0:k1H(j)h
ER(^f(c0:k1)i
,
making it a solution to our original problem.
Proposition 8. Every active reward learning problem hMnr;Q;C;h;r;Pi;H;kiwith
jQj= 1 can be reduced to a non-active reward learning problem.
Proof. Let the sole question in Qbeq. We construct the non-active reward learning problem
ashMnr;C;h;r;Pi;H0;ki, withH0(cj) =H(cjq;).
Suppose the solution to the new problem is f0. Then we can construct a solution to the
original problem as follows. First, note that R
Qmust beR
Q(qijq0:i1;c0:i1) = 1[qi=q],
since there is only one possible question q. Then by inverting the steps in the proof of Propo-
sition 7, we can see that f0is a maximizer of EP;q0:k1R
Q;c0:k1H(jqi;)h
ER(^f(jc0:k1))i
.
Thus, by dening f(q0:k1;c0:k1) =f0(c0:k1), we get a maximizer to our original problem,
makinghR
Q;fia solution to the original problem.
8.2 Assistance
The key idea of assistance is that helpful behaviors like reward learning are incentivized
whenRdoes not know the true reward Rand can only learn about it by observing human
behavior. So, we model the human Has part of the environment, leading to a two-agent
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 72
POMDP, and assume there is some true reward Rthat onlyHhas access to, while the
robotRonly has access to a model relating RtoH's behavior. Intuitively, as Racts in
the environment, it will also observe H's behavior, which it can use to make inferences about
the true reward.
Following Hadeld-Menell et al. (2016)1, we dene an assistance game Mas a tuple
M=hS;fAH;ARg;f
H;
Rg;fOH;ORg;T;PS;
;h;r;Pii:
HereSis a nite set of states, AHa nite set of actions for H,
Ha nite set of observations
forH, andOH:S!(
H) an observation function for H(respectivelyAR;
R;ORfor
R). The transition function T:SAHAR!(S) gives the probability over next states
given the current state and both actions. The initial state is sampled from PS2(S).  is
a set of possible reward function parameters which parameterize a class of reward functions
r:SAHARS!R, andPis the distribution from which is sampled.
2(0;1) is
a discount factor.
As with POMDPs, policies can depend on history. Both HandRare able to observe each
other's actions, and on a given timestep, Racts before H. We useR
t: (
RAHAR)tto
denoteR's observations until time t, andH
tforH's observations; thus R's policy can be
written asR(aRjoR
t;R
t1), whileH's can be written as H(aHjoH
t;aR
t;H
t1;). Note that
unlikeH,Rdoes not observe the reward parameter , and must infer much like it does the
hidden state.
Afully observable assistance game is one in which both HandRcan observe the
full state. In such cases, we omit
H;
R;OHandOR.
Since we have not yet specied how Hbehaves, it is not clear what the agent should
optimize for. Should it be playing a Nash strategy or optimal strategy pair of the game,
and if so, which one? Should it use a non-equilibrium policy, since humans likely do not use
equilibrium strategies? This is a key hyperparameter in assistance games, as it determines the
communication protocol forHandR. For maximum generality, we can equip the assistance
game with a policy-conditioned belief B: R!(H) overH, which species how the
human responds to the agent's choice of policy (Halpern and Pass, 2018). The agent's goal is
to maximize expected reward given this belief. We use policy-conditioned beliefs as opposed
to a simple unconditional distribution over human policies, because it allows us to model
a wide range of situations, including situations with prior coordination, or where humans
adapt to the robot's policy as a result of prior interactions.
Prior work on assistance games (Hadeld-Menell et al., 2016; Malik et al., 2018; Woodward
et al., 2019) focuses on nding optimal strategy pairs. This corresponds to a belief that H
will know and perfectly adapt to R's policy, as formalized below:
1Relative to Hadeld-Menell et al. (2016), our denition allows for partial observability and requires that
the initial distribution over Sand  be independent. We also have Hchoose her action sequentially after R,
rather than simultaneously with R, in order to better parallel the reward learning setting.
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 73
Proposition 9. LetM=hS;fAH;ARg;f
H;
Rg;fOH;ORg;T;PS;
;h;r;Piibe an
assistance game. Let B(R)(H)/ 1
EJR (H;R) = max
~H2HEJR (~H;R)
be an associated
policy-conditioned belief. Let Rbe the solution to hM;Bi. ThenhB(R);Riis an optimal
strategy pair.
Proof. LethH;Ribe an arbitrary strategy pair. Then EJR (H;R)EJR (B(R);R) by
the denition of B, andEJR (B(R);R)EJR (B(R);R) by the denition of R. Thus
EJR (H;R)EJR (B(R);R). SincehH;Riwas assumed to be arbitrary, hB(R);Ri
is an optimal strategy pair.
However, our goal is to compare assistance to reward learning. Typical reward learning
algorithms assume access to a model of human decision-making: for example, Hmight be
modeled as optimal (Ng and Russell, 2000) or Boltzmann-rational (Ziebart et al., 2010). As
a result, we also assume that we have access to a model of human decision-making H. Note
thatHdepends on : we are eectively assuming that we know how Hchooses how to
behave given a particular reward r. This assumption corresponds to the policy-conditioned
beliefB(R)(~H) = 1
~H=H
. We dene an assistance problem Pas a pairhM;Hi
whereHis a human policy for the assistance game M.
Given an assistance problem, a robot policy Rinduces a probability distribution over
trajectories: hs0;;H;Ri,2[SAHAR](whereXdenotes a sequence of X).
We denote the support of this distribution by Traj(R). The expected reward of a robot policy
forhM;Hiis given by
ER(R) = E
s0PS;P;hs0;;H;Ri"1X
t=0
tr(st;aH
t;aR
t;st+1)#
:
Asolution ofhM;Hiis a robot policy that maximizes expected reward: R=argmax
~RER(~R).
Solving assistance problems
Once theHis given,Hcan be thought of as an aspect of the environment, and can be
thought of as a particularly useful piece of information for estimating how good actions are.
This suggests that we can reduce the assistance problem to an equivalent POMDP, and then
solve the POMDP. Following Desai (2017), the key idea is to embed Hin the transition
functionTand embed in the state. It turns out we must also include the human's previous
actionaHin the state, so that the robot can observe it, and so that the reward can be
computed.
In theory, to embed potentially non-Markovian HinT, we need to embed the entire
history of the trajectory in the state, but this leads to extremely large POMDPs. In our
experiments, we only consider Markovian human policies, for which we do not need to embed
the full history, keeping the state space manageable. Thus, the policy can be written as
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 74
H(aHjoH;aR;). To ensure that Rmust inferfrom human behavior, as in the original
assistance game, the observation function does notreveal, but does reveal the previous
human action aH.
Formally, suppose that we have an assistance problem hM;Hiwith:
M=hS;fAH;ARg;f
H;
Rg;fOH;ORg;T;PS;
;h;r;Pii:
Then, the transformation M7!M0is given as follows:
S0,SAH State space

0,
RAHObservation space
O0(o0js0) =O0((oR;aH
1)j(s;aH
2;)) Observation function
, 1
aH
1=aH
2
OR(oRjs)
T0(s0
2js0
1;aR) =T0((s2;aH
1;2)j(s1;aH
0;1);aR) Transition function
,T(s2js1;aH
1;aR) 1[2=1]
X
oH2
HOH(oHjs1)H(aH
1joH;aR;)
r0(s0
1;aR;s0
2) =r0((s1;aH
0;);aR;(s2;aH
1;)) Reward function
,r(s1;aH
1;aR;s2)
P0
0(s0) =P0
0((s;aH;)) Initial state distribution
,PS(s)P() 1
aH=aH
init
whereaH
initis arbitrary
In the case where the original assistance problem is fully observable, the resulting POMDP
is an instance of a Bayes-Adaptive MDP (Martin, 1967; Du, 2002).
Any robot policy Rcan be translated from the APOMDP Mnaturally into an identical
policy onM0. Note that in either case, policies are mappings from (
R;AH;AR)
Rto
(AR). This transformation preserves optimal agent policies:
Proposition 10. A policyRis a solution ofMif and only if it is a solution of M0.
Proof. Recall that an optimal policy in the POMDPM0is one that maximizes the
expected value:
EV() = E
s0
0P0
0;0hs0
0;i"1X
t=0
tr0(s0
t;at;st+1)#
= E
s0
0P0
0;0hs0
0;i"1X
t=0
tr(st;aH
t;at;st+1)#
where the trajectories 0s are sequences of state, action pairs drawn from the distribution
induced by the policy, starting from state s0.
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 75
Similarly, an optimal robot policy Rin the APOMDP Mis one that maximizes its
expected reward:
ER(R) = E
s0PS;P;hs0;;Ri"1X
t=0
tr(st;aH
t;aR
t;st+1)#
:
To show that the optimal policies coincide, it suces to show that for any ,ER() (inM)
is equal to EV() (inM0). To do this, we will show that induces the \same" distributions
over the trajectories. For mathematical convenience, we will abuse notation and consider
trajectories of the form ;2(S;AH;AR); it is easy to translate trajectories of this
form to trajectories in either M0orM.
We will show that the sequence ;has the same probability when the robot takes the
policyin bothM0andMby induction on the lengths of the sequence.
First, consider the case of length 1 sequences. ;= [(s;aR;aH);]. Under bothM0and
M,sandare drawn from PSandPrespectively. Similarly, aRandaHare drawn from
R(joR
0) andH(joH;aR;) respectively. So the distribution of length 1 sequences is the
same under both M0andM.
Now, consider some longer sequence ;= [(s1;aR
1;aH
1);::::;(st;aR
t;aH
t);]. By the induc-
tive hypothesis, the distribution of ( s1;aH
1;aR
1);::::;(st1;aH
t1;aR
t1) andare identical; it
suces to show that ( st;aH
t;aR
t) has the same distribution, conditioned on the other parts of
;, underM0and underM. Yet by construction, stis drawn from the same distribution
T(jst1;aH
t1;aR
t1),aH
tis drawn from the same distribution H(joH
t;aR
t;), andaR
tis drawn
from the same distribution R(joR
t;R
t1)).
WhenMis fully observable, in the reduced POMDP is the only part of the state
not directly observable to the robot, making it an instance of a hidden-goal MDP (Fern
et al., 2014). For computational tractability, much of the work on hidden goals (Javdani
et al., 2015; Fern et al., 2014) selects actions assuming that all goal ambiguity is resolved in
one step . This eectively separates reward learning and control in the same way as typical
reward learning algorithms, thus negating many of the benets we highlight in future sections.
Intention-aware motion planning (Bandyopadhyay et al., 2013) also embeds the human goal
in the state in order to avoid collisions with humans during motion planning, but does not
consider applications for assistance.
Macindoe et al. (2012) uses the formulation of a POMDP with a hidden goal to produce
an assistive agent in a cops and robbers gridworld environment. Nikolaidis et al. (2015)
assumes a dataset of joint human-robot demonstrations, which they leverage to learn \types"
of humans that can then be inferred online using a POMDP framework. This is similar to
solving an assistance problem, where we think of the dierent values of as dierent \types"
of humans. Chen et al. (2018) uses an assistance-style framework in which the unknown
parameter is the human's trust in the robot (rather than the reward ). Woodward et al.
(2019) uses deep reinforcement learning to solve an assistance game in which the team must
collect either plums or lemons. To our knowledge, these are the only prior works that use
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 76
an assistive formulation in a way that does not ignore the information-gathering aspect of
actions. While these works typically focus on algorithms to solve assistance games, we instead
focus on the qualitative benets of using an assistance formulation.
Since we can reduce an assistance problem to a regular POMDP, we can use any POMDP
solver to nd the optimal R. In our examples for this paper, we use an exact solver when
feasible, and point-based value iteration (PBVI) (Pineau et al., 2003) or deep reinforcement
learning (DRL) when not. When using DRL, we require recurrent models, since the optimal
policy can depend on history.
A common confusion is to ask how DRL can be used, given that it requires a reward
signal, whereas Rdoes not know the reward function by assumption. This stems from a
misunderstanding of what it means for R\not to know" the reward function. When DRL is
run, at the beginning of each episode, a specic value of is sampled as part of the initial
state. The learned policy Ris not provided with : it can only see its observations oRand
human actions aH, and so it is accurate to say that R\does not know" the reward function.
However, the reward is calculated by the DRL algorithm, not by R, and the algorithm can
and does use the sampled value of for this computation. Rcan then implicitly learn the
correlation between the actions aHchosen byH, and the high reward values that the DRL
algorithm computes; this can be thought of as an implicit estimation of in order to choose
the right actions.
8.3 Reward learning as two-phase communicative
assistance
There are two key dierences between reward learning and assistance. First, reward learning
algorithms split reward learning and control into two separate phases, while assistance merges
them into a single phase. Second, in reward learning, the human's only role is to communicate
reward information to the robot, while in assistance the human can help with the task. These
two properties exactly characterize the dierence between the two: reward learning problems
and communicative assistance problems with two phases can be reduced to each other, in a
very natural way.
Acommunicative assistance problem is one in which the transition function Tand
the reward function rare independent of the choice of human action aH, and the human
policyH(joH;aR;) is independent of the observation oH. Thus, in a communicative
assistance problem, H's actions only serve to respond to R, and have no eects on the state
or the reward (other than by in
uencing R). Such problems can be cast as instances of
HOP-POMDPs (Rosenthal and Veloso, 2011).
For the notion of two phases, we will also need to classify robot actions as communicative
or not. We will assume that there is some distinguished action aR
noopthat \does nothing".
Then, a robot action ^aRiscommunicative if for anys;aH;s0we haveT(s0js;aH;^aR) =
T(s0js;aH;aR
noop) andR(s;aH;^aR;s0) =R(s;aH;aR
noop;s0). A robot action is physical if it
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 77
is not communicative.
Now consider a communicative assistance problem hM;Hiwith noop action aR
noopand
let the optimal robot policy be R. Intuitively, we would like to say that there is an initial
communication phase in which the only thing that happens is that Hresponds to questions
fromR, and then a second action phase in which Hdoes nothing and Racts. Formally, the
assistance problem is two phase with actions at tactif it satises the following property:
9aH
noop2AH;82Traj(R);
8t>><
>>>:P0(^s0); ^s=nk1;
1[^s0=ni+1];^s=niwithi<k1
T(^s0j^s;aR);^s2SandaR2A;
1[s0=s]; elseTransition function
r0
(^s;aH;aR;^s0),8
>>><
>>>:1; ^s2NandaR=2Q;
1; ^s2SandaR2Q;
0; ^s2NandaR2Q;
r(s;aR;s0);elseReward function
H0(aHjoH;aR;),(
H(aHjaR;); aR2Q
c0; elseHuman policy
aR0
noop,q0 Distinguished noop action
Technically r0
should not be allowed to return 1. However, since SandAare nite,ris
bounded, and so there exists some large nite negative number that is functionally equivalent
to1 that we could use instead.
Looking at the denitions, we can see T0andr0are independent of aH, andH0is
independent of oH, making this a communicative assistance problem. By inspection, we
can see that every q2Qis a communicative robot action. Any aR=2Qmust not be
a communicative action, because the reward r0
diers between aRandq0. Thus, the
communicative robot actions are Qand the physical robot actions are A.
Note that by construction of P0
SandT, we must have si=nifori2f0;1;:::k1g, after
whichskis sampled from P0and allst2Sfortk. Given this, by inspecting r0
, we can see
that an optimal policy must have aR
0:k12QandaR
k:=2Qto avoid the1 rewards. Since
aR
k:=2Q, we haveaH
k:=c0. Thus, setting aH
noop=c0, we have that the assistance problem is
two phase with actions at tact=k, as required.
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 79
Let a policy R0for the assistance problem be reasonable if it never assigns positive
probability to aR2Awhent<
>:R0
Q(aR
tjaR
0:t1;aH
0:t1); t<k andaR
0:t2Q0
f0(aR
0:k1;aH
0:k1)(aR
tjoR
k:t;aR
k:t1); tkandaR
0:k12Q0andaR
k:t2A0
0; else:
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 81
We show that there must exist a solution to Pthat is the analogous policy to some pair.
Assume towards contradiction that this is not the case, and that there is a solution Rthat
is not the analogous policy to some pair. Then we have a few cases:
1.Rassigns positive probability to aR
i=a =2Q0fori<k . This contradicts the two-phase
assumption.
2.Rassigns positive probability to aR
i=q2Qforik. This contradicts the two-phase
assumption.
3.R(aR
tjoR
t;R
t1) depends on the value of oR
ifor somei < k . Since both aH
0:k1and
aR
0:k1cannot aect the state or reward (as they are communicative), the distribution
overoR
0:k1is xed and independent of R, and so there must be some other Rthat
is independent of oR
0:k1that does at least as well. That Rwould be the analogous
policy to some pair, giving a contradiction.
Now, suppose we have some pair hR0
Q;f0i, and let its analogous policy be R. Then we
have:
E
P
q0:k1R0
Q
c0:k1H0[ER(f0(q0:k1;c0:k1))]
=E
P2
64E
q0:k1R
c0:k1H[ER(f0(q0:k1;c0:k1))]3
75
=E
P2
66664E
q0:k1R
c0:k1H2
66664E
s0P0
0
aR
tf0(q0:k1;c0:k1)
st+1T0(jst;aR
t)"1X
t=0
tr0
(st;aR
t;st+1)#3
777753
77775
=E
P2
66664E
q0:k1R
c0:k1H2
66664E
skP0
0
aR
tR(jhc0:k1;ok:ti;hq0:k1;ak:t1i)
st+1T0(jst;aR
t)"
1

k1X
t=k
tr0
(st;aR
t;st+1)#3
777753
77775
=E
P2
66664E
q0:k1R
c0:k1H2
66664E
skP0
0
aR
tR(jhc0:k1;ok:ti;hq0:k1;ak:t1i)
st+1T0(jst;aR
t)"
1

k1X
t=k
tr(st;aH
;aR
t;st+1)#3
777753
77775
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 82
However, since all the actions in the rst phase are communicative and thus don't impact
state or reward, the rst ktimesteps in the two phase assistance game have constant reward
in expectation. Let C=Es0:khPk1
t=0
tr(st;aH
;aR
noop;st+1)i
. This gives us:
E
P;q0:k1R0
Q;c0:k1H[ER(f0(q0:k1;c0:k1))]
=E
P"
E
s0PS;P;hs0;;H;Ri"
1

k1X
t=0
tr(st;aH
t;aR
t;st+1)##
1

kC
=1

k
ER(R)C
:
Thus, ifhR0
Q;f0iis a solution to the active reward learning problem, then Ris a solution
of the two-phase communicative assistance problem.
Corollary 14. If a two-phase communicative assistance problem hM;H;aR
noopihas exactly
one communicative robot action, it can be reduced to an equivalent non-active reward learning
problem.
Proof. Apply Proposition 13 followed by Proposition 8. (Note that the construction from
Proposition 13 does lead to an active reward learning problem with a single question, meeting
the precondition for Proposition 8.)
8.4 Qualitative improvements for general assistance
We have seen that reward learning is equivalent to two-phase communicative assistance
problems, where inferring the reward distribution can be separated from control using the
reward distribution. However, for general assistance games, it is necessary to merge estimation
and control, leading to several new qualitative behaviors. When the two phase restriction is
lifted, we observe plans conditional on future feedback and relevance aware active learning .
When the communicative restriction is lifted, we observe learning from physical actions .
We demonstrate these qualitative behaviors in simple environments using point-based
value iteration (PBVI) or deep reinforcement learning (DRL). For communicative assistance
problems, we also consider two baselines:
1.Active reward learning. This is the reward learning paradigm discussed so far.
2.Interactive reward learning. This is a variant of reward learning that aims to
recover some of the benets of interactivity, by alternating reward learning and acting
phases. During an action phase, Rchooses actions that maximize expected reward
under its current belief over (without \knowing" that its belief may change), while
during a reward learning phase, Rchooses questions that maximizes information gain.
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 83
Plans conditional on future feedback
Here, we show how an assistive agent can make plans that depend on obtaining information
aboutin the future. The agent can rst take some \preparatory" actions whose results can
be used later once the agent has claried details about . A reward learning agent would not
be able to do this, as it would require three phases (acting, then learning, then acting again).
We illustrate this with our original kitchen environment (Figure 8.1). Rmust bake a
pie forH, but doesn't know what type of pie Hwants: Apple, Blueberry, or Cherry. Each
type has a weight specifying the preference for that pie. Assuming people tend to like apple
pie the most and cherry pie the least, we have AUniform [2;4],BUniform [1;3], and
CUniform [0;2]. We dene the questions Q=fqA;qB;qCg, whereqXmeans \What is the
value ofX?", and thus, the answer set is C=R.
Rcan select ingredients to assemble the pie. Eventually, Rmust use \bake", which bakes
the selected ingredients into a nished pie, resulting in reward that depends on what type of
pie has been created. Hinitially starts outside the room, but will return at some prespecied
time.rassigns a cost of asking a question of 0 :1 ifHis inside the room, and 3 otherwise.
The horizon is 6 timesteps.
In this environment, Rneeds to bake either apple or blueberry pie (cherry is never
preferred over apple) within 6 timesteps, and may query Habout her preferences about the
pie. Making the pie takes 3 timesteps: rst Rmust make
our into dough, then it must
add one of the llings, and nally it must bake the pie. Baking the correct pie results in +2
reward, while baking the wrong one results in a penalty of -1. In addition, Hmight be away
for several timesteps at the start of the episode. Querying Hcosts 0.1 when she is present
and 3 when she is away.
We use PBVI to train an agent for this assistance problem with dierent settings for how
longHis initially away.
Assistance. Regardless of H's preferences ,Rwill need to use
our to make pie dough. So,
Ralways makes the pie dough rst, before querying Habout her preferences. Whether R
then queries Habout her preferences depends on how late Hreturns. If Harrives home
before timestep 5, Rwill query her about her preferences and then make the appropriate pie
as expected. However, if Hwill arrive later, then there will not be enough time to query her
for her preferences and bake a pie. Instead, Rbakes an apple pie, since its prior suggests
that that's what Hwants.
This behavior, where Rtakes actions (making dough) that are robustly good but waits
on actions (adding the lling) whose reward will be claried in the future, is very related to
conservative agency (Turner et al., 2020).
Reward learning. The assistance solution requires Rto act (to make dough), then to learn
preferences, and then to act again (to make pie). A reward learning agent can only have two
phases, and so we see one of two suboptimal behaviors. First, Rcould stay in the learning
phase until Hreturns home, then ask which pie she prefers, and then make the pie from
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 84
scratch. Second, Rcould make an apple pie without asking Hher preferences. (In this case
there would be no learning phase.) Which of these happens depends on the particular method
and hyperparameters used.
Interactive reward learning. Adding interactivity is not sucient to get the correct
behavior. Suppose we start with an action phase. The highest reward plan under R's current
belief overis to bake an apple pie, so that's what it will do, as long as the phase lasts long
enough. Conversely, suppose we start with a learning phase. In this case, Rdoes nothing
untilHreturns, and then asks about her preferences. Once we switch to an action phase, it
bakes the appropriate pie from scratch.
Why was integration of reward learning and control needed? The plan \make pie
dough and then wait" (a control behavior) is only a good plan because the agent has some
way of learning what lling to use in the future (a reward learning behavior). Thus, this plan
can only be selected because when selecting control behaviors the agent can reason about
future reward learning behaviors.
Relevance aware active learning
?
Figure 8.2: The wormy-apples kitchen en-
vironment. Hwants an apple, but R
might discover worms in the apple, and
have to dispose of it in either of the trash
or compost bins.Once we relax the two-phase restriction, Rstarts
to further optimize whether and when it asks
questions. In particular, since Rmay be uncertain
about whether a question's answer will even be
necessary,Rwill only ask questions once they
become immediately relevant to the task at hand.
In contrast, a reward learning agent would have
to decide at the beginning of the episode (during
the learning phase) whether or not to ask these
questions, and so cannot evaluate how relevant
they are.
Consider for example a modication to the
kitchen environment: Rknows that Hwants an
apple pie, but when Rpicks up some apples, there
is a 20% chance that it nds worms in some of
the apples. Ris unsure whether Hwants her
compost bin to have worms, and so does not
know whether to dispose of the bad apples in the
trash or compost bin. The robot gets 2 reward
for making an apple pie (regardless of how it disposed of any wormy apples), and gets 2
reward if it disposes of the apples in the wrong container. Additionally, asking a question
incurs a cost of 0 :1. IdeallyRwould only clarify H's preferences around the disposal of
wormy apples when Rhappens to pick up wormy apples.
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 85
We use PBVI to solve this environment, and report the results below.
Assistance. An assistive Ronly asks about wormy apples when it needs to dispose of one.
Ralways starts by picking up apples. If the apple does not have worms, Rimmediately uses
the apples to bake the pie. If some apples have worms and the cost of asking a question is
suciently low, RelicitsH's preferences and disposes of the apples appropriately. It then
bakes the pie with the remaining apples. This policy always gets the 2 reward from baking
the pie and never incurs the 2 reward from disposing of wormy apples in the wrong bin. It
only has to ask Ha question 80% of the time, and so incurs the 0 :1 cost of asking a question
20% of the time, leading to a total expected undiscounted reward of 1 :98.
This behavior, in which questions are asked only if they are useful for constraining future
behavior, has been shown previously using probabilistic recipe trees (PRTs) (Kamar et al.,
2009), but to our knowledge has not been shown with optimization-based approaches.
Reward learning. A reward learning policy must have only two phases and so would show
one of two undesirable behaviors: either it would always ask Hwhere to dispose of wormy
apples, or it never asks and instead guesses when it does encounter wormy apples.
With a lower discount rate ( = 0:9),R's policy never asks questions and instead simply
tries to make the apple pie, guessing which bin to dispose of wormy apples in if it encounters
any. Intuitively, since it would have to always ask the question at the beginning, it would
always incur a cost of 0 :1 as well as delay the pie by a timestep resulting in 10% less value,
and this is only valuable when there turn out to be worms andits guess about which bin to
dispose of them in is incorrect, which only happens 10% of the time. This ultimately isn't
worthwhile. This achieves an expected undiscounted reward of 1 :8.
With a higher discount rate of = 0:99, the two-phase policy will always ask about which
bin to dispose of wormy apples in, achieving 1 :9 expected undiscounted reward.
Interactive reward learning. If we start in the action phase and Rpicks up wormy apples,
it will dispose of them in an arbitrary bin without asking Habout her preferences, because
it doesn't \know" that it will get the opportunity to do so. Alternatively, if we start with a
learning phase, Rwill askHwhere to dispose of wormy apples, even if Rwould never pick
up any wormy apples.
Why was integration of reward learning and control needed? The question \where
should I dispose of wormy apples" (a reward learning behavior) should only be asked when
the agent picks up apples and they happen to be wormy (a control behavior). This can only
happen when the question selection mechanism is able to reason about how the resulting
information will be used for control.
While this example might seem trite, complex settings usually have many more questions.
ShouldRask ifHprefers seedless apples, should scientists ever invent them? Perhaps
Rshould ask if Hstill prefers apple over cherry when angry? Asking about all possible
situations is not scalable.
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 86
Learning from physical actions
So far we have considered communicative assistance problems, in which Honly provides
feedback rather than acting to maximize reward herself. Allowing Hto have physical actions
enables a greater variety of potential behaviors. Most clearly, when Rknows the reward
(that is,Pputs support over a single ), assistance games become equivalent to human-AI
collaboration (Nikolaidis and Shah, 2013; Carroll et al., 2019; Dimitrakakis et al., 2017).
With uncertain rewards, Rcan learn just by observing how Hacts in an environment, and
then work with Hto maximize reward, all within a single episode , as in shared autonomy with
intent inference (Javdani et al., 2015; Brooks and Szar, 2019) and other works that interpret
human actions as communicative Whitney et al. (2017). This can signicantly reduce the
burden on Hin providing reward information to R. Some work has shown that in such
situations, humans tend to be pedagogic : they knowingly take individually suboptimal actions,
in order to more eectively convey the goal to the agent (Ho et al., 2016; Hadeld-Menell
et al., 2016). An assistive Rwho knows this can quickly learn what Hwants, and help her
accomplish her goals.
+
+
+
+ +
 +
Figure 8.3: The CakeOrPie vari-
ant of the kitchen environment.
His equally likely to prefer cake
or pie. Communication must
take place through physical ac-
tions alone.We illustrate this with a variant of our kitchen environ-
ment, shown in Figure 8.3. There are no longer questions
and answers. Both HandRcan move to an adjacent free
space, and pick up and place the various objects. Only R
may bake the dessert. Ris uncertain whether Hprefers
cake or cherry pie. Preparing the desired recipe provides a
base value of V= 10, while the other recipe provides a base
value ofV= 1. Since Hdoesn't want the preparation to
take too long, the actual reward when a dessert is made is
given byrt=Vf(t), withf(t) = 1(t=N)4, andN= 20
as the episode horizon.
For both recipes, it is individually more ecient for
Hto pick up the dough rst. However, we assume His
pedagogic and wants to quickly show Rwhich recipe she
wants. So, if she wants cake, she will pick up the chocolate
rst to signal to Rthat cake is the preferred dessert.
It is not clear how exactly to think about this from
a reward learning perspective: there aren't any commu-
nicative human actions since every action alters the state
of the environment. In addition, there is no clear way
to separate out a given trajectory into two phases. This
situation cannot be easily coerced into the reward learning
paradigm, and so we only report the results in the assis-
tance paradigm. We solve the environment using Deep
Q-Networks (DQN; (Mnih et al., 2013)). We ran 6 seeds
for 5Mtimesteps and a learning rate of 104; results are shown in Figure 8.4.
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 87
0 1 2 3 4 5
Timesteps1e6456789Average Returnpedagogic H, optimal R
non-pedagogic H, optimal RCakeOrPie Environment
Figure 8.4: DQN smoothed learning curves on the CakeOrPie environment, with 6 seeds over
5Mtimesteps and learning rate of 104.
The assistive Rhandles this situation perfectly. It waits to see which ingredient Hwalks
towards, infers which recipe Hwants, and then helps Hby putting in the ingredients from
its side of the environment and baking the dessert. This is equivalent to pragmatic reasoning
(Goodman and Frank, 2016): \ Hwould have gone towards the chocolate if she wanted cake,
so the fact that she picked up the dough implies that she wants cherry pie". We emphasize
thatRis not explicitly programmed to reason in this manner.
Note thatRis not limited to learning from H's physical actions: Rcan also use its own
physical actions to \query" the human for information (Woodward et al., 2019; Sadigh et al.,
2016).
8.5 Discussion
So far we have discussed the benets of assistance over reward learning. Are there any
downsides?
The major limitation of assistance is that assistance problems are signicantly more
computationally complex, since we treat the unknown reward as the hidden state of a
POMDP. We are hopeful that this can be solved through the application of deep reinforcement
learning, which has been demonstrated to scale to large state, action and observation
spaces (OpenAI, 2018; Vinyals et al., 2019). Another avenue for future work is to modify
active reward learning algorithms in order to gain the benets outlined in Section 8.4, while
maintaining their computational eciency.
In addition, assistive agents will typically extract more information from Hthan reward
learning agents. While this leads to benets when correct inferences are made, it can also
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 88
lead to signicantly worse failures when His misspecied. We don't see this as a major
limitation: to the extent this is a major worry, we can design Hso that the robot only
makes inferences about human behavior in specic situations. For example, by having Hbe
independent of in a given state s, we ensure that the robot does not make any inferences
aboutin that state.
Limitations of assistance and reward learning
In addition to the downsides above, which are specic to assistance, there a number of
challenges that apply to both reward learning and assistance:
Human modeling. A major motivation for both paradigms is that reward specication
is very dicult. However, now we need to specify a prior over reward functions, and the
human model H. Consequently, misspecication can still lead to bad results (Armstrong
et al., 2020; Carey, 2018). While it should certainly be easier to specify a prior over with a
\grain of truth" on the true reward than to specify directly, it is less clear that we can
specifyHwell.
One possibility is to add uncertainty over the human policy H. However, this can only
go so far: information about must come from somewhere . IfRis suciently uncertain
aboutandH, then it cannot learn about the reward (Armstrong and Mindermann, 2018).
Thus, for good performance we need to model H. While imitation learning can lead to good
results (Carroll et al., 2019), the best results will likely require insights from a broad range of
elds that study human behavior.
Assumption that Hknows.Both paradigms assume that Hknows her reward exactly,
but in practice, human preferences change over time (Allais, 1979; Cyert and DeGroot, 1975;
Shogren et al., 2000). We could model this as the human changing their subgoals (Michini
and How, 2012; Park et al., 2020), adapting to the robot (Nikolaidis et al., 2017) or learning
from experience (Chan et al., 2019).
Dependence on uncertainty. All of the behaviors of Section 8.4, as well as previously
explored benets such as o switch corrigibility (Hadeld-Menell et al., 2017a), depend on
Rexpecting to gain information about . However, Rwill eventually exhaust the available
information about . If everything is perfectly specied, this is not a problem: Rwill have
converged to the true . However, in the case of misspecication, after convergence Ris
eectively certain in an incorrect , which has many troubling problems that we sought to
avoid in the rst place (Yudkowsky, year unknown).
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 89
8.6 Modeling the state of the world with assistance
Our original goal in this chapter was to nd a more principled way of integrating the reward
inferred using RLSP with information learned from human feedback and then act on that
information. The assistance paradigm provides a natural way of doing so. We model the
environment as an assistance problem, but enforce the condition that for the rst Ttimesteps,
Rdoes not yet exist, and is only deployed after Ttimesteps. At the time of deployment, R
can update its prior Pbased on the observed state of the environment, and subsequently it
may continue to ask Hfor more feedback about what to do.
Concretely, let us consider some state of the world problem hM;s0;H;T;h;R;Pii. We
can view this as capturing the rst Ttimesteps of a special assistance problem with a discrete
deployment event. We have hhS;fAH;ARg;f
H;
Rg;fOH;ORg;T;PS;
;h;r;Pii;Hi
as a standard assistance problem, and add as a restriction that Rmust take a noop action
aR
noopfor the rst Ttimesteps, and additionally is not allowed to observe the environment
orH's actions. After these Ttimesteps,Ris deployed and can act as normal within the
assistance problem.
This is technically speaking not a standard assistance problem, because we are imposing
conditions on R's policyR, as well as what Rgets to observe. We might consider it a
constrained assistance problem. Nonetheless, all of the discussion in the previous sections
still applies, and in particular, it can still be reduced to a POMDP, and solved using some
POMDP solver.
From this perspective, the point of the algorithms introduced in Chapters 5 and 6 that
solve the state of the world problem is that they are more ecient algorithms that can solve
a subpart of this constrained assistance problem. Specically, they compute how Rshould
update its belief over upon its deployment when observing s0. However, in order to get an
optimal policy for the assistance problem, we need to have the full distribution over , which
only Algorithm 1 does; Algorithms 2 and 3 only compute a point estimate and thus should
be thought of as ecient approximations.
Consider for example the kitchen oce environment in Figure 8.5. In this environment, H
wants to get work done on her laptop (which Rcannot do on her behalf), and also eventually
wants to eat a pie as in our original kitchen environment in Figure 8.1. In addition, there is a
vase in the middle of the room, which Hdoes not want to break, as in Figure 1.2. However,
Rdoes not know any of this, and is uncertain about the rewards of all three of these features.
As before,Rcan askHquestions about her preferences in order to clarify what it should do.
At the beginning of a trajectory sT, beforeRhas been deployed, Hhas just entered the
room through a door, ready to begin acting in the environment. When Ris deployed in state
s0, it observes that His now working at her laptop, and the vase remains intact. Simply by
observing the state of the environment, Rcan infer that Hprobably cares about getting her
work done (since she is at the laptop), and that she doesn't want to break the vase (since she
must have taken the long way around the vase in order to get to the laptop). However, this
doesn't tell Rwhat it should do: it can't help with H's work, and keeping the vase intact is
only an injunction of what not to do, rather than what to do.
CHAPTER 8. USING ASSISTANCE INSTEAD OF REWARD LEARNING TO
INTEGRATE INFORMATION 90
Figure 8.5: Kitchen oce environment. In this environment, Ris uncertain both about what
type of pie Hwould like, as well as whether it is acceptable to break vases. At the time of
deployment, Robserves that His working at her laptop, but the vase is still intact, and so
Rcan infer that vases should not be broken. However, Rstill does not know which pie H
would prefer, and so asks her for her preferences before nishing the pie.
Ris aware that Hmight want a specic type of pie, and that is something that it can
help with. As a result, it asks Habout her pie preferences, and then makes the appropriate
pie type for her to eat.
91
Chapter 9
Conclusion
One major goal of articial intelligence is to build articial agents that capably perform
tasks that we want. In order to build such agents, we must have some way of transmitting
information about what we want to the agent. The current paradigm in reinforcement
learning is to accomplish this by specifying a reward function , that denes which states are
desirable and which are not. Unfortunately, it is very dicult for humans to correctly specify
such a reward function.
Instead, it is important that our agents treat all source of information as observations
that provide information about the intended goal, rather than denitions of it. Once we
make this distinction, it becomes evident that our agents should collect many dierent kinds
of information, and integrate them together, in order to make better inferences about the
intended goal.
This dissertation identied a potentially important source of such information: the state
of the world. Unlike other sources of information, such as human demonstrations and
comparisons, this source is free: it does not require humans to change their behavior at all.
The agent must simply look at the world, infer what humans have put eort into, and make
appropriate inferences about what it should and should not do. While this will likely not be
enough to uniquely identify the task that the agent should perform, it can reduce the amount
of information that the agent needs to collect from other, more expensive sources.
We have demonstrated both theoretically and empirically that learning from the state
of the world can enable agents to make meaningful inferences, at least within some simple
environments. However, there is still much work to be done to turn this into a practical
method that can be applied in realistic scenarios.
9.1 Avenues for improvement
There are multiple avenues for further research, both conceptually and practically.
CHAPTER 9. CONCLUSION 92
Conceptual changes
Handling multiple humans. In many realistic environments, there is not a single human H
who is in
uencing the environment: many people act simultaneously, and the state is a result
of joint optimization by all of them. One approach is to infer H's goals by modeling other
people as part of the (very complex) environment. However, this seems like a particularly
challenging approach, and may require a fairly complex human model in order to account for
interpersonal interactions: a simple Boltzmann rational model may not be sucient.
An alternative approach is to distinguish all of the humans as separate from the environ-
ment, and to instead use the state of the world to infer the norms by which they operate.
These norms are likely the most relevant aspect of the reward function for Rto know, as
they determine how Rshould constrain its behavior within the group. They may also be
easier to infer, since by denition nearly all of the humans will agree on the norms and so
they will have an outsized impact on the state of the world.
Humans are optimized for the environment. Even when the state of the world is
optimized for human preferences, this may not be a result of human optimization, as assumed
in this dissertation. For example, we prefer that the atmosphere contain oxygen for us to
breathe. That the atmosphere has enough oxygen for us to breathe is not a result of human
optimization { indeed, it seems that the atmosphere meets this preference in spite of human
action. Rather, humans were optimized through the process of evolution to be able to breathe
the existing atmosphere.
The algorithms presented in this dissertation would not be able to infer such a preference,
except indirectly, for example by inferring that humans prefer not to die or get sick, and thus
instrumentally preferring that the atmosphere contain sucient oxygen. The issue is that
these algorithms assume that humans are the source of all optimization, and thus neglect the
fact that humans have themselves be optimized. It is unclear how to modify the algorithms
to account for this issue.
Learning tasks to perform. The apple collection environment (Figure 7.1d) and MuJoCo
skill learning (Section 7.3) demonstrate that it is possible to learn aspects of what should be
done in an environment. However, it is not clear that this is desirable , since the robot may
perform a task that has been inferred, instead of the task that Hexplicitly sets for it. Just
becauseRhas inferred that Hwants the apples harvested doesn't mean it should go do so,
especially if Hhas currently asked Rto wash the dishes.
It is possible that this would not be a problem in a suitable formulation: as long as R
assigns suciently high probability to HaskingRfor the thing that Hwants most, Rwill
followH's instructions, unless it has reason to think that His incorrect (Milli et al., 2017).
(For example, Rshould probably disobey Hif it knows that the house is on re and knows
thatHdoes not know this.)
Nevertheless, we may want to prevent Rfrom learning to autonomously pursue tasks
CHAPTER 9. CONCLUSION 93
based solely on the state of the world. Ideally, we would be able to decompose the inferred
reward into the frame conditions frame, which specify what Rshould not do, and the task
rewardtask, which specify things that could be good for Rto do. Given such a decomposition,
Rcould ensure that it takes frame into account in its decision-making process, while ignoring
the implications of task. One way to do this is to look across a variety of environments,
or a variety of humans, for commonalities in the reward: it seems likely that these would
correspond to frame conditions.
Practical improvements
Other sources of empirical knowledge. Learning from the state of the world can be
thought of as a solution to the \is-ought problem" (Hume, 2003) for articial agents: it allows
an agent to learn what it ought to do, given only knowledge about what istrue. (Readers
familiar with Hume may wonder what assumption allows us to bridge the gap from \is" facts
to \ought" facts. In this case, the bridging assumption is that Rought to do that which H
has put eort into (and what Hhas put eort into is an \is" fact.)
However, our algorithms so far either require us to have a full, complete understanding of
empirical facts as with RLSP (Algorithm 2), or they use the barest of empirical knowledge,
that which can be gleaned from random rollouts, as in Deep RLSP (Algorithm 3). It seems
clear that we can do better, by leveraging better algorithms and data sources for learning
empirical facts about the world, before trying to learn from the state of the world. For example,
how might we leverage the common-sense knowledge of large language models (Radford et al.,
2019; Brown et al., 2020)?
Human models. In this dissertation we used the simple model of Boltzmann rationality as
our model of H. While this worked well in our simple conceptual environments, it is not clear
that this will scale well to larger environments. Recent work in inverse reinforcement learning
has considered accounting for human biases (Evans et al., 2016; Majumdar et al., 2017; Shah
et al., 2019b); similar approaches may be needed for learning from the state of the world.
Handling partial observability. Our formalism for learning from the state of the world
(Chapter 3) assumes that His acting in an MDP without reward MnR . However, most
realistic environments are partially observable; neither HnorRcan observe the entire
world state at every timestep. While we have generalized to partial observability using the
assistance formulation, assistance problems are quite computationally challenging to solve.
Can we generalize our original formalism to handle partial observability, without requiring
the increase in computational complexity of assistance?
It is relatively easy to handle partial observability for H: the only dierence is that our
human model Hmust now work o of observations, rather than states. However, it is more
challenging to handle partial observability for R: what exactly does Rget to observe? It
should no longer be able to observe the full state of the world s0, but it also is not sucient
CHAPTER 9. CONCLUSION 94
to just observe o0, because this may only be a small part of the world. For example, if R
can't see an unbroken vase in o0, but does see after two actions in o2, then it should still
learn from the state of the world that vases probably should not be broken. The crucial
dierence is that when observing a full state s0, the posterior P(js0) captures allof the
information we can gain from simulating the past, due to the Markov property of states. In
contrast,P(jo0) does not capture all of this information, and future observations o1;o2;:::
can provide additional insights.
This is handled naturally by the assistance formulation, in which R's optimal policy
will seek out new observations in future timesteps that can then be leveraged to learn from
the state of the world (as long as this information is suciently valuable). Are there good
heuristics that allow us to capture this behavior, without requiring us to solve a full assistance
problem?
Integrating RLSP with assistance. While we formalized learning from the state of the
world using the assistance framework, even with full observability, the optimal policy for an
assistance problem typically requires the full posterior P(js0), whereas both RLSP and
Deep RLSP compute a point estimate of . How can we use these algorithms (or analogs
thereof) in order to more eciently solve the assistance formulation of learning from the
state of the world?
Learned models. The gradient in Deep RLSP depends on having access to a good model
ofH, (H)1, andT1, as well as a good learned feature function f. We used relatively
simple multilayer perceptrons (MLPs) and variational autoencoders (VAEs) to learn these
models, as we found them to be the most stable and easiest to tune. However, progress in
these areas is rapid, especially in self-supervised representation learning. It seems likely that
applying well-tuned state-of-the-art techniques will allow for signicantly improved results,
and could be a good approach to few-shot imitation learning for robotics.
9.2 Applications
Our fundamental insight is that the state of the world can inform us about human preferences
by showing us what humans put eort into. While we formalized this in the paradigm of
sequential decision-making, and the ideas in the previous section continue to work within
this paradigm, the general idea is broader and can be applied in many domains. We provide
a few examples as illustration, but it seems likely that there are many more applications that
have not yet been thought of.
Satellite imagery. Satellite imagery is a rich source of data about the state of the literal
world. It is already being used for many purposes, such as predicting how wealthy various
regions are in order to better target policy and aid (Abelson et al., 2014; Yeh et al., 2020)
and predicting crop yields (Pantazi et al., 2016). An intelligent AI system with enough
CHAPTER 9. CONCLUSION 95
background knowledge of the world should be able to infer a lot from such a dataset. For
example, if the AI system already knows the wealth of a region, then the prevalence of schools
in that region may be informative of how much the people of that region value education.
Social media. A person's social media feed can be a rich source of information about their
preferences. Social media companies can learn a lot about an individual user through their
pattern of clicks and views on a variety of posts. However, even without access to the internal
datasets, we can learn a lot about both individual people and various groups by looking at
social media feeds. We can model a social feed as joint optimization by the user and the
recommendation algorithm on the set of posts shown to the user. Thus, by observing the set
of posts that are currently being shown, we can infer what the user must care about reading.
(This does require having a suciently good model of the workings of the recommendation
algorithm, which may be challenging.)
Learning from prices. By using the formalism of sequential decision-making, we have
implicitly made the assumption that time is the unit of eort. At each timestep, Hhas a
variety of options available to her; the fact that she chooses one option over many others
thus provides signicant information about what Hcares about.
However, we could also view money as the unit of eort: everyone has a limited supply of
it, and can use it to pursue many dierent goals, and so the choice of what Hdoes with her
money is very informative about her preferences.
We can even think of the market prices and trading volumes of various items as informative
about the \preferences" of humanity as whole. For example, despite being more expensive,
vegan meat substitutes are growing in popularity; from this an agent might deduce that the
preference against meat-eating is rising.
It should be noted that prices would be thought of as a lower bound on value, because
prices are a function both of how valuable an item is as well as how easy it is to produce it.
Just because clean water is cheap doesn't mean it is not important to human preferences.
9.3 Closing thoughts
AI systems are poised to have a massive impact on human civilization within the next
century. Researchers today have a unique opportunity to shape this impact to ensure that
it is benecial for humanity. By showing how an AI system can learn what it should do
by reasoning about the provenance of the state of the world, this dissertation takes a step
towards AI systems that do what we intend them to do. With further progress towards
such AI systems, we can create a world in which AI systems assist us in achieving our goals,
whether they be as mundane as washing laundry or as grand as colonizing the stars.
96
Bibliography
Brian Abelson, Kush R Varshney, and Joy Sun. Targeting direct cash transfers to the
extremely poor. In Proceedings of the 20th ACM SIGKDD international conference on
Knowledge discovery and data mining , pages 1563{1572, 2014.
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option
discovery algorithms. arXiv preprint arXiv:1807.10299 , 2018.
Maurice Allais. The so-called allais paradox and rational decisions under uncertainty. In
Expected utility hypotheses and the Allais paradox , pages 437{681. Springer, 1979.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan
Man e. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 , 2016.
Oana Fabiana Andreescu. Static analysis of functional programs with an application to the
frame problem in deductive verication . PhD thesis, Rennes 1, 2017.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder,
Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight
experience replay. Advances in neural information processing systems , 30:5048{5058, 2017.
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot
learning from demonstration. Robotics and autonomous systems , 57(5):469{483, 2009.
Stuart Armstrong and Benjamin Levinstein. Low impact articial intelligences. arXiv preprint
arXiv:1705.10720 , 2017.
Stuart Armstrong and S oren Mindermann. Occam's razor is insucient to infer the preferences
of irrational agents. In Advances in Neural Information Processing Systems , pages 5598{
5609, 2018.
Stuart Armstrong, Jan Leike, Laurent Orseau, and Shane Legg. Pitfalls of learning a reward
function online. arXiv preprint arXiv:2004.13654 , 2020.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli,
and Edward Grefenstette. Learning to understand goal specications by modelling reward.
InInternational Conference on Learning Representations (ICLR) , 2019.
BIBLIOGRAPHY 97
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine
Intelligence 15, Intelligent Agents [St. Catherine's College, Oxford, July 1995] , pages
103{129, Oxford, UK, UK, 1999. Oxford University. ISBN 0-19-853867-7. URL http:
//dl.acm.org/citation.cfm?id=647636.733043 .
Andrea Bajcsy, Dylan P Losey, Marcia K O'Malley, and Anca D Dragan. Learning robot
objectives from physical human interaction. Proceedings of Machine Learning Research , 78:
217{226, 2017.
Tirthankar Bandyopadhyay, Kok Sung Won, Emilio Frazzoli, David Hsu, Wee Sun Lee, and
Daniela Rus. Intention-aware motion planning. In Algorithmic Foundations of Robotics X ,
pages 475{491. Springer, 2013.
Christopher M Bishop. Mixture density networks. Neural Computing Research Group Report,
Aston University, 1994.
Erdem Byk, Malayandi Palan, Nicholas C Landol, Dylan P Losey, and Dorsa Sadigh.
Asking easy questions: A user-friendly approach to active reward learning. arXiv preprint
arXiv:1910.04365 , 2019.
Andreea Bobu, Dexter RR Scobee, Jaime F Fisac, S Shankar Sastry, and Anca D Dragan.
Less is more: Rethinking probabilistic models of human behavior. In Proceedings of the
2020 ACM/IEEE International Conference on Human-Robot Interaction , pages 429{437,
2020.
Nick Bostrom. Superintelligence: Paths, Dangers, Strategies . Oxford University Press, Inc.,
USA, 2014.
David D Bourgin, Joshua C Peterson, Daniel Reichman, Stuart J Russell, and Thomas L
Griths. Cognitive model priors for predicting human decisions. In International conference
on machine learning , pages 5133{5141, 2019.
Craig Boutilier, Ronen I Brafman, Carmel Domshlak, Holger H Hoos, and David Poole.
Cp-nets: A tool for representing and reasoning with conditional ceteris paribus preference
statements. Journal of articial intelligence research , 21:135{191, 2004.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie
Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.
Connor Brooks and Daniel Szar. Balanced information gathering and goal-oriented actions
in shared autonomy. In 2019 14th ACM/IEEE International Conference on Human-Robot
Interaction (HRI) , pages 85{94, 2019.
Daniel S Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond
suboptimal demonstrations via inverse reinforcement learning from observations. arXiv
preprint arXiv:1904.06387 , 2019.
BIBLIOGRAPHY 98
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020.
Lars Buesing, Theophane Weber, S ebastien Racaniere, SM Eslami, Danilo Rezende, David P
Reichert, Fabio Viola, Frederic Besse, Karol Gregor, Demis Hassabis, et al. Learning and
querying fast generative models for reinforcement learning. arXiv preprint arXiv:1802.03006 ,
2018.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A
Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355 ,
2018.
Ryan Carey. Incorrigibility in the CIRL framework. In Proceedings of the 2018 AAAI/ACM
Conference on AI, Ethics, and Society , pages 30{35, 2018.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griths, Sanjit Seshia, Pieter Abbeel, and
Anca Dragan. On the utility of learning about humans for human-ai coordination. In
Advances in Neural Information Processing Systems , pages 5174{5185, 2019.
Lawrence Chan, Dylan Hadeld-Menell, Siddhartha Srinivasa, and Anca Dragan. The assistive
multi-armed bandit. In 2019 14th ACM/IEEE International Conference on Human-Robot
Interaction (HRI) , pages 354{363. IEEE, 2019.
Min Chen, Stefanos Nikolaidis, Harold Soh, David Hsu, and Siddhartha Srinivasa. Plan-
ning with trust for human-robot collaboration. In Proceedings of the 2018 ACM/IEEE
International Conference on Human-Robot Interaction , pages 307{315, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Georey Hinton. A simple framework
for contrastive learning of visual representations. In Proceedings of International Conference
on Machine Learning (ICML) , 2020.
Rohan Choudhury, Gokul Swamy, Dylan Hadeld-Menell, and Anca D Dragan. On the
utility of model learning in hri. In 2019 14th ACM/IEEE International Conference on
Human-Robot Interaction (HRI) , pages 317{325. IEEE, 2019.
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism
prediction instruments. Big Data , 5(2):153{163, 2017.
Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. Back to basics: Benchmarking
canonical evolution strategies for playing atari. arXiv preprint arXiv:1802.08842 , 2018.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.
Deep reinforcement learning from human preferences. In Advances in Neural Information
Processing Systems , pages 4299{4307, 2017.
BIBLIOGRAPHY 99
Jack Clark and Dario Amodei. Faulty reward functions in the wild, 2016. URL https:
//blog.openai.com/faulty-reward-functions .
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-
training text encoders as discriminators rather than generators. In International Conference
on Learning Representations (ICLR) , 2020.
Cohn, Robert W. Maximizing Expected Value of Information in Decision Problems by
Querying on a Wish-to-Know Basis. PhD thesis, University of Michigan, 2016.
Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic
decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD
International Conference on Knowledge Discovery and Data Mining , pages 797{806, 2017.
Richard M Cyert and Morris H DeGroot. Adaptive utility. In Adaptive Economic Models ,
pages 223{246. Elsevier, 1975.
Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, and Jan Peters. Active reward
learning. In Robotics: Science and systems , 2014.
Nishant Desai. Uncertain reward-transition mdps for negotiable reinforcement learning. 2017.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of
deep bidirectional transformers for language understanding. In Proceedings of the Conference
of the North American Chapter of the Association for Computational Linguistics: Human
Language Technologies, Volume 1 (Long and Short Papers) , 2019.
Christos Dimitrakakis, David C Parkes, Goran Radanovic, and Paul Tylkin. Multi-view
decision processes: the helper-ai problem. In Advances in Neural Information Processing
Systems , pages 5443{5452, 2017.
Andreas Doerr, Christian Daniel, Martin Schiegg, Duy Nguyen-Tuong, Stefan Schaal, Marc
Toussaint, and Sebastian Trimpe. Probabilistic recurrent state-space models. In Proceedings
of International Conference on Machine Learning (ICML) , 2018.
Michael O Du. Optimal learning: Computational procedures for Bayes-adaptive Markov
decision processes. PhD thesis, University of Massachusetts Amherst, 2002.
Ashley D Edwards, Himanshu Sahni, Yannick Schroeker, and Charles L Isbell. Imitating
latent policies from observation. arXiv preprint arXiv:1805.07914 , 2018.
Brochu Eric, Nando D Freitas, and Abhijeet Ghosh. Active preference learning with discrete
choice data. In Advances in Neural Information Processing Systems , pages 409{416, 2008.
Owain Evans, Andreas Stuhlm uller, and Noah Goodman. Learning the preferences of
ignorant, inconsistent agents. In Proceedings of the Thirtieth AAAI Conference on Articial
Intelligence , 2016.
BIBLIOGRAPHY 100
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you
need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 , 2018.
Alan Fern, Sriraam Natarajan, Kshitij Judah, and Prasad Tadepalli. A decision-theoretic
model of assistance. Journal of Articial Intelligence Research , 50:71{104, 2014.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal
control via policy optimization. In International Conference on Machine Learning , pages
49{58, 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse
reinforcement learning. arXiv preprint arXiv:1710.11248 , 2017.
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to
goals: Inverse reinforcement learning for vision-based instruction following. arXiv preprint
arXiv:1902.07742 , 2019.
Sunil Gandhi, Tim Oates, Tinoosh Mohsenin, and Nicholas Waytowich. Learning from
observations using a single video demonstration and human feedback. In Proceedings of
the 18th International Conference on Autonomous Agents and MultiAgent Systems , 2019.
Yang Gao, Jan Peters, Antonios Tsourdos, Shao Zhifei, and Er Meng Joo. A survey of inverse
reinforcement learning techniques. International Journal of Intelligent Computing and
Cybernetics , 2012.
Noah D Goodman and Michael C Frank. Pragmatic language interpretation as probabilistic
inference. Trends in Cognitive Sciences , 20(11):818{829, 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: O-
policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings
of International Conference on Machine Learning (ICML) , 2018.
Dylan Hadeld-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Cooperative
Inverse Reinforcement Learning. In Advances in Neural Information Processing Systems ,
pages 3909{3917, 2016.
Dylan Hadeld-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The o-switch
game. In Workshops at the Thirty-First AAAI Conference on Articial Intelligence , 2017a.
Dylan Hadeld-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan.
Inverse reward design. In Advances in Neural Information Processing Systems , pages
6765{6774, 2017b.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and
James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of
International Conference on Machine Learning (ICML) , 2019.
BIBLIOGRAPHY 101
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control:
Learning behaviors by latent imagination. In International Conference on Learning
Representations (ICLR) , 2020.
Joseph Y Halpern and Rafael Pass. Game theory with translucent players. International
Journal of Game Theory , 47(3):949{976, 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for
unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition (CVPR) , 2020.
Christopher Hesse, John Schulman, Vicki Pfau, Alex Nichol, Oleg Klimov, and Larissa
Schiavo. Retro contest, 2018. https://openai.com/blog/retro-contest/ .
Ashley Hill, Antonin Ran, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene
Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert,
Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https:
//github.com/hill-a/stable-baselines , 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in
Neural Information Processing Systems , 2016.
Mark K Ho, Michael Littman, James MacGlashan, Fiery Cushman, and Joseph L Austerweil.
Showing versus doing: Teaching by demonstration. In Advances in Neural Information
Processing Systems , pages 3027{3035, 2016.
David Hume. A treatise of human nature . Courier Corporation, 2003.
Borja Ibarz, Jan Leike, Tobias Pohlen, Georey Irving, Shane Legg, and Dario Amodei.
Reward learning from human preferences and demonstrations in atari. In Advances in
Neural Information Processing Systems , 2018.
Alexis Jacq, Matthieu Geist, Ana Paiva, and Olivier Pietquin. Learning from a learner. In
International Conference on Machine Learning , pages 2990{2999, 2019.
Shervin Javdani, Siddhartha S Srinivasa, and J Andrew Bagnell. Shared autonomy via
hindsight optimization. Robotics Science and Systems: online proceedings , 2015, 2015.
Hong Jun Jeon, Smitha Milli, and Anca D Dragan. Reward-rational (implicit) choice: A
unifying formalism for reward learning. arXiv preprint arXiv:2002.04833 , 2020.
Daniel Kahneman. Thinking, fast and slow . Macmillan, 2011.
Ece Kamar, Ya'akov Gal, and Barbara J Grosz. Incorporating helpful behavior into collabora-
tive planning. In Proceedings of The 8th International Conference on Autonomous Agents
and Multiagent Systems (AAMAS) . Springer Verlag, 2009.
BIBLIOGRAPHY 102
Antti Kangasr a asi o and Samuel Kaski. Inverse reinforcement learning from summary data.
Machine Learning , 107(8-10):1517{1535, 2018.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick Van der Smagt. Deep
variational bayes lters: Unsupervised learning of state space models from raw data. In
International Conference on Learning Representations (ICLR) , 2017.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International
Conference on Learning Representations (ICLR) , 2014.
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-os in the fair
determination of risk scores. arXiv preprint arXiv:1609.05807 , 2016.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement:
The tamer framework. In Proceedings of the fth international conference on Knowledge
capture , pages 9{16. ACM, 2009.
Victoria Krakovna. Specication gaming examples in AI, 2018. URL https://vkrakovna.
wordpress.com/2018/04/02/specification-gaming-examples-in-ai/ .
Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg.
Penalizing side eects using stepwise relative reachability. arXiv preprint arXiv:1806.01186 ,
2018.
Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J Russell, and Pieter Abbeel. Learning
plannable representations with causal infogan. In Advances in Neural Information Processing
Systems , 2018.
Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and
Sergio Guadarrama. Predictive information accelerates learning in rl. Advances in Neural
Information Processing Systems , 33, 2020.
Joel Lehman, Je Clune, and Dusan Misevic. The surprising creativity of digital evolution.
InArticial Life Conference Proceedings , pages 55{56. MIT Press, 2018.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg.
Scalable agent alignment via reward modeling: a research direction. arXiv preprint
arXiv:1811.07871 , 2018.
David A Levin and Yuval Peres. Markov chains and mixing times , volume 107. American
Mathematical Soc., 2017.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and
review. arXiv preprint arXiv:1805.00909 , 2018.
BIBLIOGRAPHY 103
Manuel Lopes, Francisco Melo, and Luis Montesano. Active learning for reward estimation
in inverse reinforcement learning. In Joint European Conference on Machine Learning and
Knowledge Discovery in Databases , pages 31{46. Springer, 2009.
James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, David Roberts, Matthew E Taylor,
and Michael L Littman. Interactive learning from policy-dependent human feedback. arXiv
preprint arXiv:1701.06049 , 2017.
Owen Macindoe, Leslie Pack Kaelbling, and Tom as Lozano-P erez. POMCoP: Belief space
planning for sidekicks in cooperative games. In Eighth Articial Intelligence and Interactive
Digital Entertainment Conference , 2012.
Anirudha Majumdar, Sumeet Singh, Ajay Mandlekar, and Marco Pavone. Risk-sensitive
inverse reinforcement learning via coherent risk models. In Robotics: Science and Systems ,
2017.
Dhruv Malik, Malayandi Palaniappan, Jaime F Fisac, Dylan Hadeld-Menell, Stuart Russell,
and Anca D Dragan. An ecient, generalized Bellman update for cooperative inverse
reinforcement learning. arXiv preprint arXiv:1806.03820 , 2018.
James John Martin. Bayesian decision problems and Markov chains . Wiley, 1967.
Lucas Maystre and Matthias Grossglauser. Just sort it! A simple and eective approach to
active preference learning. In Proceedings of the 34th International Conference on Machine
Learning , pages 2344{2353, 2017.
John McCarthy and Patrick J Hayes. Some philosophical problems from the standpoint of
articial intelligence. In Readings in Articial Intelligence , pages 431{450. Elsevier, 1981.
Bernard Michini and Jonathan P How. Bayesian nonparametric inverse reinforcement learning.
InJoint European conference on machine learning and knowledge discovery in databases ,
pages 148{163. Springer, 2012.
Smitha Milli, Dylan Hadeld-Menell, Anca Dragan, and Stuart Russell. Should robots be
obedient? arXiv preprint arXiv:1705.09990 , 2017.
S oren Mindermann, Rohin Shah, Adam Gleave, and Dylan Hadeld-Menell. Active inverse
reward design. arXiv preprint arXiv:1809.03060 , 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan
Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine.
Visual reinforcement learning with imagined goals. In Advances in Neural Information
Processing Systems , pages 9191{9200, 2018.
BIBLIOGRAPHY 104
Andrew Y Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. In
International Conference on Machine learning , 2000.
Stefanos Nikolaidis and Julie Shah. Human-robot cross-training: computational formulation,
modeling and evaluation of a human team training strategy. In 2013 8th ACM/IEEE
International Conference on Human-Robot Interaction (HRI) , pages 33{40. IEEE, 2013.
Stefanos Nikolaidis, Ramya Ramakrishnan, Keren Gu, and Julie Shah. Ecient model
learning from joint-action demonstrations for human-robot collaborative tasks. In 2015
10th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , pages
189{196. IEEE, 2015.
Stefanos Nikolaidis, David Hsu, and Siddhartha Srinivasa. Human-robot mutual adaptation
in collaborative tasks: Models and experiments. The International Journal of Robotics
Research , 36(5-7):618{634, 2017.
Stephen M Omohundro. The basic AI drives. In Articial General Intelligence , pages 483{492,
2008.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive
predictive coding. arXiv preprint arXiv:1807.03748 , 2018.
OpenAI. OpenAI Five, 2018. https://openai.com/blog/openai-five/ .
Xanthoula Eirini Pantazi, Dimitrios Moshou, Thomas Alexandridis, Rebecca L Whetton, and
Abdul Mounem Mouazen. Wheat yield prediction using machine learning and advanced
sensing techniques. Computers and Electronics in Agriculture , 121:57{65, 2016.
Daehyung Park, Michael Noseworthy, Rohan Paul, Subhro Roy, and Nicholas Roy. Inferring
task goals and constraints using bayesian nonparametric inverse reinforcement learning. In
Conference on Robot Learning , pages 1005{1014, 2020.
Joelle Pineau, Geo Gordon, Sebastian Thrun, et al. Point-based value iteration: An anytime
algorithm for POMDPs. In IJCAI , pages 1025{1032, 2003.
Alec Radford, Jerey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI ,
pages 2586{2591, 2007.
Stephanie Rosenthal and Manuela Veloso. Modeling humans as observation providers using
pomdps. In 2011 RO-MAN , pages 53{58. IEEE, 2011.
Stuart Russell. Of myths and moonshine, 2014. URL https://www.edge.org/
conversation/the-myth-of-ai#26015 .
BIBLIOGRAPHY 105
Stuart Russell. Human Compatible: Articial Intelligence and the Problem of Control .
Penguin, 2019.
Dorsa Sadigh, S Shankar Sastry, Sanjit A Seshia, and Anca Dragan. Information gathering
actions over human internal state. In 2016 IEEE/RSJ International Conference on
Intelligent Robots and Systems (IROS) , pages 66{73. IEEE, 2016.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based
learning of reward functions. In Robotics: Science and Systems , 2017.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function
approximators. In International Conference on Machine Learning , pages 1312{1320, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal
policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.
Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca D Dragan. On the feasibility of learning,
rather than assuming, human biases for reward inference. arXiv preprint arXiv:1906.09624 ,
2019a.
Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander, Pieter Abbeel, and Anca Dragan.
Preferences implicit in the state of the world. In International Conference on Learning
Representations , 2019b.
Archit Sharma, Shane Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-
aware unsupervised skill discovery. In International Conference on Learning Representations
(ICLR) , 2020.
Jason F Shogren, John A List, and Dermot J Hayes. Preference learning in consecutive
experimental auctions. American Journal of Agricultural Economics , 82(4):1016{1021,
2000.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur
Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the
game of go without human knowledge. nature , 550(7676):354{359, 2017.
Nisan Stiennon, Long Ouyang, Je Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec
Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback.
arXiv preprint arXiv:2009.01325 , 2020.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation
learning from reinforcement learning. arXiv preprint arXiv:2009.08319 , 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press,
2018.
BIBLIOGRAPHY 106
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based
control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems ,
2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from
observation. arXiv preprint arXiv:1807.06158 , 2018.
Alexander Matt Turner. Optimal farsighted agents tend to seek power. arXiv preprint
arXiv:1912.01683 , 2019.
Alexander Matt Turner, Dylan Hadeld-Menell, and Prasad Tadepalli. Conservative agency
via attainable utility preservation. In Proceedings of the AAAI/ACM Conference on AI,
Ethics, and Society , pages 385{391, 2020.
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Woj-
ciech M. Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo
Ewalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Agapiou, Junhyuk Oh, Valentin
Dalibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor
Cai, David Budden, Tom Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfa, Toby Pohlen,
Yuhuai Wu, Dani Yogatama, Julia Cohen, Katrina McKinney, Oliver Smith, Tom Schaul,
Timothy Lillicrap, Chris Apps, Koray Kavukcuoglu, Demis Hassabis, and David Silver.
Alphastar: Mastering the real-time strategy game StarCraft II. https://deepmind.com/
blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/ , 2019.
Steven Wang, Sam Toyer, Adam Gleave, and Scott Emmons. The imitation library
for imitation learning and inverse reinforcement learning. https://github.com/
HumanCompatibleAI/imitation , 2020.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer:
Interactive agent shaping in high-dimensional state spaces. arXiv preprint arXiv:1709.10163 ,
2017.
David Whitney, Eric Rosen, James MacGlashan, Lawson LS Wong, and Stefanie Tellex.
Reducing errors in object-fetching interactions through social feedback. In 2017 IEEE
International Conference on Robotics and Automation (ICRA) , pages 1006{1013. IEEE,
2017.
Nils Wilde, Dana Kulic, and Stephen L Smith. Active preference learning using maximum
regret. arXiv preprint arXiv:2005.04067 , 2020.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes F urnkranz. A survey of
preference-based reinforcement learning methods. Journal of Machine Learning Research ,
18(1):4945{4990, 2017.
Mark Woodward, Chelsea Finn, and Karol Hausman. Learning to interactively learn and
assist. arXiv preprint arXiv:1906.10187 , 2019.
BIBLIOGRAPHY 107
Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell,
Stefano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep
learning to understand economic well-being in africa. Nature communications , 11(1):1{11,
2020.
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and
Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-
learning. arXiv preprint arXiv:1802.01557 , 2018.
Eliezer Yudkowsky. Problem of fully updated deference, year unknown. URL https:
//arbital.com/p/updated\_deference/ .
Shun Zhang, Edmund Durfee, and Satinder Singh. Approximately-optimal queries for planning
in reward-uncertain markov decision processes. In Twenty-Seventh International Conference
on Automated Planning and Scheduling , 2017.
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle
of maximum causal entropy. 2010.