Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior

Where Do YouThink You’re Going?:
Inferring Beliefs about Dynamics from Behavior
Siddharth Reddy, Anca D. Dragan, Sergey Levine
Department of Electrical Engineering and Computer Science
University of California, Berkeley
{sgr,anca,svlevine}@berkeley.edu
Abstract
Inferring intent from observed behavior has been studied extensively within the
frameworks of Bayesian inverse planning and inverse reinforcement learning.
These methods infer a goal or reward function that best explains the actions of
the observed agent, typically a human demonstrator. Another agent can use this
inferred intent to predict, imitate, or assist the human user. However, a central
assumption in inverse reinforcement learning is that the demonstrator is close to
optimal. While models of suboptimal behavior exist, they typically assume that
suboptimal actions are the result of some type of random noise or a known cognitive
bias, like temporal inconsistency. In this paper, we take an alternative approach,
and model suboptimal behavior as the result of internal model misspeciﬁcation: the
reason that user actions might deviate from near-optimal actions is that the user has
an incorrect set of beliefs about the rules – the dynamics – governing how actions
affect the environment. Our insight is that while demonstrated actions may be
suboptimal in the real world, they may actually be near-optimal with respect to the
user’s internal model of the dynamics. By estimating these internal beliefs from
observed behavior, we arrive at a new method for inferring intent. We demonstrate
in simulation and in a user study with 12 participants that this approach enables us
to more accurately model human intent, and can be used in a variety of applications,
including offering assistance in a shared autonomy framework and inferring human
preferences.
1 Introduction
Characterizing the drive behind human actions in the form of a goal or reward function is broadly
useful for predicting future behavior, imitating human actions in new situations, and augmenting
human control with automated assistance – critical functions in a wide variety of applications, in-
cluding pedestrian motion prediction [ 57], virtual character animation [ 38], and robotic teleoperation
[35]. For example, remotely operating a robotic arm to grasp objects can be challenging for a human
user due to unfamiliar or unintuitive dynamics of the physical system and control interface. Existing
frameworks for assistive teleoperation and shared autonomy aim to help users perform such tasks
[35,29,46,8,45]. These frameworks typically rely on existing methods for intent inference in the
sequential decision-making context, which use Bayesian inverse planning or inverse reinforcement
learning to learn the user’s goal or reward function from observed control demonstrations. These
methods typically assume that user actions are near-optimal, and deviate from optimality due to
random noise [56], speciﬁc cognitive biases in planning [16, 15, 4], or risk sensitivity [33].
Seehttps://sites.google.com/view/inferring-internal-dynamics for supplementary materi-
als, including videos and code.
32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada.
The key insight in this paper is that suboptimal behavior can also arise from a mismatch between the
dynamics of the real world and the user’s internal beliefs of the dynamics, and that a user policy that
appears suboptimal in the real world may actually be near-optimal with respect to the user’s internal
dynamics model. As resource-bounded agents living in an environment of dazzling complexity,
humans rely on intuitive theories of the world to guide reasoning and planning [ 21,26]. Humans
leverage internal models of the world for motor control [ 53,30,14,34,49], goal-directed decision
making [ 7], and representing the mental states of other agents [ 39]. Simpliﬁed internal models can
systematically deviate from the real world, leading to suboptimal behaviors that have unintended
consequences, like hitting a tennis ball into the net or skidding on an icy road. For example, a classic
study in cognitive science shows that human judgments about the physics of projectile motion are
closer to Aristotelian impetus theory than to true Newtonian dynamics – in other words, people tend
to ignore or underestimate the effects of inertia [ 11]. Characterizing the gap between internal models
and reality by modeling a user’s internal predictions of the effects of their actions allows us to better
explain observed user actions and infer their intent.
The main contribution of this paper is a new algorithm for intent inference that ﬁrst estimates a
user’s internal beliefs of the dynamics of the world using observations of how they act to perform
known tasks, then leverages the learned internal dynamics model to infer intent on unknown tasks.
In contrast to the closest prior work [28, 22], our method scales to problems with high-dimensional,
continuous state spaces and nonlinear dynamics. Our internal dynamics model estimation algorithm
assumes the user takes actions with probability proportional to their exponentiated soft Q-values. We
ﬁt the parameters of the internal dynamics model to maximize the likelihood of observed user actions
on a set of tasks with known reward functions, by tying the internal dynamics to the soft Q function
via the soft Bellman equation. At test time, we use the learned internal dynamics model to predict the
user’s desired next state given their current state and action input.
We run experiments ﬁrst with simulated users, testing that we can recover the internal dynamics, even
in MDPs with a continuous state space that would otherwise be intractable for prior methods. We
then run a user study with 12 participants in which humans play the Lunar Lander game (screenshot
in Figure 1). We recover a dynamics model that explains user actions better than the real dynamics,
which in turn enables us to assist users in playing the game by transferring their control policy from
the recovered internal dynamics to the real dynamics.
2 Background
Inferring intent in sequential decision-making problems has been heavily studied under the framework
of inverse reinforcement learning (IRL), which we build on in this work. The aim of IRL is to learn
a user’s reward function from observed control demonstrations. IRL algorithms are not directly
applicable to our problem of learning a user’s beliefs about the dynamics of the environment, but they
provide a helpful starting point for thinking about how to extract hidden properties of a user from
observations of how they behave.
In our work, we build on the maximum causal entropy (MaxCausalEnt) IRL framework [ 55,6,44,36,
28]. In an MDP with a discrete action space A, the human demonstrator is assumed to follow a policy
that maximizes an entropy-regularized reward R(s;a;s0)under dynamics T(s0js;a). Equivalently,
(ajs),exp (Q(s;a))P
a02Aexp (Q(s;a0)); (1)
whereQis the softQfunction, which satisﬁes the soft Bellman equation [55],
Q(s;a) =Es0T(js;a)[R(s;a;s0) +
V(s0)]; (2)
withVthe soft value function,
V(s),log X
a2Aexp (Q(s;a))!
: (3)
Prior work assumes Tis the true dynamics of the real world, and ﬁts a model of the reward Rthat
maximizes the likelihood (given by Equation 1) of some observed demonstrations of the user acting
in the real world. In our work, we assume access to a set of training tasks for which the rewards R
are known, ﬁt a model of the internal dynamics Tthat is allowed to deviate from the real dynamics,
then use the recovered dynamics to infer intent (e.g., rewards) in new tasks.
2
3 Internal Dynamics Model Estimation
We split up the problem of intent inference into two parts: learning the internal dynamics model from
user demonstrations on known tasks (the topic of this section), and using the learned internal model
to infer intent on unknown tasks (discussed later in Section 4). We assume that the user’s internal
dynamics model is stationary, which is reasonable for problems like robotic teleoperation when the
user has some experience practicing with the system but still ﬁnds it unintuitive or difﬁcult to control.
We also assume that the real dynamics are known ex-ante or learned separately.
Our aim is to recover a user’s implicit beliefs about the dynamics of the world from observations
of how they act to perform a set of tasks. The key idea is that, when their internal dynamics model
deviates from the real dynamics, we can no longer simply ﬁt a dynamics model to observed state
transitions. Standard dynamics learning algorithms typically assume access to (s;a;s0)examples,
with (s;a)features and s0labels, that can be used to train a classiﬁcation or regression model
p(s0js;a)using supervised learning. In our setting, we instead have (s;a)pairs that indirectly encode
the state transitions that the user expected to happen, but did not necessarily occur, because the user’s
internal model predicted different outcomes s0than those that actually occurred in the real world.
Our core assumption is that the user’s policy is near-optimal with respect to the unknown internal
dynamics model. To this end, we propose a new algorithm for learning the internal dynamics from
action demonstrations: inverse soft Q-learning.
3.1 Inverse Soft Q-Learning
The key idea behind our algorithm is that we can ﬁt a parametric model of the internal dynamics
modelTthat maximizes the likelihood of observed action demonstrations on a set of training tasks
with known rewards by using the soft Qfunction as an intermediary.1We tie the internal dynamics
Tto the softQfunction via the soft Bellman equation (Equation 2), which ensures that the soft Q
function is induced by the internal dynamics T. We tie the soft Qfunction to action likelihoods using
Equation 1, which encourages the soft Qfunction to explain observed actions. We accomplish this
by solving a constrained optimization problem in which the demonstration likelihoods appear in the
objective and the soft Bellman equation appears in the constraints.
Formulating the optimization problem. Assume the action space Ais discrete.2Leti2
f1;2;:::;ngdenote the training task, Ri(s;a;s0)denote the known reward function for task i,T
denote the unknown internal dynamics, and Qidenote the unknown soft Qfunction for task i. We
representQiusing a function approximator Qiwith parameters i, and the internal dynamics using
a function approximator Tparameterized by . Note that, while each task merits a separate soft Q
function since each task has different rewards, all tasks share the same internal dynamics.
Recall the soft Bellman equation (Equation 2), which constrains Qito be the soft Qfunction for
rewardsRiand internal dynamics T. An equivalent way to express this condition is that Qisatisﬁes
i(s;a) = 08s;a, whereiis the soft Bellman error:
i(s;a),Qi(s;a)Z
s02ST(s0js;a) (Ri(s;a;s0) +
Vi(s0)) ds0: (4)
We impose the same condition on QiandT, i.e.,i;(s;a) = 08s;a. We assume our represen-
tations are expressive enough that there exist values of iandthat satisfy the condition. We ﬁt
parametersiandto maximize the likelihood of the observed demonstrations while respecting the
soft Bellman equation by solving the constrained optimization problem
minimize
fign
i=1;nX
i=1X
(s;a)2Ddemo
ilogi(ajs) (5)
subject toi;(s;a) = 08i2f1;2;:::;ng;s2S;a2A;
whereDdemo
iare the demonstrations for task i, andidenotes the action likelihood given by Qi
and Equation 1.
1Our algorithm can in principle learn from demonstrations even when the rewards are unknown, but in
practice we ﬁnd that this relaxation usually makes learning the correct internal dynamics too difﬁcult.
2We assume a discrete action space to simplify our exposition and experiments. Our algorithm can be
extended to handle MDPs with a continuous action space using existing sampling methods [25].
3
Solving the optimization problem. We use the penalty method [ 5] to approximately solve the
constrained optimization problem described in Equation 5, which recasts the problem as unconstrained
optimization of the cost function
c(;),nX
i=1X
(s;a)2Ddemo
ilogi(ajs) +
2nX
i=1Z
s2SX
a2A(i;(s;a))2ds; (6)
whereis a constant hyperparameter, idenotes the action likelihood given by Qiand Equation 1,
andi;denotes the soft Bellman error, which relates QitoTthrough Equation 4.
For MDPs with a discrete state space S, we minimize the cost as is. MDPs with a continuous state
space present two challenges: (1) an intractable integral over states in the sum over penalty terms,
and (2) integrals over states in the expectation terms of the soft Bellman errors (recall Equation
4). To tackle (1), we resort to constraint sampling [ 10]; speciﬁcally, randomly sampling a subset of
state-action pairsDsamp
ifrom rollouts of a random policy in the real world. To tackle (2), we choose
a deterministic model of the internal dynamics T, which simpliﬁes the integral over next states in
Equation 4 to a single term3.
In our experiments, we minimize the objective in Equation 6 using Adam [ 31]. We use a mix of
tabular representations, structured linear models, and relatively shallow multi-layer perceptrons to
modelQiandT. In the tabular setting, iis a table of numbers with a separate entry for each
state-action pair, and can be a table with an entry between 0 and 1 for each state-action-state triple.
For linear and neural network representations, iandare sets of weights.
3.2 Regularizing the Internal Dynamics Model
One issue with our approach to estimating the internal dynamics is that there tend to be multiple
feasible internal dynamics models that explain the demonstration data equally well, which makes
the correct internal dynamics model difﬁcult to identify. We propose two different solutions to this
problem: collecting demonstrations on multiple training tasks, and imposing a prior on the learned
internal dynamics that encourages it to be similar to the real dynamics.
Multiple training tasks. If we only collect demonstrations on n= 1training tasks, then at any given
statesand actiona, the recovered internal dynamics may simply assign a likelihood of one to the
next states0that maximizes the reward function R1(s;a;s0)of the single training task. Intuitively, if
our algorithm is given user demonstrations on only one task, then the user’s actions can be explained
by an internal dynamics model that always predicts the best possible next state for that one task
(e.g., the target in a navigation task), no matter the current state or user action. We can mitigate this
problem by collecting demonstrations on n>1training tasks, which prevents degenerate solutions
by forcing the internal dynamics to be consistent with a diverse set of user policies.
Action intent prior. In our experiments, we also explore another way to regularize the learned
internal dynamics: imposing the prior that the learned internal dynamics Tshould be similar to the
known real dynamics Trealby restricting the support of T(js;a)to statess0that are reachable in the
real dynamics. Formally,
T(s0js;a),X
aint2ATreal(s0js;aint)f(aintjs;a) (7)
whereais the user’s action, aintis the user’s intended action, and f:SA2![0;1]captures
the user’s ‘action intent’ – the action they would have taken if they had perfect knowledge of the
real dynamics. This prior changes the structure of our internal dynamics model to predict the user’s
intended action with respect to the real dynamics, rather than directly predicting their intended next
state. Note that, when we use this action intent prior, Tis no longer directly modeled. Instead, we
modelfand use Equation 7 to compute T.
In our experiments, we examine the effects of employing multiple training tasks and imposing the
action intent prior, together and in isolation.
3Another potential solution is sampling states to compute a Monte Carlo estimate of the integral.
4
Figure 1: A high-level schematic of our internal-to-real dynamics transfer algorithm for shared autonomy,
which uses the internal dynamics model learned by our method to assist the user with an unknown control task;
in this case, landing the lunar lander between the ﬂags. The user’s actions are assumed to be consistent with
their internal beliefs about the dynamics T, which differ from the real dynamics Treal. Our system models the
internal dynamics to determine where the user is trying to go next, then acts to get there.
4 Using Learned Internal Dynamics Models
The ability to learn internal dynamics models from demonstrations is broadly useful for intent
inference. In our experiments, we explore two applications: (1) shared autonomy, in which a human
and robot collaborate to solve a challenging real-time control task, and (2) learning the reward
function of a user who generates suboptimal demonstrations due to internal model misspeciﬁcation.
In (1), intent is formalized as the user’s desired next state, while in (2), the user’s intent is represented
by their reward function.
4.1 Shared Autonomy via Internal-to-Real Dynamics Transfer
Many control problems involving human users are challenging for autonomous agents due to partial
observability and imprecise task speciﬁcations, and are also challenging for humans due to constraints
such as bounded rationality [ 48] and physical reaction time. Shared autonomy combines human and
machine intelligence to perform control tasks that neither can on their own, but existing methods
have the basic requirement that the machine either needs a description of the task or feedback from
the user, e.g., in the form of rewards [ 29,8,45]. We propose an alternative algorithm that assists the
user without knowing their reward function by leveraging the internal dynamics model learned by our
method. The key idea is formalizing the user’s intent as their desired next state. We use the learned
internal dynamics model to infer the user’s desired next state given their current state and control
input, then execute an action that will take the user to the desired state under the real dynamics;
essentially, transferring the user’s policy from the internal dynamics to the real dynamics, akin to
simulation-to-real transfer for robotic control [ 13]. See Figure 1 for a high-level schematic of this
process.
Equipped with the learned internal dynamics model T, we perform internal-to-real dynamics transfer
by observing the user’s action input, computing the induced distribution over next states using the
internal dynamics, and executing an action that induces a similar distribution over next states in the
real dynamics. Formally, for user control input ah
tand statest, we execute action at, where
at,arg min
a2ADKL(T(st+1jst;ah
t)kTreal(st+1jst;a)) (8)
4.2 Learning Rewards from Misguided User Demonstrations
Most existing inverse reinforcement learning algorithms assume that the user’s internal dynamics are
equivalent to the real dynamics, and learn their reward function from near-optimal demonstrations.
We explore a more realistic setting in which the user’s demonstrations are suboptimal due to a
mismatch between their internal dynamics and the real dynamics. Users are ‘misguided’ in that their
behavior is suboptimal in the real world, but near-optimal with respect to their internal dynamics.
In this setting, standard IRL algorithms that do not distinguish between the internal and the real
dynamics learn incorrect reward functions. Our method can be used to learn the internal dynamics,
then explicitly incorporate the learned internal dynamics into an IRL algorithm’s behavioral model of
the user.
5
0 20 40
Number of Training Tasks0.000.250.500.751.00Internal Dynamics Accuracy
Grid World Navigation
Random
Action Intent Prior
No Prior
0 2000 4000 6000 8000 10000
Number of Gradient Steps0.00.51.01.52.0Internal Dynamics L2 Error2D Continuous-State Navigation
Our Method
Random
0 10 20 30 40 50
Number of Gradient Steps−0.50.00.51.0True RewardLearning Rewards from Misguided Demos
IRL + Our Method
SERD (Baseline)
Random Policy
MaxCausalEnt IRL (Baseline)Figure 2: Left, Center: Error bars show standard error on ten random seeds. Our method learns accurate
internal dynamics models, the regularization methods in Section 3.2 increase accuracy, and the approximations
for continuous-state MDPs in Section 3.1 do not compromise accuracy. Right: Error regions show standard
error on ten random tasks and ten random seeds each. Our method learns an internal dynamics model that
enables MaxCausalEnt IRL to learn rewards from misguided user demonstrations.
In our experiments, we instantiate prior work with MaxCausalEnt IRL [ 55], which inverts the
behavioral model from Equation 1 to infer rewards from demonstrations. We adapt it to our setting,
in which the real dynamics are known and the internal dynamics are either learned (separately by our
algorithm) or assumed to be the same as the known real dynamics. MaxCausalEnt IRL cannot learn
the user’s reward function from misguided demonstrations when it makes the standard assumption
that the internal dynamics are equal to the real dynamics, but can learn accurate rewards when it
instead uses the learned internal dynamics model produced by our algorithm.
5 User Study and Simulation Experiments
The purpose of our experiments is two-fold: (1) to test the correctness of our algorithm, and (2)
to test our core assumption that a human user’s internal dynamics can be different from the real
dynamics, and that our algorithm can learn an internal dynamics model that is useful for assisting
the user through internal-to-real dynamics transfer. To accomplish (1), we perform three simulated
experiments that apply our method to shared autonomy (see Section 4.1) and to learning rewards
from misguided user demonstrations (see Section 4.2). In the shared autonomy experiments, we
ﬁrst use a tabular grid world navigation task to sanity-check our algorithm and analyze the effects of
different regularization choices from Section 3.2. We then use a continuous-state 2D navigation task
to test our method’s ability to handle continuous observations using the approximations described in
Section 3.1. In the reward learning experiment, we use the grid world environment to compare the
performance of MaxCausalEnt IRL [ 55] when it assumes the internal dynamics are the same as the
real dynamics to when it uses the internal dynamics learned by our algorithm. To accomplish (2), we
conduct a user study in which 12 participants play the Lunar Lander game (see Figure 1) with and
without internal-to-real dynamics transfer assistance. We summarize these experiments in Sections
5.1 and 5.2. Further details are provided in Section 9.1 of the appendix.
5.1 Simulation Experiments
Shared autonomy. The grid world provides us with a domain where exact solutions are tractable,
which enables us to verify the correctness of our method and compare the quality of the approximation
in Section 3.1 with an exact solution to the learning problem. The continuous task provides a more
challenging domain where exact solutions via dynamic programming are intractable. In each setting,
we simulate a user with an internal dynamics model that is severely biased away from the real
dynamics of the simulated environment. The simulated user’s policy is near-optimal with respect to
their internal dynamics, but suboptimal with respect to the real dynamics. Figure 2 (left and center
plots) provides overall support for the hypothesis that our method can effectively learn tabular and
continuous representations of the internal dynamics for MDPs with discrete and continuous state
spaces. The learned internal dynamics models are accurate with respect to the ground truth internal
dynamics, and internal-to-real dynamics transfer successfully assists the simulated users. The learned
internal dynamics model becomes more accurate as we increase the number of training tasks, and the
action intent prior (see Section 3.2) increases accuracy when the internal dynamics are similar to the
real dynamics. These results conﬁrm that our approximate algorithm is correct and yields solutions
6
0.00 0.25 0.50 0.75 1.00
Crash Rate0.000.250.500.751.00Success RateLunar Lander User Study (12 users)
Unassisted
Assisted
0.00 0.25 0.50 0.75 1.00
Unassisted Success Rate0.000.250.500.751.00Assisted Success RateLunar Lander User Study (12 users)
0.02 0.03 0.04 0.05
Game Speed0.00.10.2LikelihoodLunar Lander User Study
Real Dynamics
Internal Dynamics
Figure 3: Human users ﬁnd the default game environment – the real dynamics – to be difﬁcult and unintuitive,
as indicated by their poor performance in the unassisted condition (top center and right plots) and their subjective
evaluations (in Table 1). Our method observes suboptimal human play in the default environment, learns a setting
of the game physics under which the observed human play would have been closer to optimal, then performs
internal-to-real dynamics transfer to assist human users in achieving higher success rates and lower crash rates
(top center and right plots). The learned internal dynamics has a slower game speed than the real dynamics
(bottom left plot). The bottom center and right plots show successful (green) and failed (red) trajectories in the
unassisted and assisted conditions.
that do not signiﬁcantly deviate from those of an exact algorithm. Further results and experimental
details are discussed in Sections 9.1.1 and 9.1.2 of the appendix.
Learning rewards from misguided user demonstrations. Standard IRL algorithms, such as Max-
CausalEnt IRL [ 55], can fail to learn rewards from user demonstrations that are ‘misguided’, i.e.,
systematically suboptimal in the real world but near-optimal with respect to the user’s internal dy-
namics. Our algorithm can learn the internal dynamics model, and we can then explicitly incorporate
the learned internal dynamics into the MaxCausalEnt IRL algorithm to learn accurate rewards from
misguided demonstrations. We assess this method on a simulated grid world navigation task. Figure 2
(right plot) supports our claim that standard IRL is ineffective at learning rewards from misguided user
demonstrations. After using our algorithm to learn the internal dynamics and explicitly incorporating
the learned internal dynamics into an IRL algorithm’s model of the user, we see that it’s possible
to recover accurate rewards from these misguided demonstrations. Additional information on our
experimental setup is available in Section 9.1.3 of the appendix.
In addition to comparing to the standard MaxCausalEnt IRL baseline, we also conducted a comparison
(shown in Figure 2) with a variant of the Simultaneous Estimation of Rewards and Dynamics (SERD)
algorithm [ 28] that simultaneously learns rewards and the internal dynamics instead of assuming
that the internal dynamics are equivalent to the real dynamics. This baseline performs better than
random, but still much worse than our method. This result is supported by the theoretical analysis
in Armstrong et al. [ 2], which characterizes the difﬁculty of simultaneously deducing a human’s
rationality – in our case, their internal dynamics model – and their rewards from demonstrations.
5.2 User Study on the Lunar Lander Game
Our previous experiments were conducted with simulated expert behavior, which allowed us to
control the corruption of the internal dynamics. However, it remains to be seen whether this model of
suboptimality effectively reﬂects real human behavior. We test this hypothesis in the next experiment,
which evaluates whether our method can learn the internal dynamics accurately enough to assist real
users through internal-to-real dynamics transfer.
Task description. We use the Lunar Lander game from OpenAI Gym [ 9] (screenshot in Figure 1)
to evaluate our algorithm with human users. The objective of the game is to land on the ground,
7
Table 1: Subjective evaluations of the Lunar Lander user study from 12 participants. Means reported below
for responses on a 7-point Likert scale, where 1 = Strongly Disagree, 4 = Neither Disagree nor Agree, and 7 =
Strongly Agree. p-values from a one-way repeated measures ANOV A with the presence of assistance as a factor
inﬂuencing responses.
p-value Unassisted Assisted
I enjoyed playing the game <:001 3.92 5.92
I improved over time <:0001 3.08 5.83
I didn’t crash <:001 1.17 3.00
I didn’t ﬂy out of bounds <:05 1.67 3.08
I didn’t run out of time >:05 5.17 6.17
I landed between the ﬂags <:001 1.92 4.00
I understood how to complete the task <:05 6.42 6.75
I intuitively understood the physics of the game <:01 4.58 6.00
My actions were carried out >:05 4.83 5.50
My intended actions were carried out <:01 2.75 5.25
without crashing or ﬂying out of bounds, using two lateral thrusters and a main engine. The action
spaceAconsists of six discrete actions. The state s2R9encodes position, velocity, orientation, and
the location of the landing site, which is one of nine values corresponding to n= 9distinct tasks.
The physics of the game are forward-simulated by a black-box function that takes as input seven
hyperparameters, which include engine power and game speed. We manipulate whether or not the
user receives internal-to-real dynamics transfer assistance using an internal dynamics model trained
on their unassisted demonstrations. The dependent measures are the success and crash rates in each
condition. The task and evaluation protocol are discussed further in Section 9.2 of the appendix.
Analysis. In the default environment, users appear to play as though they underestimate the strength
of gravity, which causes them to crash into the ground frequently (see the supplementary videos).
Figure 3 (bottom left plot) shows that our algorithm learns an internal dynamics model characterized
by a slower game speed than the real dynamics, which makes sense since a slower game speed
induces smaller forces and slower motion – conditions under which the users’ action demonstrations
would have been closer to optimal. These results support our claim that our algorithm can learn an
internal dynamics model that explains user actions better than the real dynamics.
When unassisted, users often crash or ﬂy out of bounds due to the unintuitive nature of the thruster
controls and the relatively fast pace of the game. Figure 3 (top center and right plots) shows that users
succeed signiﬁcantly more often and crash signiﬁcantly less often when assisted by internal-to-real
dynamics transfer (see Section 9.2 of the appendix for hypothesis tests). The assistance makes
the system feel easier to control (see the subjective evaluations in Table 1 of the appendix), less
likely to tip over (see the supplementary videos), and move more slowly in response to user actions
(assistance led to a 30% decrease in average speed). One of the key advantages of assistance was its
positive effect on the rate at which users were able to switch between different actions: on average,
unassisted users performed 18 actions per minute (APM), while assisted users performed 84 APM.
Quickly switching between ﬁring various thrusters enabled assisted users to better stabilize ﬂight.
These results demonstrate that the learned internal dynamics can be used to effectively assist the
user through internal-to-real dynamics transfer, which in turn gives us conﬁdence in the accuracy
of the learned internal dynamics. After all, we cannot measure the accuracy of the learned internal
dynamics by comparing it to the ground truth internal dynamics, which is unknown for human users.
6 Related Work
The closest prior work in intent inference and action understanding comes from inverse planning [ 3]
and inverse reinforcement learning [ 37], which use observations of a user’s actions to estimate the
user’s goal or reward function. We take a fundamentally different approach to intent inference: using
action observations to estimate the user’s beliefs about the world dynamics.
The simultaneous estimation of rewards and dynamics (SERD) instantiation of MaxCausalEnt IRL
[28] aims to improve the sample efﬁciency of IRL by forcing the learned real dynamics model
to explain observed state transitions as well as actions. The framework includes terms for the
8
demonstrator’s beliefs of the dynamics, but the overall algorithm and experiments of Herman et al.
[28] constrain those beliefs to be the same as the real dynamics. Our goal is to learn an internal
dynamics model that may deviate from the real dynamics. To this end, we propose two new internal
dynamics regularization techniques, multi-task training and the action intent prior (see Section 3.2),
and demonstrate their utility for learning an internal dynamics model that differs from the real
dynamics (see Section 5.1). We also conduct a user experiment that shows human actions in a game
environment can be better explained by a learned internal dynamics model than by the real dynamics,
and that augmenting user control with internal-to-real dynamics transfer results in improved game
play. Furthermore, the SERD algorithm is well-suited to MDPs with a discrete state space, but
intractable for continuous state spaces. Our method can be applied to MDPs with a continuous state
space, as shown in Sections 5.1 and 5.2.
Golub et al. [ 22] propose an internal model estimation (IME) framework for brain-machine interface
(BMI) control that learns an internal dynamics model from control demonstrations on tasks with
linear-Gaussian dynamics and quadratic reward functions. Our work is (1) more general in that it
places no restrictions on the functional form of the dynamics or the reward function, and (2) does not
assume sensory feedback delay, which is the fundamental premise of using IME for BMI control.
Rafferty et al. [ 43,41,42] use an internal dynamics learning algorithm to infer a student’s incorrect
beliefs in online learning settings like educational games, and leverage the inferred beliefs to generate
personalized hints and feedback. Our algorithm is more general in that it is capable of learning
continuous parameters of the internal dynamics, whereas the cited work is only capable of identifying
the internal dynamics given a discrete set of candidate models.
Modeling human error has a rich history in the behavioral sciences. Procrastination and other time-
inconsistent human behaviors have been characterized as rational with respect to a cost model that
discounts the cost of future action relative to that of immediate action [ 1,32]. Systematic errors
in human predictions about the future have been partially explained by cognitive biases like the
availability heuristic and regression to the mean [ 50]. Imperfect intuitive physics judgments have
been characterized as approximate probabilistic inferences made by a resource-bounded observer [ 26].
We take an orthogonal approach in which we assume that suboptimal behavior is primarily caused by
incorrect beliefs of the dynamics, rather than uncertainty or biases in planning and judgment.
Humans are resource-bounded agents that must take into account the computational cost of their
planning algorithm when selecting actions [ 24]. One way to trade-off the ability to ﬁnd high-value
actions for lower computational cost is to plan using a simpliﬁed, low-dimensional model of the
dynamics [ 27,19]. Evidence from the cognitive science literature suggests humans ﬁnd it difﬁcult
to predict the motion of objects when multiple information dimensions are involved [ 40]. Thus, we
arrive at an alternative explanation for why humans may behave near-optimally with respect to a
dynamics model that differs from the real dynamics: even if users have perfect knowledge of the
real dynamics, they may not have the computational resources to plan under the real dynamics, and
instead choose to plan using a simpliﬁed model.
7 Discussion
Limitations. Although our algorithm models the soft Qfunction with arbitrary neural network
parameterizations, the internal dynamics parameterizations we use are smaller, with at most seven
parameters for continuous tasks. Increasing the number of dynamics parameters would require a
better approach to regularization than those proposed in Section 3.2.
Summary. We contribute an algorithm that learns a user’s implicit beliefs about the dynamics of the
environment from demonstrations of their suboptimal behavior in the real environment. Simulation
experiments and a small-scale user study demonstrate the effectiveness of our method at recovering a
dynamics model that explains human actions, as well as its utility for applications in shared autonomy
and inverse reinforcement learning.
Future work. The ability to learn internal dynamics models from demonstrations opens the door to
new directions of scientiﬁc inquiry, like estimating young children’s intuitive theories of physics and
psychology without eliciting verbal judgments [ 52,18,23]. It also enables applications that involve
intent inference, including adaptive brain-computer interfaces for prosthetic limbs [ 12,47] that help
users perform control tasks that are difﬁcult to fully specify.
9
8 Acknowledgements
We would like to thank Oleg Klimov for open-sourcing his implementation of the Lunar Lander
game, which was originally developed by Atari in 1979, and inspired by the lunar modules built in
the 1960s and 70s for the Apollo space program. We would also like to thank Eliezer Yudkowsky for
the fanﬁction novel, Harry Potter and the Methods of Rationality – Harry’s misadventure with the
rocket-assisted broomstick in chapter 59 inspired us to try to close the gap between intuitive physics
and the real world. This work was supported in part by a Berkeley EECS Department Fellowship for
ﬁrst-year Ph.D. students, Berkeley DeepDrive, computational resource donations from Amazon, NSF
IIS-1700696, and AFOSR FA9550-17-1-0308.
References
[1] George A Akerlof. Procrastination and obedience. The American Economic Review , 81(2):1–19, 1991.
[2]Stuart Armstrong and Sören Mindermann. Impossibility of deducing preferences and rationality from
human policy. arXiv preprint arXiv:1712.05812 , 2017.
[3]Chris L Baker, Rebecca Saxe, and Joshua B Tenenbaum. Action understanding as inverse planning.
Cognition , 113(3):329–349, 2009.
[4]Leon Bergen, Owain Evans, and Joshua Tenenbaum. Learning structured preferences. In Proceedings of
the Annual Meeting of the Cognitive Science Society , volume 32, 2010.
[5] Dimitri P Bertsekas. Constrained optimization and Lagrange multiplier methods . Academic press, 2014.
[6]Michael Bloem and Nicholas Bambos. Inﬁnite time horizon maximum causal entropy inverse reinforcement
learning. In Decision and Control (CDC), 2014 IEEE 53rd Annual Conference on , pages 4911–4916.
IEEE, 2014.
[7]Matthew Botvinick and James An. Goal-directed decision making in prefrontal cortex: a computational
framework. In Advances in neural information processing systems , pages 169–176, 2009.
[8]Alexander Broad, TD Murphey, and Brenna Argall. Learning models for shared control of human-machine
systems with unknown dynamics. Robotics: Science and Systems Proceedings , 2017.
[9]Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and
Wojciech Zaremba. Openai gym, 2016.
[10] Giuseppe Calaﬁore and Fabrizio Dabbene. Probabilistic and randomized methods for design under
uncertainty . Springer, 2006.
[11] Alfonso Caramazza, Michael McCloskey, and Bert Green. Naive beliefs in “sophisticated” subjects:
Misconceptions about trajectories of objects. Cognition , 9(2):117–123, 1981.
[12] Jose M Carmena. Advances in neuroprosthetic learning and control. PLoS biology , 11(5):e1001561, 2013.
[13] Mark Cutler and Jonathan P How. Efﬁcient reinforcement learning for robots using informative simulated
priors. In Robotics and Automation (ICRA), 2015 IEEE International Conference on , pages 2605–2612.
IEEE, 2015.
[14] Michel Desmurget and Scott Grafton. Forward modeling allows feedback control for fast reaching
movements. Trends in cognitive sciences , 4(11):423–431, 2000.
[15] Owain Evans and Noah D Goodman. Learning the preferences of bounded agents. In NIPS Workshop on
Bounded Optimality , volume 6, 2015.
[16] Owain Evans, Andreas Stuhlmüller, and Noah D Goodman. Learning the preferences of ignorant, inconsis-
tent agents. In AAAI , pages 323–329, 2016.
[17] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via
policy optimization. In International Conference on Machine Learning , pages 49–58, 2016.
[18] Jerry A Fodor. A theory of the child’s theory of mind. Cognition , 1992.
[19] David Fridovich-Keil, Sylvia L Herbert, Jaime F Fisac, Sampada Deglurkar, and Claire J Tomlin. Plan-
ning, fast and slow: A framework for adaptive real-time safe trajectory planning. arXiv preprint
arXiv:1710.04731 , 2017.
[20] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement
learning. arXiv preprint arXiv:1710.11248 , 2017.
[21] Tobias Gerstenberg and Joshua B Tenenbaum. Intuitive theories. Oxford handbook of causal reasoning ,
pages 515–548, 2017.
10
[22] Matthew Golub, Steven Chase, and M Yu Byron. Learning an internal dynamics model from control
demonstration. In Proceedings of the 30th International Conference on Machine Learning (ICML-13) ,
pages 606–614, 2013.
[23] Alison Gopnik and Henry M Wellman. The theory theory. Mapping the mind: Domain speciﬁcity in
cognition and culture , page 257, 1994.
[24] Thomas L Grifﬁths, Falk Lieder, and Noah D Goodman. Rational use of cognitive resources: Levels of
analysis between the computational and the algorithmic. Topics in cognitive science , 7(2):217–229, 2015.
[25] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep
energy-based policies. arXiv preprint arXiv:1702.08165 , 2017.
[26] Jessica Hamrick, Peter Battaglia, and Joshua B Tenenbaum. Internal physics models guide probabilistic
judgments about object dynamics. In Proceedings of the 33rd annual conference of the cognitive science
society , pages 1545–1550. Cognitive Science Society Austin, TX, 2011.
[27] Sylvia L Herbert, Mo Chen, SooJean Han, Somil Bansal, Jaime F Fisac, and Claire J Tomlin. Fastrack: a
modular framework for fast and guaranteed safe motion planning. arXiv preprint arXiv:1703.07373 , 2017.
[28] Michael Herman, Tobias Gindele, Jörg Wagner, Felix Schmitt, and Wolfram Burgard. Inverse reinforcement
learning with simultaneous estimation of rewards and dynamics. In Artiﬁcial Intelligence and Statistics ,
pages 102–110, 2016.
[29] Shervin Javdani, Siddhartha S Srinivasa, and J Andrew Bagnell. Shared autonomy via hindsight optimiza-
tion. arXiv preprint arXiv:1503.07619 , 2015.
[30] Mitsuo Kawato. Internal models for motor control and trajectory planning. Current opinion in neurobiology ,
9(6):718–727, 1999.
[31] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980 , 2014.
[32] Jon Kleinberg and Sigal Oren. Time-inconsistent planning: a computational problem in behavioral
economics. In Proceedings of the ﬁfteenth ACM conference on Economics and computation , pages
547–564. ACM, 2014.
[33] Anirudha Majumdar, Sumeet Singh, Ajay Mandlekar, and Marco Pavone. Risk-sensitive inverse reinforce-
ment learning via coherent risk models. In Robotics: Science and Systems , 2017.
[34] Biren Mehta and Stefan Schaal. Forward models in visuomotor control. Journal of Neurophysiology ,
88(2):942–953, 2002.
[35] Katharina Muelling, Arun Venkatraman, Jean-Sebastien Valois, John E Downey, Jeffrey Weiss, Shervin
Javdani, Martial Hebert, Andrew B Schwartz, Jennifer L Collinger, and J Andrew Bagnell. Autonomy
infused teleoperation with application to brain computer interface controlled manipulation. Autonomous
Robots , pages 1–22, 2017.
[36] Gergely Neu and Csaba Szepesvári. Apprenticeship learning using inverse reinforcement learning and
gradient methods. arXiv preprint arXiv:1206.5264 , 2012.
[37] Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, pages
663–670, 2000.
[38] Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example-guided
deep reinforcement learning of physics-based character skills. arXiv preprint arXiv:1804.02717 , 2018.
[39] David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain
sciences , 1(4):515–526, 1978.
[40] Dennis R Profﬁtt and David L Gilden. Understanding natural dynamics. Journal of Experimental
Psychology: Human Perception and Performance , 15(2):384, 1989.
[41] Anna N Rafferty and Thomas L Grifﬁths. Diagnosing algebra understanding via inverse planning.
[42] Anna N Rafferty, Rachel Jansen, and Thomas L Grifﬁths. Using inverse planning for personalized feedback.
InEDM , pages 472–477, 2016.
[43] Anna N Rafferty, Michelle M LaMar, and Thomas L Grifﬁths. Inferring learners’ knowledge from their
actions. Cognitive Science , 39(3):584–618, 2015.
[44] Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. Urbana , 51(61801):1–4,
2007.
[45] Siddharth Reddy, Sergey Levine, and Anca Dragan. Shared autonomy via deep reinforcement learning.
arXiv preprint arXiv:1802.01744 , 2018.
[46] Wilko Schwarting, Javier Alonso-Mora, Liam Pauli, Sertac Karaman, and Daniela Rus. Parallel autonomy
in automated vehicles: Safe motion generation with minimal intervention. In Robotics and Automation
(ICRA), 2017 IEEE International Conference on , pages 1928–1935. IEEE, 2017.
11
[47] Krishna V Shenoy and Jose M Carmena. Combining decoder design and neural adaptation in brain-machine
interfaces. Neuron , 84(4):665–680, 2014.
[48] Herbert A Simon. Bounded rationality and organizational learning. Organization science , 2(1):125–134,
1991.
[49] Emanuel Todorov. Optimality principles in sensorimotor control. Nature neuroscience , 7(9):907, 2004.
[50] Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases. science ,
185(4157):1124–1131, 1974.
[51] Eiji Uchibe. Model-free deep inverse reinforcement learning by logistic regression. Neural Processing
Letters , pages 1–15, 2017.
[52] Friedrich Wilkening and Trix Cacchione. Children’s intuitive physics. The Wiley-Blackwell Handbook of
Childhood Cognitive Development, Second edition , pages 473–496, 2010.
[53] Daniel M Wolpert, Zoubin Ghahramani, and Michael I Jordan. An internal model for sensorimotor
integration. Science , 269(5232):1880–1882, 1995.
[54] Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforcement
learning. arXiv preprint arXiv:1507.04888 , 2015.
[55] Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum
causal entropy. In Proceedings of the 27th International Conference on International Conference on
Machine Learning , pages 1255–1262. Omnipress, 2010.
[56] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse
reinforcement learning. In AAAI , volume 8, pages 1433–1438. Chicago, IL, USA, 2008.
[57] Brian D Ziebart, Nathan Ratliff, Garratt Gallagher, Christoph Mertz, Kevin Peterson, J Andrew Bagnell,
Martial Hebert, Anind K Dey, and Siddhartha Srinivasa. Planning-based prediction for pedestrians. In
Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on , pages 3931–3936.
IEEE, 2009.
12


Negotiable Reinforcement Learning for Pareto
Optimal Sequential Decision-Making
Nishant Desai
Center for Human-Compatible AI
University of California, Berkeley
nishantdesai@berkeley.eduAndrew Critch
Department of EECS
University of California, Berkeley
critch@berkeley.edu
Stuart Russell
Computer Science Division
University of California, Berkeley
russell@cs.berkeley.edu
Abstract
It is commonly believed that an agent making decisions on behalf of two or more
principals who have different utility functions should adopt a Pareto optimal policy,
i.e. a policy that cannot be improved upon for one principal without making
sacriﬁces for another. Harsanyi’s theorem shows that when the principals have a
common prior on the outcome distributions of all policies, a Pareto optimal policy
for the agent is one that maximizes a ﬁxed, weighted linear combination of the
principals’ utilities. In this paper, we derive a more precise generalization for the
sequential decision setting in the case of principals with different priors on the
dynamics of the environment. We refer to this generalization as the Negotiable
Reinforcement Learning (NRL) framework. In this more general case, the relative
weight given to each principal’s utility should evolve over time according to how
well the agent’s observations conform with that principal’s prior. To gain insight
into the dynamics of this new framework, we implement a simple NRL agent and
empirically examine its behavior in a simple environment.
1 Introduction
It has been argued that the ﬁrst AI systems with generally super-human cognitive abilities will play a
pivotal decision-making role in directing the future of civilization [Bostrom, 2014]. If that is the case,
an important question will arise: Whose values will the ﬁrst super-human AI systems serve? Since
safety is a crucial consideration in developing such systems, assuming the institutions building them
come to understand the risks and the time investments needed to address them [Baum, 2016], they
will have a large incentive to cooperate in their design rather than racing under time-pressure to build
competing systems [Armstrong et al., 2016].
Therefore, consider two nations—allies or adversaries—who must decide whether to cooperate in the
deployment of an extremely powerful AI system. Implicitly or explicitly, the resulting system would
have to strike compromises when conﬂicts arise between the wishes of those nations. How can they
specify the degree to which that system would be governed by the distinctly held principles of each
nation? More mundanely, suppose a couple purchases a domestic robot. How should the robot strike
compromises when conﬂicts arise between the commands of its owners?
It is already an interesting and difﬁcult problem to robustly align an AI system’s values with those
of asingle human (or a group of humans in close agreement). Inverse reinforcement learning (IRL)
32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada.
[Russell, 1998] [Ng and Russell, 2000] [Abbeel and Ng, 2004] and cooperative inverse reinforcement
learning (CIRL) [Hadﬁeld-Menell et al., 2016] represent successively realistic early approaches to
this problem. But supposing some adequate solution eventually exists for aligning the values of a
machine intelligence with a single human decision-making unit, how should the values of a system
serving multiple decision-makers be “aligned”?
In the present work, we attempt to begin answering this question. We begin by observing some
deﬁciencies of optimizing a ﬁxed, linear weighted sum of principals’ utilities, as prescribed by
Harsanyi’s social aggregation theorem, in the case that those principals have differing beliefs about
the probability distributions dictating the agent’s observations. We show that building a decision
agent whose policy optimizes such an objective is not, in general, ex-ante Pareto optimal (i.e. "as
evaluated before the agent has taken any actions"). Intuitively, linear weighted aggregation fails
because, before the jointly owned agent has taken any actions, each principal evaluates its policy with
respect to their own beliefs, meaning that a policy that selectively prefers one principal over the other
conditioned on its observations can be desirable to both principals.
Section 3 addresses the shortcomings of Harsanyi-style preference aggregation by presenting the
Negotiable Reinforcement Learning (NRL) framework. In this domain, we model each principal’s
prior on the environment and utility function as a Partially Observable Markov Decision Process
(POMDP). We place necessary and sufﬁcient conditions on Pareto optimality for an agent acting over
these POMDPs with policy . We then construct a third POMDP and show that the optimal policy for
this single POMDP satisﬁes the conditions for Pareto optimality. We refer to an agent implementing
a policy that solves this reduced POMDP as a NRL agent.
Following directly from this reduction is the intriguing property that a Pareto optimal policy must,
over time, prefer the utility function of the principal whose beliefs are a better predictor of the agent’s
observations. This counter-intuitive result constitutes the main theorem of this paper. This can be
seen as settling a kind of bet between the two principals: whichever principal makes the correct
predictions gets to have their utility prioritized. In Section 4, we implement a simple NRL agent and
make empirical observations of this bet settling behavior.
2 Related work
Social choice theory. The entirety of social choice theory and voting theory may be viewed as
an attempt to specify an agreeable formal policy to enact on behalf of a group. Harsanyi’s utility
aggregation theorem [Harsanyi, 1980] suggests one form of solution: maximizing a linear combination
of group members’ utility functions. The present work shows that this solution is inappropriate when
principals have different beliefs, and Theorem 4 may be viewed as an extension of Harsanyi’s form
that accounts simultaneously for differing priors and the prospect of future observations. Indeed,
Harsanyi’s form follows as a direct corollary of Theorem 4 when principals do share the same beliefs.
Multi-agent systems. Zhang and Shah [2014] may be considered a sequential decision-making
approach to social choice: they use MDPs to represent the decisions of principals in a competitive
game, and exhibit an algorithm for the principals that, if followed, arrives at a Pareto optimal Nash
equilibrium satisfying a certain fairness criterion. Among the literature surveyed here, this paper
is the closest to the present work in terms of its intended application: roughly speaking, achieving
mutually desirable outcomes via sequential decision-making. However, the work is concerned with
an ongoing interaction between the principals, rather than selecting a policy for a single agent to
follow as in this paper.
Multi-objective sequential decision-making. There is also a good deal of work on Multi-
Objective Optimization (MOO) [Tzeng and Huang, 2011], including for sequential decision-making,
where solution methods have been called Multi-Objective Reinforcement Learning (MORL). For
instance, Gábor et al. [1998] introduce a MORL method called Pareto Q-learning for learning a set
of a Pareto optimal polices for a Multi-Objective MDP (MOMDP). Soh and Demiris [2011] deﬁne
Multi-Reward Partially Observable Markov Decision Processes (MR-POMDPs), and use genetic
algorithms to produce non-dominated sets of policies for them. Roijers et al. [2015] refer to the same
problems as Multi-objective POMDPS (MOPOMDPs), and provide a bounded approximation method
for the optimal solution set for all possible weightings of the objectives. Wang [2014] surveys MORL
2
methods, and contributes Multi-Objective Monte-Carlo Tree Search (MOMCTS) for discovering
multiple Pareto optimal solutions to a multi-objective optimization problem.
However, none of these or related works address scenarios where the objectives are derived from
principals with differing beliefs, from which the priority-shifting phenomenon of Theorem 4 arises.
Differing beliefs are likely to play a key role in negotiations, so for that purpose, the formulation of
multi-objective decision-making adopted here is preferable.
3 Negotiable Reinforcement Learning
Consider, informally, a scenario wherein two principals — perhaps individuals, companies, or states
— are considering cooperating to build or otherwise obtain a machine that will then interact with an
environment on their behalf.1In such a scenario, the principals will tend to bargain for “how much”
the machine will prioritize their separate interests, so to begin, we need some way to quantify “how
much” each principal is prioritized.
For instance, one might model the machine as maximizing the expected value, given its observations,
of some utility function Uof the environment that equals a weighted sum
w(1)U(1)+w(2)U(2)(1)
of the principals’ individual utility functions U(1)andU(2), as Harsanyi’s social aggregation theorem
recommends [Harsanyi, 1980]. Then the bargaining process could focus on choosing the values of
the weightsw(i).
However, this turns out to be a bad idea. As we shall see in the following example, this solution form
is not generally compatible with Pareto optimality when agents have different beliefs. Harsanyi’s
setting does not account for agents having different priors, nor for decisions being made sequentially,
after future observations. In such a setting, we need a new form of solution, exhibited here.
A cake-splitting scenario. Alice (principal 1) and Bob (principal 2) are about to be presented with
a cake which they can choose to split in half to share, or give entirely to one of them. They have
(built or purchased) a robot that will make the cake-splitting decision on their behalf. Alice’s utility
function returns 0if she gets no cake, 20if she gets half a cake, or 30if she gets a whole cake. Bob’s
utility function works similarly.
However, Alice and Bob have slightly different beliefs about how the environment works. They both
agree on the state of the environment that the robot will encounter at ﬁrst: a room with a cake in
it (s1=“cake” ). But Alice and Bob have different predictions about how the robot’s sensors will
perceive the cake: Alice thinks that when the robot perceives the cake, it is 90% likely to appear
with a red tint ( o1=“red" ), and 10% likely to appear with a green tint ( o1=“green" ), whereas Bob
believes the exact opposite. In either case, upon seeing the cake, the robot will either give Alice the
entire cake ( a1=(all, none) ), split the cake half-and-half ( a1=(half, half) ), or give Bob the entire
cake (a1=(none, all)). Moreover, Alice and Bob have common knowledge of all these facts.
Now, consider the following Pareto optimal policy that favors Alice (principal 1) when o1is red, and
Bob (principal 2) when o1is green:
^(jred) = 100% (all, none)
^(jgreen ) = 100% (none, all)
This policy can be viewed intuitively as a bet between Alice and Bob about the value of o1, and is
highly appealing to both principals:
E(1)[U(1); ^] = 90%(30) + 10%(0) = 27
E(2)[U(2); ^] = 10%(0) + 90%(30) = 27
In particular, ^is more appealing to both Alice and Bob than an agreement to deterministically split
the cake (half, half). We start to see that when principals evaluate their expected returns under a
policywith respect to differing beliefs about action outcomes, they may mutually agree to a policy
that favors one principal over the other, contingent on the action-observation history. We formalize
this intuition and explore its consequences in the following sections.
1The results here all generalize from two principals to nprincipals being combined successively in any order,
but for clarity of exposition, the two person case is prioritized.
3
3.1 A POMDP formulation
Let us formalize the machine’s decision-making situation using the structure of a Partially Observable
Markov Decision Process (POMDP) [Sondik, 1973]. It is assumed that the principals will have
common knowledge of the policy = (1;:::;n)they select for the machine to implement, but that
the principals may have different beliefs about how the environment works, and of course different
utility functions. We refer to this as the common knowledge assumption .
We encode each principal j’s outlook as a POMDP, D(j)= (S(j);A;T(j);U(j);O;
(j);n), which
simultaneously represents that principal’s beliefs about the environment, and the principal’s utility
function.
S(j)is a set of possible states sof the environment.
Ais the set of possible actions aavailable to the NRL agent.
T(j)is the conditional probabilities principal jbelieves will govern the environment state transitions,
i.e.,P(j)(si+1jsi;ai).
U(j)is principalj’s utility function from sequences of environmental states (s1;:::;sn)toR.2
Ois the set of possible observations oof the NRL agent.

(j)is the conditional probabilities principal jbelieves will govern the agent’s observations, i.e.,
P(j)(oijsi).
nis the time horizon.
Thus, principal j’s subjective probability of an outcome (s;o;a), for any s2(S(j))n, is given by a
probability distribution P(j)that takesas a parameter:
P(j)(s;o;a;) :=P(j)(s1)nY
i=1P(j)(oijsi)(aijoia<i)P(j)(si+1jsi;ai) (2)
We say that the POMDPs D(1)andD(2)arecompatible if any policy for one may be viewed as
a policy for the other, i.e., they have the same set of actions Aand observationsO, and the same
number of time steps n. By the common knowledge assumption, principals’ outlooks are assumed to
be encoded by a set of compatible POMDPs.
3.2 Pareto optimal policies
In this context, where a policy may be evaluated relative to more than one POMDP, we use
superscripts to represent which POMDP is governing the probabilities and expectations, e.g.,
E(j)
[U(j)] :=X
s2(S(j))nP(j)(s;)U(j)(s)
represents the expectation in D(j)of the utility function U(j), assuming policy is followed.
Deﬁnition 1 (Pareto optimal policies) .A policyisPareto optimal for a compatible pair of POMDPs
(D(1);D(2))if for any other policy 0, either
E(1)
[U(1)]E(1)
0[U(1)]orE(2)
[U(2)]E(2)
0[U(2)]:
It is assumed that, during negotiation, the principals will be seeking a Pareto optimal policy for the
agent to follow, relative to the POMDPs D(1)andD(2)describing each principal’s outlook.
If we allow the policy to come from the space of stochastic policies, mapping action-observation
histories to distributions over actions:
:
A;O
7!A,
then we can show that the following condition is necessary and sufﬁcient for to be Pareto optimal.
2For the sake of generality, U(j)isnot assumed to be stationary, as reward functions often are.
4
Lemma 2. A policyis Pareto optimal to principals 1and2if and only if there exist weights
w1;w20withw1+w2= 1such that
2argmax
2
w(1)E(1)
[U(1)] +w(2)E(2)
[U(2)]
(3)
Proof. See supplementary material.
3.3 Reduction to a single POMDP
We shall soon see that any Pareto optimal policy must favor, as time progresses, optimizing the utility
of whichever principal’s beliefs were a better predictor of the NRL agent’s inputs. This phenomenon
is most easily shown via a reduction of the POMDPs describing the outlooks of principals 1 and 2 to
a third POMDP, as follows.
For any weights, w(1);w(2)0withw(1)+w(2)= 1, we deﬁne a new POMDP that works by
ﬂipping a (w(1);w(2))-weighted coin, and then running D(1)orD(2)thereafter, according to the
coin ﬂip. Explicitly, we deﬁne a POMDP mixture as a POMDP D=w(1)D(1)+w(2)D(2). We give
Dthe same action space Aand observation space Oas the compatible POMDPs D(1)andD(2). In
the state space of D, we include a latent, stochastic, binary variable B2f1;2gthat is ﬁxed for all
time and initial belief given by P(B= 1) =w(1)andP(B= 2) =w(2). In the case that B= 1, we
draw states, transition probabilities, observation probabilities, and utilities from the parameters of D1.
Formally,
PD(s;o;ajB= 1;) =P(1)(s;o;a;),
and
E(D)
[UjB= 1] = E(1)
[U1].
Likewise for B= 2. This is a generalized version of a well-known POMDP reduction used in the
Bayesian Reinforcement Learning literature [Ghavamzadeh et al., 2016].
Given any policy , the expected payoff of inD=w1D1+w2D2is exactly
P(B= 1)E(UjB= 1) + P(B= 2)E(UjB= 2) (4)
=w(1)E(1)
[U(1)] +w(2)E(2)
[U(2)] (5)
Therefore, using the above deﬁnitions, Lemma 2 may be restated in the following equivalent form:
Lemma 3. Given a pair (D(1);D(2))of compatible POMDPs, a policy is Pareto optimal for that
pair if and only if there exist weights w(j)such thatis an optimal policy for the single POMDP
given byw(1)D(1)+w(2)D(2).
3.4 Structural properties of Pareto optimal POMDP reduction
Expressed in the form of Equation 3, it might not be clear how a Pareto optimal policy makes use of its
observations over time, aside from storing them in memory. For example, is there any sense in which
the NRL agent carries “beliefs" about the environment that it “updates" at each time step? Lemma 3
allows us to answer this and related questions by translating theorems about single POMDPs into
theorems about compatible pairs of POMDPs.
We introduce the notation hito represent the NRL agent’s action-observation history at time i, i.e.
hi= (oi;a<i). At timestep i, the NRL agent has a belief over the latent value B, conditioned on hi.
Note that given an action-observation history hi, the NRL agent’s posterior belief over the value of B
is proportional to the probabilities assigned by each principal’s outlook to the realized observation
sequence:
P(B=jjhi)/w(j)P(j)(oija<i).
This relation, combined with the expected value expression in Equation 4, reveals a pattern in how
the weights on the principals’ conditionally expected utilities must change over time, which is the
main result of this paper:
5
Theorem 4 (Pareto optimal policy recursion) .Given a pair (D(1);D(2))of compatible POMDPs of
lengthn, a policyis Pareto optimal if and only if its components iforinsatisfy the following
backward recursion for some pair of weights w(1);w(2)0withw(1)+w(2)= 1:
(hi)2argmax
2A
w(1)P(1)
oija<i
E(1)
[U(1)jhi;ai]
+w(2)P(2)
oija<i
E(2)
[U(2)jhi;ai]
In words, to achieve Pareto optimality, the machine must
1.use each principal’s own beliefs when estimating the degree to which a decision favors that
principal’s utility function, and
2.shift the relative priorities of the principals’ expected utilities in the machine’s decision
objective over time, by a factor proportional to how well the principals predict the machine’s
inputs.
Proof. By Lemma 3, the Pareto-optimality of for(D(1);D(2))is equivalent to its classical opti-
mality forD=w(1)D(1)+w(2)D(2)for some (w(1);w(2)). Writing Pfor probabilities in D, this is
equivalent to =(hi)maximizing the following expression F()for eachi2f0;:::;ng:
F() =E(D)
[Ujhi;ai]: (6)
The above property states, in words, that the optimal policy is the one that maximizes future expected
rewards given an observation history, without regard to any alternate histories and by recursively
assuming the same selection process for future timesteps. This is a standard formulation for POMDPs,
and is exactly Bellman’s Principle of Optimality [Bellman, 1957, Chap III, 3.]
The tower property of expectation allows us to write the above expectation factor as
E(D)[Ujhi;ai] =P(B= 1jhi)E(D)[Ujhi;ai;B = 1]
+P(B= 2jhi)E(D)[Ujhi;ai;B = 2]:(7)
Now, observe that, by Bayes’ rule, the posterior probability
P(B=jjhi)/w(j)P(j)(oija<i). (8)
By construction, E(D)[Ujhi;ai;B =j] =E(j)[U(j)jhi;ai]. Substituting this
expression and (8) into Equation 7 gives us the result.
An intuition about this property is gained by noting that as the NRL agent takes actions in the single
POMDPw(1)D(1)+w(2)D(2), its posterior belief about the value of the latent variable Bis exactly
equal to its belief about which utility function is “correct” for the POMDP it is acting in.
When the principals have the same beliefs, they always assign the same probability to the machine’s
inputs, so the weights on their respective expectations do not change over time. In this case, Harsanyi’s
utility aggregation formula is recovered as a special instance.
3.5 Interpretation as Bet Settling
Theorem 4 shows that a Pareto optimal policy must tend, over time, toward prioritizing the expected
utility of whichever principal’s beliefs best predict the machine’s inputs better. From some perspec-
tives, this is a little counterintuitive: not only must the machine gradually place more predictive weight
on whichever principal’s prior is a better predictor, but it must reward that principal by attending
more to their utility function as well.
Thus, a machine implementing a Pareto optimal policy can be viewed as a kind of bet-settling device.
If Alice is 90% sure the Four Horsemen will appear tomorrow and Bob is 80% sure they won’t, it
makes sense for Alice to ask—while bargaining with Bob for the machine’s policy—that the machine
6
Figure 1: The agent initially heads for Goal 2in the top-right corner. However, at Frame 9 it has
observed eight deterministic transitions in a row. Its conﬁdence in MDP 1 is high enough at that point
that it veers downward and heads to Goal 1Shown: Frames 0, 6, 12, and 15 of the trajectory.
prioritize her values more if the Four Horsemen arrive tomorrow, in exchange for prioritizing Bob’s
values more if they don’t. Both parties will be happy with this agreement in expectation.
4 Evaluation
4.1 Experiment Environment
Our experiments are run in a modiﬁed version of the FrozenLake environment in OpenAI Gym
[Brockman et al., 2016]. FrozenLake is a grid world environment that simulates a goal MDP. The
agent receives a reward upon reaching the speciﬁed goal position. The agent can choose to move
NORTH ,SOUTH ,EAST , orWEST , and the transition model can be chosen to be either stochastic or
deterministic. Under a stochastic transition model, simulating the eponymous frozen lake, the agent’s
action fails with probability 0.2, and the agent transitions into one of the unintended neighboring
positions. In the FrozenLake environment, state is fully observed by the agent. We modify the
environment to support multiple possible goal states, labeled 1and2, corresponding to the utilities of
Principal 1 and 2, respectively. For simplicity, we treat the state as fully-observable.
In these experiments Principal 1 assigns utility to the agent reaching each goal labeled 1and has the
belief that the environment has a deterministic transition model. Principal 2 assigns utility to the
agent reaching each goal labeled 2and has the belief that the environment has a stochastic transition
model. The agent is initialized with initial belief state w1, corresponding to a subjective belief that the
agent is in Principal 1’s MDP, M1, with probability w1and Principal 2’s MDP, M2, with probability
1w1=w2. We use point-based value iteration to learn a the belief-space policy. The agent is then
placed in either M1orM2, and we observe over time as the agent acts according to its belief state.
4.2 Experiments
Observed Behavior In this set of experiments, we run the NRL agent in order to verify that its
behavior resembles a type of bet-settling. The true environment is chosen to be deterministic for this
experiment. After running point-based value iteration [Pineau et al., 2003] with a belief set of 331
points, we execute the resulting policy in this environment. A portion of the agent’s trajectory is
seen in Figure 1. The purple arrows represent the agent’s choice of action at each physical position
under the current belief state. The color of each square represents the agent’s subjective value at each
position under the current belief.
Observing the trajectory, we see that it initially moves towards goal 2. However, each time an action
succeeds, the agent’s belief in the stochastic environment decreases. By the ninth frame, the agent’s
belief in the stochastic world, and as a result its belief that goal 2grants reward, is low enough that
the policy shifts to push the agent to goal 1. This is the type of behavior we would expect of an agent
that maximizes each principal’s utility based on the likelihood of their beliefs.
Next, we use the same policy and place the agent in a stochastic world. The very ﬁrst action results in
a stochastic transition, and the likelihood of Principal 1’s belief immediately falls to 0. At that point,
the agent knows it is in Principal 2’s MDP and believes that goal 2will give it reward. It heads to
goal 2accordingly.
7
Figure 2: Left: Principal 1’s predicted expected reward while in MDP 2. Right: Principal 2’s predicted
expected reward during trajectory depicted in MDP 1.
Full trajectories for both of these experiments are presented in the supplementary material.
Subjective State Value During Execution The promise of compromise behavior may bring parties
into agreement over the decision to build a NRL agent. However if, after the agent begins taking
actions, parties feel that the agent will not take actions that guarantee them high future rewards, those
parties may be tempted to end cooperation. The next experiment attempts to test how losing parties
assess the agent’s behavior during execution.
To address this concern, we turn back to the trajectories discussed in the previous section. In the
ﬁrst trajectory, the agent is, in fact, operating in Principal 1’s MDP. At each timestep, we ask what
Principal 2 believes their expected sum of future rewards to be. Recall that each principal believes
the agent is acting in their own MDP. Since both principals know the agent’s policy, physical state,
and belief state at each timestep, each principal can estimate their future rewards by simulating the
agent acting in their respective MDP with initial conﬁguration given by the current conﬁguration.
For the experiments here, we rollout 100 simulations from each conﬁguration encountered along
the two trajectories. For each simulation, we place the agent within the losing principal’s MDP and
measure the average reward attained by the agent from that starting conﬁguration. Principal 2’s
assessment of their expected future rewards during execution in MDP 1 is shown in the right frame of
Figure 2. Principal 1’s assessment of their expected future rewards during execution in MDP 2 is
shown in the left frame.
We see that until frame 9, Principal 2 believes that the agent has a high chance of reaching goal 2. It
is only when the agent’s behavior shifts and it veers towards goal 1that Principal 2 begins to assess
their expected future utility as decreasing sharply. In contrast, as soon as Principal 1 observes that
w1= 0:0, they realize that the agent will never reach goal 1. Both of these assessments are consistent
with the bet-settling behavior we expect.
5 Conclusion
Insofar as Theorem 4 is not particularly mathematically sophisticated—it employs only basic facts
about convexity and linear algebra—this suggests there may be more low-hanging fruit to be found in
the domain of “machine implementable social choice theory." To recapitulate, Theorem 4 represents
two deviations from the intuition of naïve utility aggregation: to achieve Pareto optimality for
principals with differing beliefs, an agent must (1) use each principal’s own beliefs in evaluating how
well an action will serve that principal’s utility function, and (2) shift the relative priority it assigns
to each principal’s expected utilities over time by a factor proportional to how well that principal’s
beliefs predict the machine’s inputs.
As a ﬁnal remark, consider that social choice theory and bargaining theory were both pioneered during
the Cold War, when it was particularly compelling to understand the potential for cooperation between
human institutions that might behave competitively. In the coming decades, machine intelligence
will likely bring many new challenges for cooperation, as well as new means to cooperate, and new
reasons to do so. As such, new technical aspects of social choice and bargaining, along the lines of
this paper, will likely continue to emerge.
8
References
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In
Proceedings of the twenty-ﬁrst international conference on Machine learning , page 1. ACM, 2004.
Stuart Armstrong, Nick Bostrom, and Carl Shulman. Racing to the Precipice: A Model of Artiﬁcial
Intelligence Development. AI & SOCIETY , 31(2):201–206, 2016.
Seth D Baum. On the Promotion of Safe and Socially Beneﬁcial Artiﬁcial Intelligence. AI &
SOCIETY , pages 1–9, 2016.
Richard Bellman. Dynamic Programming . Princeton University Press, 1957.
Nick Bostrom. Superintelligence: Paths, dangers, strategies . OUP Oxford, 2014.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and
Wojciech Zaremba. OpenAI Gym, 2016.
Zoltán Gábor, Zsolt Kalmár, and Csaba Szepesvári. Multi-Criteria Reinforcement Learning. In ICML ,
volume 98, pages 197–205, 1998.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian Reinforcement
Learning: A Survey. ArXiv e-prints , September 2016.
Dylan Hadﬁeld-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. Cooperative Inverse
Reinforcement Learning, 2016.
John C Harsanyi. Cardinal Welfare, Individualistic Ethics, and Interpersonal Comparisons of Utility.
InEssays on Ethics, Social Behavior, and Scientiﬁc Explanation , pages 6–23. Springer, 1980.
Andrew Y Ng and Stuart J Russell. Algorithms for Inverse Reinforcement Learning. In ICML , pages
663–670, 2000.
Joelle Pineau, Geoff Gordon, and Sebastian Thrun. Point-based Value Iteration: An Anytime
Algorithm for POMDPs. IJCAI’03, pages 1025–1030, 2003.
Diederik M Roijers, Shimon Whiteson, and Frans A Oliehoek. Point-based planning for multi-
objective pomdps. In IJCAI 2015: Proceedings of the Twenty-Fourth International Joint Conference
on Artiﬁcial Intelligence , pages 1666–1672, 2015.
Stuart Russell. Learning Agents for Uncertain Environments. In Proceedings of the eleventh annual
conference on Computational learning theory , pages 101–103. ACM, 1998.
Harold Soh and Yiannis Demiris. Evolving Policies for Multi-Reward Partially Observable Markov
Decision Processes (MR-POMDPs). In Proceedings of the 13th annual conference on Genetic and
evolutionary computation , pages 713–720. ACM, 2011.
Edward J. Sondik. The Optimal Control of Partially Observable Markov Processes over a Finite
Horizon. Oper. Res. , 21(5):1071–1088, October 1973.
Gwo-Hshiung Tzeng and Jih-Jeng Huang. Multiple Attribute Decision Making: Methods and
Applications . CRC press, 2011.
Weijia Wang. Multi-Objective Sequential Decision Making . PhD thesis, Université Paris Sud-Paris
XI, 2014.
Chongjie Zhang and Julie A Shah. Fairness in Multi-Agent Sequential Decision-Making. In Advances
in Neural Information Processing Systems , pages 2636–2644, 2014.
9

Nishant Desai
Andrew Critch
Stuart J Russell