Learning Robot Objectives
from Physical Human Interaction
Andrea Bajcsy
University of California, Berkeley
abajcsy@berkeley.eduDylan P. Losey
Rice University
dlosey@rice.edu
Marcia K. O’Malley
Rice University
omalleym@rice.eduAnca D. Dragan
University of California, Berkeley
anca@berkeley.edu
Abstract: When humans and robots work in close proximity, physical interaction
is inevitable. Traditionally, robots treat physical interaction as a disturbance, and
resume their original behavior after the interaction ends. In contrast, we argue
that physical human interaction is informative : it is useful information about how
the robot should be doing its task. We formalize learning from such interactions
as a dynamical system in which the task objective has parameters that are part
of the hidden state, and physical human interactions are observations about these
parameters. We derive an online approximation of the robot’s optimal policy in
this system, and test it in a user study. The results suggest that learning from
physical interaction leads to better robot task performance with less human effort.
Keywords: physical human-robot interaction, learning from demonstration
1 Introduction
Imagine a robot performing a manipulation task next to a person, like moving the person’s coffee
mug from a cabinet to the table (Fig. 1). As the robot is moving, the person might notice that the
robot is carrying the mug too high above the table. Knowing that the mug would break if it were to
slip and fall from so far up, the person easily intervenes and starts pushing the robot’s end-effector
down to bring the mug closer to the table. In this work, we focus on how the robot should then
respond to such physical human-robot interaction (pHRI).
Several reactive control strategies have been developed to deal with pHRI [1, 2, 3]. For instance,
when a human applies a force on the robot, it can render a desired impedance or switch to gravity
compensation and allow the human to easily move the robot around. In these strategies, the moment
the human lets go of the robot, it resumes its original behavior—our robot from earlier would go
back to carrying the mug too high, requiring the person to continue intervening until it ﬁnished the
task (Fig. 1, left).
Although such control strategies guarantee fast reaction to unexpected forces, the robot’s return to
its original motion stems from a fundamental limitation of traditional pHRI strategies: they miss
the fact that human interventions are often intentional and occur because the robot is doing some-
thing wrong. While the robot’s original behavior may have been optimal with respect to the robot’s
pre-deﬁned objective function, the fact that a human intervention was necessary implies that this
objective function was not quite right.
Our insight is that because pHRI is intentional, it is also informative —it provides observations
about the correct robot objective function, and the robot can leverage these observations to learn
that correct objective.
Returning to our example, if the person is applying forces to push the robot’s end-effector closer to
the table, then the robot should change its objective function to reﬂect this preference, and complete
the rest of the current task accordingly, keeping the mug lower (Fig. 1, right). Ultimately, hu-
man interactions should not be thought of as disturbances, which perturb the robot from its desired
behavior, but rather as corrections, which teach the robot its desired behavior.
In this paper, we make the following contributions:
Formalism. We formalize reacting to pHRI as the problem of acting in a dynamical system to
optimize an objective function, with two caveats: 1) the objective function has unknown parameters
1st Conference on Robot Learning (CoRL 2017), Mountain View, United States.
θ
θ
θ
Figure 1: A person interacts with a robot that treats interactions as disturbances (left), and a robot that learns
from interactions (right). When humans are treated as disturbances, force plots reveal that people have to
continuously interact since the robot returns to its original, incorrect trajectory. In contrast, a robot that learns
from interactions requires minimal human feedback to understand how to behave (i.e., go closer to the table).
, and 2) human interventions serve as observations about these unknown parameters: we model
human behavior as approximately optimal with respect to the true objective. As stated, this problem
is an instance of a Partially Observable Markov Decision Process (POMDP). Although we cannot
solve it in real-time using POMDP solvers, this formalism is crucial to converting the problem of
reacting to pHRI into a clearly deﬁned optimization problem. In addition, our formalism enables
pHRI approaches to be justiﬁed and compared in terms of this optimization criterion.
Online Solution. We introduce a solution that adapts learning from demonstration approaches to
our online pHRI setting [4, 5], but derive it as an approximate solution to the problem above. This
enables the robot to adapt to pHRI in real-time, as the current task is unfolding. Key to this ap-
proximation is simplifying the observation model: rather than interpreting instantaneous forces as
noisy-optimal with respect to the value function given, we interpret them as implicitly inducing
a noisy-optimal desired trajectory . Reasoning in trajectory space enables an efﬁcient approximate
online gradient approach to estimating .
User Study. We conduct a user study with the JACO2 7-DoF robotic arm to assess how online
learning from physical interactions during a task affects the robot’s objective performance, as well
as subjective participant perceptions.
Overall, our work is a ﬁrst step towards learning robot objectives online from pHRI.
2 Related Work
We propose using pHRI to correct the robot’s objective function while the robot is performing its cur-
rent task . Prior research has focused on (a) control strategies for reacting to pHRI without updating
the robot’s objective function, or (b) learning the robot’s objectives—from ofﬂine demonstrations—
in a manner that generalizes to future tasks, but does not change the behavior during the current task.
An exception is shared autonomy work, which does correct the robot’s objective function online, but
only when the objective is parameterized by the human’s desired goal in free-space .
Control Strategies for Online Reactions to pHRI. A variety of control strategies have been de-
veloped to ensure safe and responsive pHRI. They largely fall into three categories [6]: impedance
control, collision handling, and shared manipulation control. Impedance control [1] relates devia-
tions from the robot’s planned trajectory to interaction torques. The robot renders a virtual stiffness,
damping, and/or inertia, allowing the person to push the robot away from its desired trajectory, but
the robot always returns to its original trajectory after the interaction ends. Collision handling meth-
ods [2] include stopping, switching to gravity compensation, or re-timing the planned trajectory if
a collision is detected. Finally, shared manipulation [3] refers to role allocation in situations where
the human and the robot are collaborating. These control strategies for pHRI work in real-time, and
enable the robot to safely adapt to the human’s actions; however, the robot fails to leverage these in-
terventions to update its understanding of the task—left alone, the robot would continue to perform
the task in the same way as it had planned before any human interactions. By contrast, we focus on
enabling robots to adjust how they perform the current task in real time.
Ofﬂine Learning of Robot Objective Functions. Inverse Reinforcement Learning (IRL) methods
focus explicitly on inferring an unknown objective function, but do it ofﬂine , after passively observ-
ing expert trajectory demonstrations [7]. These approaches can handle noisy demonstrations [8],
which become observations about the true objective [9], and can acquire demonstrations through
2
physical kinesthetic teaching [10]. Most related to our work are approaches which learn from cor-
rections of the robot’s trajectory, rather than from demonstrations [4, 5, 11]. Our work, however,
has a different goal: while these approaches focus on the robot doing better the next time it performs
the task, we focus on the robot completing its current task correctly. Our solution is analogous to
online Maximum Margin Planning [4] and co-active learning [5] for this new setting, but one of our
contributions is to derive their update rule as an approximation to our pHRI problem.
Online Learning of Human Goals. While IRL can learn the robot’s objective function after one or
more demonstrations of a task, online inference is possible when the objective is simply to reach a
goal state , and the robot moves through free space [12, 13, 14]. We build on this work by considering
general objective parameters ; this requires a more complex (non-analytic and difﬁcult to compute)
observation model, along with additional approximations to achieve online performance.
3 Learning Robot Objectives Online from pHRI
3.1 Formalizing Reacting to pHRI
We consider settings where a robot is performing a day-to-day task next to a person, but is not doing
it correctly (e.g., is about to spill a glass of water), or not doing it in a way that matches the person’s
preferences (e.g., is getting too close to the person). Whenever the person physically intervenes and
corrects the robot’s motion, the robot should react accordingly; however, there are many strategies
the robot could use to react. Here, we formalize the problem as a dynamical system with a true ob-
jective function that is known by the person but not known by the robot. This formulation interprets
the human’s physical forces as intentional, and implicitly deﬁnes an optimal strategy for reacting.
Notation. Letxdenote the robot’s state (its position and velocity) and uRthe robot’s action (the
torque it applies at its joints). The human physically interacts with the robot by applying external
torqueuH. The robot transitions to a next state deﬁned by its dynamics, \_x=f(x;uR+uH), where
both the human and robot can inﬂuence the robot’s motion.
POMDP Formulation. The robot optimizes a reward function r(x;uR;uH;), which trades off
between correctly completing the task and minimizing human effort
r(x;uR;uH;) =T(x;uR;uH)jjuHjj2(1)
Following prior IRL work [15, 4, 8], we parameterize the task-related part of this reward function as
a linear combination of features with weights . Note that we assume the relevant set of features
for each task are given, and we will not explore feature selection within this work.
Hereencapsulates the true objective, such as moving the glass slowly, or keeping the robot’s end-
effector farther away from the person. Importantly, this parameter is not known by the robot —robots
will not always know the right way to perform a task, and certainly not the human-preferred way. If
the robot knew , this would simply become an MDP formulation, where the states are x, the actions
areuR, the reward is r, and the person would never need to intervene.
Uncertainty over , however, turns this into a POMDP formulation, where is a hidden part of the
state. Importantly, the human’s actions are observations about under some observation model
P(uHjx;uR;). These observations uHare atypical in two ways: (a) they affect the robot’s reward,
as in [13], and (b) they inﬂuence the robot’s state, but we don’t necessarily want to account for that
when planning – the robot should not rely on the human to move the robot; rather the robot should
consideruHonly for its information value.
Observation Model. We model the human’s interventions as corrections which approximately max-
imize the robot’s reward. More speciﬁcally, we assume the noisy-rational human selects an action
uHthat, when combined with the robot’s action uR, leads to a high Q-value (state-action value)
assuming the robot will behave optimally after the current step (i.e., assuming the robot knows )
P(uHjx;uR;)/eQ(x;uR+uH;)(2)
Our choice of (2) stems from maximum entropy assumptions [8], as well as the Bolzmann distribu-
tions used in cognitive science models of human behavior [16].
Aside. We are notformulating this as a POMDP to solve it using standard POMDP solvers. Instead,
our goal is to clarify the underlying problem formulation and the existence of an optimal strategy.
3.2 Approximate Solution
Since POMDPs cannot be solved tractably for high-dimensional real-world problems, we make
several approximations to arrive at an online solution. We ﬁrst separate estimation from ﬁnding the
3
optimal policy, and approximate the policy by separating planning from control. We then simplify
the estimation model, and use maximum a posteriori estimate (MAP) instead the full belief over .
QMDP. Similar to [13], we approximate our POMDP using a QMDP by assuming the robot will
obtain full observability at the next time step [17]. Let bdenote the robot’s current belief over .
The QMDP simpliﬁes into two subproblems: (a) ﬁnding the robot’s optimal policy givenb
Q(x;uR;b) =Z
b()Q(x;uR;)d (3)
where arg maxuRQ(x;uR;b)evaluated at every state yields the optimal policy, and (b) updating
our belief over given a new observation. Unlike the actual POMDP solution, here the robot will
not try to gather information.
From Belief to Estimator. Rather than planning with the belief b, we plan with only the MAP of ^.
From Policies to Trajectories (Action). ComputingQin continuous state, action, and belief spaces
is still not tractable. We thus separate planning andcontrol . At every time step t, we do two things.
First, given our current ^t, we replan a trajectory =x0:T2that optimizes the task-related
reward. LetT()be the cumulative reward, where ()is the total feature count along trajectory
such that () =P
xt2(xt). We use a trajectory optimizer [18] to replan the robot’s desired
trajectoryt
R
t
R= arg max
^t() (4)
Second, once t
Rhas been planned, we control the robot to track this desired trajectory. We use
impedance control, which allows people to change the robot’s state by exerting torques, and provides
compliance for human safety [19, 6, 1]. After feedback linearization [20], the equation of motion
under impedance control becomes
MR(qtqt
R) +BR( \_qt\_qt
R) +KR(qtqt
R) =ut
H (5)
HereMR,BR, andKRare the desired inertia, damping, and stiffness, x= (q;\_q), whereqis the
robot’s joint position, and qR2Rdenotes the desired joint position. Within our experiments, we
implemented a simpliﬁed impedance controller without feedback linearization
ut
R=BR( \_qt
R\_qt) +KR(qtqt
R) (6)
Aside. When the robot is not updating its estimate ^, thent
R=t1
R, and our solution reduces to
using impedance control to track an unchanging trajectory [2, 19].
From Policies to Trajectories (Estimation). We still need to address the second QMDP subprob-
lem: updating ^after each new observation. Unfortunately, evaluating the observation model (2) for
any givenis difﬁcult, because it requires computing the Q-value function for that . Hence, we
will again leverage a simpliﬁcation from policies to trajectories in order to update our MAP of .
Instead of attempting to directly relate uHto, we propose an intermediate step; we interpret each
human action uHvia a intended trajectory ,H, that the human wants the robot to execute. To
compute the intended trajectory HfromRanduH, we propagate the deformation caused by uH
along the robot’s current trajectory R
H=R+A1UH (7)
where > 0scales the magnitude of the deformation, Adeﬁnes a norm on the Hilbert space of
trajectories and dictates the deformation shape [21], UH=uHat the current time, and UH= 0 at
all other times. During experiments we here used a norm Abased on acceleration [21], but we will
explore learning the choice of this norm in future work.
Importantly, our simpliﬁcation from observing human action uHto implicitly observing the human’s
intended trajectory Hmeans we no longer have to evaluate the Q-value ofuR+uHgiven some
value. Instead, the observation model now depends on the total reward of the implicitly observed
trajectory:
P(HjR;)/eT(H)jjuHjj2eT(H)jjHRjj2(8)
This is analogous to (2), but in trajectory space—a distribution over implied trajectories, given and
the current robot trajectory.
4
O
2
O
1
Figure 2: Algorithm (left) and visualization (right) of one iteration of our online learning from pHRI method
in an environment with two obstacles O1;O2. The originally planned trajectory, t
R(black dotted line), is
deformed by the human’s force into the human’s preferred trajectory, t
H(solid black line). Given these two
trajectories, we compute an online update of and can replan a better trajectory t+1
R(orange dotted line).
3.3 Online Update of the Estimate
The probability distribution over at time step tisP(0
H;::;t
Hj;0
R;::;t
R)P(). However, since
is continuous, and the observation model is not Gaussian, we opt not to track the full belief, but
rather to track the maximum a posteriori estimate (MAP). Our update rule for this estimate will
reduce to online Maximum Margin Planning [4] if we treat Has the demonstration, and to co-
adaptive learning [5], if we treat Has the original trajectory with one waypoint corrected. One of
our contributions, however, is to derive this update rule from our MaxEnt observation model in (8).
MAP. Assuming the observations are conditionally independent given , the MAP for time t+ 1is
^t+1= arg max
P(0
H;::;t
Hj0
R;::;t
R;)P() = arg max
tX
=0logP(
Hj
R;) + logP()
(9)
Inspecting the right side of (9), we need to deﬁne both P(HjR;)and the prior P(). To ap-
proximateP(HjR;), we use (8) with Laplace’s method to compute the normalizer. Taking a
second-order Taylor series expansion of the objective function about R, the robot’s current best
guess at the optimal trajectory, we obtain a Gaussian integral that can be evaluated in closed form
P(HjR;) =eT(H)jjHRjj2
R
eT()jjRjj2deT
(H)(R)
jjHRjj2(10)
Let^0be our initial estimate of . We propose the prior
P() =e1
2jj^0jj2(11)
whereis a positive constant. Substituting (10) and (11) into (9), the MAP reduces to
^t+1arg max
(tX
=0T
(
H)(
R)
1
2jj^0jj2)
(12)
Notice that the jjHRjj2terms drop out, because this penalty for human effort does not explicitly
depend on. Solving the optimization problem (12) by taking the gradient with respect to , and
then setting the result equal to zero, we ﬁnally arrive at
^t+1=^0+tX
=0
(
H)(
R)
=^t+
(t
H)(t
R)
(13)
Interpretation. This update rule is actually the online gradient [22] of (9) under our Laplace ap-
proximation of the observation model. It has an intuitive interpretation: it shifts the weights in the
direction of the human’s intended feature count. For example, if Hstays farther from the person
thanR, the weights in associated with distance-to-person features will increase.
Relation to Prior Work. This update rule is analogous to two related works. First, it would be the
online version of Maximum Margin Planning (MMP) [4] if the trajectory t
Hwere a new demon-
5
(a) Task 1: Cup orientation
 (b) Task 2: Distance to table
 (c) Task 3: Laptop avoidance
Figure 3: Simulations depicting the robot trajectories for each of the three experimental tasks. The black path
represents the original trajectory and the blue path represents the human’s desired trajectory.
stration. Unlike MMP, our robot does not complete a trajectory, and only then get a full new demon-
stration; instead, our t
His an estimate of the human’s intended trajectory based on the force applied
during the robot’s execution of the current trajectory t
R. Second, the update rule would be co-active
learning [5] if the trajectory t
Hweret
Rwith one waypoint modiﬁed, as opposed to a propagation
ofut
Halong the rest of t
R. Unlike co-active learning, however, our robot receives corrections con-
tinually, and continually updates the current trajectory in order to complete the current task well.
Nonetheless, we are excited to see similar update rules emerge from different optimization criteria.
Summary. We formalized reacting to pHRI as a POMDP with the correct objective parameters as
a hidden state, and approximated the solution to enable online learning from physical interaction.
At every time step during the task where the human interacts with the robot, we ﬁrst propagate uH
to implicitly observe the corrected trajectory H(simpliﬁcation of the observation model), and then
update ^via Equation (13) (MAP instead of belief). We replan with the new estimate (approximation
of the optimal policy), and use impedance control to track the resulting trajectory (separation of
planning from control). We summarize and visualize this process in Fig. 2.
4 User Study
We conducted an IRB-approved user study to investigate the beneﬁts of in-task learning. We de-
signed tasks where the robot began with the wrong objective function, and participants phsyically
corrected the robot’s behavior1.
4.1 Experiment Design
Independent Variables. We manipulated the pHRI strategy with two levels: learning and
impedance . The robot either used our method (Algorithm 1) to react to physical corrections and
re-plan a new trajectory during the task; or used impedance control (our method without updating
^) to react to physical interactions and then return to the originally planned trajectory.
Dependent Measures. We measured the robot’s performance with respect to the true objective,
along with several subjective measures. One challenge in designing our experiment was that each
person might have a different internal objective for any given task, depending on their experience
and preferences. Since we do not have direct access to every person’s internal preferences, we
deﬁned the true objective ourselves, and conveyed the objectives to participants by demonstrating
the desired optimal robot behavior (see an example in Fig. 3(a), where the robot is supposed to keep
the cup upright). We instructed participants to get the robot to achieve this desired behavior with
minimal human physical intervention.
For each robot attempt at a task, we evaluated the task related and effort related parts of the ob-
jective:T()(a cost to be minimized and not a reward to be maximized in our experiment) andP
tjjut
Hjj1. We also evaluate the total amount of time spent interacting physically with the robot.
For our subjective measures, we designed 4 multi-item scales shown in Table 1: did participants
think the robot understood how they wanted to task done, did they feel like they had to exert a lot
of effort to correct the robot, was it easy to anticipate the robot’s reactions, and how good of a
collaborator was the robot.
Hypotheses :
H1.Learning signiﬁcantly decreases interaction time, effort, and cumulative trajectory cost.
1For video footage of the experiment, see: https://www.youtube.com/watch?v=1MkI6DH1mcw
6
Cup Table Laptop
Task02004006008001000Total Eﬀort (Nm)\*\*\*Average Total Human Eﬀort
Impedance
Learning
Cup Table Laptop
Task0246810121416Interact Time (s)\* \*\*Average Total Interaction Time
Impedance
LearningFigure 4: Learning from pHRI decreases human effort and interaction time across all experimental tasks (total
trajectory time was 15s). An asterisk (\*) means p<0:0001 .
H2.Participants will believe the robot understood their preferences, feel less interaction effort, and
perceive the robot as more predictable and more collaborative in the learning condition.
Tasks. We designed three household manipulation tasks for the robot to perform in a shared
workspace (see Fig. 3), plus a familiarization task. As such, the robot’s objective function con-
sidered two features: velocity and a task-speciﬁc feature. For each task, the robot carried a cup
from a start to a goal pose with an initially incorrect objective , requiring participants to correct its
behavior during the task.
During the familiarization task, the robot’s original trajectory moved too close to the human. Partic-
ipants had to physically interact with the robot to get it to keep the cup further away from their body.
In Task 1, the robot would not care about tilting the cup mid-task, risking spilling if the cup was too
full. Participants had to get the robot to keep the cup upright. In Task 2, the robot would move the
cup too high in the air, risking breaking it if it were to slip, and participants had to get the robot to
keep it closer to the table. Finally, in Task 3, the robot would move the cup over a laptop to reach
it’s ﬁnal goal pose, and participants had to get the robot to keep the cup away from the laptop.
Participants. We used a within-subjects design and counterbalanced the order of the pHRI strategy
conditions. In total, we recruited 10 participants (5 male, 5 female, aged 18-34) from the UC
Berkeley community, all of whom had technical backgrounds.
Procedure. For each pHRI strategy, participants performed the familiarization task, followed by
the three tasks, and then ﬁlled out our survey. They attempted each task twice with each strategy
for robustness, and we recorded the attempt number for our analysis. Since we artiﬁcially set the
true objective for participants to measure objective performance, we showed participants both the
original and desired robot trajectory before interaction (Fig. 3), so that they understood the objective.
4.2 Results
Objective. We conducted a factorial repeated measures ANOV A with strategy (impedance or learn-
ing) and trial number (ﬁrst attempt or second attempt) as factors, on total participant effort, inter-
action time, and cumulative true cost2(see Figure 4 and Figure 5). Learning resulted in signiﬁ-
cantly less interaction force ( F(1;116) = 86:29;p < 0:0001 ) and interaction time ( F(1;116) =
75:52;p < 0:0001 ), and signiﬁcantly better task cost ( F(1;116) = 21:85;p < 0:0001 ). Interest-
ingly, while trial number did not signiﬁcantly affect participant’s performance with either method,
attempting the task a second time yielded a marginal improvement for the impedance strategy, but
not for the learning strategy. This may suggest that it is easier to get used to the impedance strategy.
Overall, this supports H1, and aligns with the intuition that if humans are truly intentional actors,
then using interaction forces as information about the robot’s objective function enables robots to
better complete their tasks with less human effort compared to traditional pHRI methods.
Subjective. Table 1 shows the results of our participant survey. We tested the reliability of our 4
scales, and found the understanding, effort, and collaboration scales to be reliable, so we grouped
them each into a combined score. We ran a one-way repeated measures ANOV A on each resulting
score. We found that the robot using our method was perceived as signiﬁcantly ( p<0:0001 ) more
understanding, less difﬁcult to interact with, and more collaborative. However, we found no signif-
icant difference between our method and the baseline impedance method in terms of predictability.
2For simplicity, we only measured the value of the feature that needed to be modiﬁed in the task, and
computed the absolute difference from the feature value of the optimal trajectory.
7
Cup Table Laptop
Task020406080100Cost Value\*
\*
\*Average Cost Across Tasks
Impedance
Learning
Desired
Figure 5: (left) Average cumulative cost for each task as compared to the desired total trajectory cost. An
asterisk (\*) means p<0:0001 . (right) Plot of sample participant data from laptop task: desired trajectory is in
blue, trajectory with impedance condition is in gray, and learning condition trajectory is in orange.
Participant comments suggest that while the robot adapted quickly to their corrections when learn-
ing (e.g. “The robot seemed to quickly ﬁgure out what I cared about and kept doing it on its own”),
determining what the robot was doing during learning was less apparent (e.g. “If I pushed it hard
enough sometimes it would seem to fall into another mode and then do things correctly”).
Therefore, H2 was partially supported: although our learning algorithm was not perceived as more
predictable, participants believed that the robot understood their preferences more, took less effort
to interact with, and was a more collaborative partner.
Questions Cronbach’s  Imped LSM Learn LSM F(1,9) p-valueunderstandingBy the end, the robot understood how I wanted it to do the task.
0.94 1.70 5.10 118.56<.0001Even by the end, the robot still did not know how I wanted it to do the task.
The robot learned from my corrections.
The robot did not understand what I was trying to accomplish.effortI had to keep correcting the robot.0.98 1.25 5.10 85.25<.0001
The robot required minimal correction.predictIt was easy to anticipate how the robot will respond to my corrections. 0.8 4.90 4.70 0.06 0.82
The robot’s response to my corrections was surprising. 0.8 3.10 3.70 0.89 0.37collabThe robot worked with me to complete the task.0.98 1.80 4.80 55.86<.0001
The robot did not collaborate with me to complete the task.
Table 1: Results of ANOV A on subjective metrics collected from a 7-point Likert-scale survey.
5 Discussion
Summary. We propose that robots should not treat human interaction forces as disturbances, but
rather as informative actions. We show that this results in robots capable of in-task learning—
robots that update their understanding of the task which they are performing and then complete it
correctly, instead of relying on people to guide them until the task is done. We test this concept with
participants who not only teach the robot to ﬁnish its task according to their preferences, but also
subjectively appreciate the robot’s learning.
Limitations and Future Work. Ours is merely a step in exploring learning robot objectives from
pHRI. We opted for an approximation closest to the existing literature, but other possible better on-
line solutions are possible. In our user study, we assumed knowledge of the two relevant reward
features. In reality, reward functions will have larger feature sets and human interactions may only
give information about a certain subset of relevant weights. The robot will thus need to disambiguate
what the person is trying to correct, likely requiring active information gathering. Further, devel-
oping solutions that can handle dynamical aspects, like preferences about the timing of the motion,
would require a different approach to inferring the intended human trajectory, or going back the
space of policies altogether. Finally, while we focused on in-task learning, the question of how and
when to generalize learned objectives to new task instances remains open.
8
Acknowledgments
Andrea Bajcsy and Dylan P. Losey contributed equally to this work.
We would like to thank Kinova Robotics, who quickly and thoroughly responded to our hardware
questions. This work was funded in part by an NSF CAREER, the Open Philanthropy Project, the
Air Force Ofﬁce of Scientiﬁc Research (AFOSR), and by the NSF GRFP-1450681.
References
[1] N. Hogan. Impedance control: An approach to manipulation; Part II—Implementation. Jour-
nal of Dynamic Systems, Measurement, and Control , 107(1):8–16, 1985.
[2] S. Haddadin, A. Albu-Schaffer, A. De Luca, and G. Hirzinger. Collision detection and reaction:
A contribution to safe physical human-robot interaction. In Intelligent Robots and Systems
(IROS), IEEE/RSJ International Conference on , pages 3356–3363. IEEE, 2008.
[3] N. Jarrass ´e, T. Charalambous, and E. Burdet. A framework to describe, analyze and generate
interactive motor behaviors. PLoS ONE , 7(11):e49945, 2012.
[4] N. D. Ratliff, J. A. Bagnell, and M. A. Zinkevich. Maximum margin planning. In Machine
Learning (ICML), International Conference on , pages 729–736. ACM, 2006.
[5] A. Jain, S. Sharma, T. Joachims, and A. Saxena. Learning preferences for manipulation tasks
from online coactive feedback. The International Journal of Robotics Research , 34(10):1296–
1313, 2015.
[6] S. Haddadin and E. Croft. Physical human–robot interaction. In Springer Handbook of
Robotics , pages 1835–1874. Springer, 2016.
[7] A. Y . Ng and S. J. Russell. Algorithms for inverse reinforcement learning. In Machine Learning
(ICML), International Conference on , pages 663–670. ACM, 2000.
[8] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforce-
ment learning. In AAAI , volume 8, pages 1433–1438, 2008.
[9] D. Ramachandran and E. Amir. Bayesian inverse reinforcement learning. Urbana , 51(61801):
1–4, 2007.
[10] M. Kalakrishnan, P. Pastor, L. Righetti, and S. Schaal. Learning objective functions for ma-
nipulation. In Robotics and Automation (ICRA), IEEE International Conference on , pages
1331–1336. IEEE, 2013.
[11] M. Karlsson, A. Robertsson, and R. Johansson. Autonomous interpretation of demonstrations
for modiﬁcation of dynamical movement primitives. In Robotics and Automation (ICRA),
IEEE International Conference on , pages 316–321. IEEE, 2017.
[12] A. D. Dragan and S. S. Srinivasa. A policy-blending formalism for shared control. The Inter-
national Journal of Robotics Research , 32(7):790–805, 2013.
[13] S. Javdani, S. S. Srinivasa, and J. A. Bagnell. Shared autonomy via hindsight optimization. In
Robotics: Science and Systems (RSS) , 2015.
[14] S. Pellegrinelli, H. Admoni, S. Javdani, and S. Srinivasa. Human-robot shared workspace
collaboration via hindsight optimization. In Intelligent Robots and Systems (IROS), IEEE/RSJ
International Conference on , pages 831–838. IEEE, 2016.
[15] P. Abbeel and A. Y . Ng. Apprenticeship learning via inverse reinforcement learning. In Ma-
chine Learning (ICML), International Conference on . ACM, 2004.
[16] C. L. Baker, J. B. Tenenbaum, and R. R. Saxe. Goal inference as inverse planning. In Proceed-
ings of the Cognitive Science Society , volume 29, 2007.
[17] M. L. Littman, A. R. Cassandra, and L. P. Kaelbling. Learning policies for partially observable
environments: Scaling up. In Machine Learning (ICML), International Conference on , pages
362–370. ACM, 1995.
9
[18] J. Schulman, Y . Duan, J. Ho, A. Lee, I. Awwal, H. Bradlow, J. Pan, S. Patil, K. Goldberg,
and P. Abbeel. Motion planning with sequential convex optimization and convex collision
checking. The International Journal of Robotics Research , 33(9):1251–1270, 2014.
[19] A. De Santis, B. Siciliano, A. De Luca, and A. Bicchi. An atlas of physical human–robot
interaction. Mechanism and Machine Theory , 43(3):253–270, 2008.
[20] M. W. Spong, S. Hutchinson, and M. Vidyasagar. Robot modeling and control , volume 3.
Wiley: New York, 2006.
[21] A. D. Dragan, K. Muelling, J. A. Bagnell, and S. S. Srinivasa. Movement primitives via
optimization. In Robotics and Automation (ICRA), IEEE International Conference on , pages
2339–2346. IEEE, 2015.
[22] L. Bottou. Online learning and stochastic approximations. In On-line Learning in Neural
Networks , volume 17, pages 9–42. Cambridge Univ Press, 1998.
10

Andrea Bajcsy
Dylan P Losey
Marcia K O’Malley
Anca D Dragan


Occam's razor is insufficient to infer the preferences of irrational agents

Occam’s razor is insufﬁcient to infer the preferences
of irrational agents
S¨oren Mindermann\*y
Vector Institute
University of Toronto
soeren.mindermann@gmail.comStuart Armstrong\*z
Future of Humanity Institute
University of Oxford
stuart.armstrong@philosophy.ox.ac.uk
Abstract
Inverse reinforcement learning (IRL) attempts to infer human rewards or pref-
erences from observed behavior. Since human planning systematically deviates
from rationality, several approaches have been tried to account for speciﬁc human
shortcomings. However, the general problem of inferring the reward function of an
agent of unknown rationality has received little attention. Unlike the well-known
ambiguity problems in IRL, this one is practically relevant but cannot be resolved
by observing the agent’s policy in enough environments. This paper shows (1) that
a No Free Lunch result implies it is impossible to uniquely decompose a policy
into a planning algorithm and reward function, and (2) that even with a reasonable
simplicity prior/Occam’s razor on the set of decompositions, we cannot distinguish
between the true decomposition and others that lead to high regret. To address this,
we need simple ‘normative’ assumptions, which cannot be deduced exclusively
from observations.
1 Introduction
In today’s reinforcement learning systems, a simple reward function is often hand-crafted, and still
sometimes leads to undesired behaviors on the part of RL agent, as the reward function is not well
aligned with the operator’s true goals4. As AI systems become more powerful and autonomous, these
failures will become more frequent and grave as RL agents exceed human performance, operate at
time-scales that forbid constant oversight, and are given increasingly complex tasks — from driving
cars to planning cities to eventually evaluating policies or helping run companies. Ensuring that
the agents behave in alignment with human values is known, appropriately, as the value alignment
problem [Amodei et al., 2016, Hadﬁeld-Menell et al., 2016, Russell et al., 2015, Bostrom, 2014,
Leike et al., 2017].
One way of resolving this problem is to infer the correct reward function by observing human
behaviour. This is known as Inverse reinforcement learning (IRL) [Ng and Russell, 2000, Abbeel and
Ng, 2004, Ziebart et al., 2008]. Often, learning a reward function is preferred over imitating a policy:
when the agent must outperform humans, transfer to new environments, or be interpretable. The
reward function is also usually a (much) more succinct and robust task representation than the policy,
especially in planning tasks [Abbeel and Ng, 2004]. Moreover, supervised learning of long-range and
goal-directed behavior is often difﬁcult without the reward function [Ratliff et al., 2006].
Equal contribution.
yWork performed at Future of Humanity Institute.
zFurther afﬁliation: Machine Intelligence Research Institute, Berkeley, USA.
4See for example the game CoastRunners, where an RL agent didn’t ﬁnish the course, but instead
found a bug allowing it to get a high score by crashing round in circles https://blog.openai.com/
faulty-reward-functions/ .
32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr ´eal, Canada.
Usually, the reward function is inferred based on the assumption that human behavior is optimal
or noisily optimal. However, it is well-known that humans deviate from rationality in systematic ,
non-random ways [Tversky and Kahneman, 1975]. This can be due to speciﬁc biases such as time-
inconsistency, loss aversion and hundreds of others, but also limited cognitive capacity, which leads
to forgetfulness, limited planning and false beliefs. This limits the use of IRL methods for tasks that
humans don’t ﬁnd trivial.
Some IRL approaches address speciﬁc biases [Evans et al., 2015b,a], and others assume noisy
rationality [Ziebart et al., 2008, Boularias et al., 2011]. But a general framework for inferring the
reward function from suboptimal behavior does not exist to our knowledge. Such a framework needs
to infer two unobserved variables simultaneously: the human reward function and their planning
algorithm5which connects the reward function with behaviour, henceforth called a planner .
The task of observing human behaviour (or the human policy) and inferring from it the human reward
function and planner will be termed decomposing the human policy. This paper will show there is a
No Free Lunch theorem in this area: it is impossible to get a unique decomposition of human policy
and hence get a unique human reward function. Indeed, anyreward function is possible. And hence,
if an IRL agent acts on what it believes is the human policy, the potential regret is near-maximal.
This is another form of unidentiﬁability of the reward function, beyond the well-known ones [Ng and
Russell, 2000, Amin and Singh, 2016].
The main result of this paper is that, unlike other No Free Lunch theorems, this unidentiﬁability
does not disappear when regularising with a general simplicity prior that formalizes Occam’s razor
[Vitanyi and Li, 1997]. This result will be shown in two steps: ﬁrst, that the simplest decompositions
include degenerate ones, and secondly, that the most ‘reasonable’ decompositions according to human
judgement are of high complexity.
So, although current IRL methods can perform well on many well-speciﬁed problems, they are
fundamentally and philosophically incapable of establishing a ‘reasonable’ reward function for the
human, no matter how powerful they become. In order to do this, they will need to build in ‘normative
assumptions’: key assumptions about the reward function and/or planner, that cannot be deduced
from observations, and allow the algorithm to focus on good ways of decomposing the human policy.
Future work will sketch out some potential normative assumptions that can be used in this area,
making use of the fact that humans assess each other to be irrational, and often these assessments
agree. In view of the No Free Lunch result, this shows that humans must share normative assumptions.
One of these ‘normative assumption’ approaches is brieﬂy illustrated in an appendix, while another
appendix demonstrates how to use the planner-reward formalism to deﬁne when an agent might be
manipulating or overriding human preferences. This happens when the agent pushes the human
towards situations where their policy is very suboptimal according to their reward function.
2 Related Work
In the ﬁrst IRL papers from Ng and Russell [2000] and Abbeel and Ng [2004] a max-margin algorithm
was used to ﬁnd the reward function under which the observed policy most outperforms other policies.
Suboptimal behavior was ﬁrst addressed explicitly by Ratliff et al. [2006] who added slack variables to
allow for suboptimal behavior. This ﬁnds reward functions such that the observed policy outperforms
most other policies and the biggest margin by which another policy outperforms it is minimal, i.e. the
observed policy has low regret. Shiarlis et al. [2017] introduce a modern max-margin technique with
an approximate planner in the optimisation.
However, the max-margin approach has mostly been replaced by the max entropy IRL [Ziebart et al.,
2008]. Here, the assumption is that observed actions or trajectories are chosen with probability
proportional to the exponent of their value. This assumes a speciﬁc suboptimal planning algorithm
which is noisily rational (also known as Boltzmann -rational). Noisy rationality explains human
behavior on various data sets better [Hula et al., 2015]. However, Evans et al. [2015b] and Evans et al.
[2015a] showed that this can fail since humans deviate from rationality in systematic, non-random
ways. If noisy rationality is assumed, repeated suboptimal actions throw off the inference.
5Technically we only need to infer the human reward function, but inferring that from behaviour requires
some knowledge of the planning algorithm.
2
Literature on inferring the reasoning capabilities of an agent is scarce. Evans et al. [2015b] and Evans
et al. [2015a] use Bayesian inference to identify speciﬁc planning biases such as myopic planning and
hyperbolic time-discounting. They simultaneously infer the agent’s preferences. Cundy and Filan
[2018] adds bias resulting from hierarchical planning. Hula et al. [2015] similarly let agents infer
features of their opponent’s reasoning such as planning depth and impulsivity in simple economic
games. Recent work learns the planning algorithm with two assumptions: being close to noisily
rational in a high-dimensional planner space and supervised planner-learning [Anonymous, 2019].
The related ideas of meta-reasoning [Russell, 2016], computational rationality [Lewis et al., 2014] and
resource rationality [Grifﬁths et al., 2015] may create the possibility to redeﬁne irrational behavior
as rational in an ‘ancestral’ distribution of environments where the agent optimises its rewards by
choosing among the limited computations it is able to perform or jointly minimising the cost of
computation and maximising reward. This could in theory redeﬁne many biases as computationally
optimal in some distribution of environments and provide priors on human planning algorithms.
Unfortunately the problem of doing this in practice seems to be extremely difﬁcult — and it assumes
that human goals are roughly the same as evolution’s goals, which is certainly not the case.
3 Problem setup and background
A human will be performing a series of actions, and from these, an agent will attempt to estimate
both the human’s reward function and their planning algorithm.
The environment Min which the human operates is an MDP/R, a Markov Decision Process without
reward function (a world-model [Hadﬁeld-Menell et al., 2017]). An MDP/R is deﬁned as a tuple,
hS;A;T;^siconsisting of a discrete state space S, a ﬁnite action space A, a ﬁxed starting state ^s, and
a probabilistic transition function T:SAS! [0;1]to the next state (also called the dynamics ).
At each step, the human is in a certain state s, takes a certain action a, and ends up in a new state s0
as given byT(s0js;a).
LetR=fR:SA! [1;1]g= [1;1]SAbe the space of candidate reward functions; a given
Rwill map any state-reward pair to a reward value in the interval [1;1].
Letbe the space of deterministic, Markovian policies. So is the space of functions S!A . The
human will be following the policy \_2.
The results of this paper apply to both discounted rewards and episodic environments settings6.
3.1 Planners and reward functions: decomposing the policy
The human has their reward function, and then follows a policy that presumably attempts to maximise
it. Therefore there is something that bridges between the reward function and the policy: a piece of
greater or lesser rationality that transforms knowledge of the reward function into a plan of action.
This bridge will be modeled as a plannerp:R! , a function that takes a reward and outputs a
policy. This planner encodes all the rationality, irrationality, and biases of the human. Let Pbe the
set of planners. The human is therefore deﬁned by a planner-reward pair(p;R)2PR . Similarly,
(p;R)withp(R) =is adecomposition of the policy . The task of the agent is to ﬁnd a ‘good’
decomposition of the human policy \_.
3.2 Compatible pairs and evidence
The agent can observe the human’s behaviour and infer their policy from that. In order to simplify
the problem and separate out the effect of the agent’s learning, we will assume the agent has perfect
knowledge of the human policy \_and of the environment M. At this point, the agent cannot learn
anything by observing the human’s actions, as it can already perfectly predict these.
Then a pair (p;R)is deﬁned to be compatible with \_, ifp(R) = \_— thus that pair is a possible
candidate for decomposing the human policy into the human’s planner and reward function.
6The setting is only chosen for notational convenience: it also emulates discrete POMDPs, non-
Markovianness (eg by encoding the whole history in the state) and pseudo-random policies.
3
4 Irrationality-based unidentiﬁability
Unidentiﬁability of the reward is a well-known problem in IRL [Ng and Russell, 2000]. Amin and
Singh [2016] categorise the problem into representational andexperimental unidentiﬁability. The
former means that adding a constant to a reward function or multiplying it with a positive scalar
does not change what is optimal behavior. This is unproblematic as rescaling the reward function
doesn’t change the preference ordering. The latter can be resolved by observing optimal policies in a
whole class of MDPs which contains all possible transition dynamics. We complete this framework
with a third kind of identiﬁability, which arises when we observe suboptimal agents. This kind of
unidentiﬁability is worse as it cannot necessarily be resolved by observing the agent in many tasks.
In fact, it can lead to almost arbitrary regret.
4.1 Weak No Free Lunch: unidentiﬁable reward function and half-maximal regret
The results in this section show that without assumptions about the rationality of the human, all
attempts to optimise their reward function are essentially futile. Everitt et al. [2017] work in a similar
setting as we do: in their case, a corrupted version of the reward function is observed. The problem
our case is that a ‘corrupted’ version \_of an optimal policy
\_Ris observed and used as information
to optimise for the ideal reward \_R. A No Free Lunch result analogous to theirs applies in our case;
both resemble the No Free Lunch theorems for optimisation [Wolpert and Macready, 1997].
More philosophically, this result is as an instance of the well-known is-ought problem from meta-
ethics. Hume [1888] argued that what ought to be (here, the human’s reward function) can never be
concluded from what is(here, behavior) without extra assumptions. Equivalently, the human reward
function cannot be inferred from behavior without assumptions about the planning algorithm p. In
probabilistic terms, the likelihood P(jR) =P
p2PP(jR;p)P(p)is undeﬁned without P(p). As
shown in Section 5 and Section 5.2, even a simplicity prior on pandRwill not help.
4.1.1 Unidentiﬁable reward functions
Firstly, we note that compatibility ( p(R) = \_), puts no restriction on R, and few restrictions on p:
Theorem 1. For all2andR2R, there exists a p2P such thatp(R) =.
For allp2P and2in the image of p, there exists an Rsuch thatp(R) =.
Proof. Trivial proof: deﬁne the planner7pas mapping all ofRto; thenp(R) =. The second
statement is even more trivial, as is in the image of p, so there must exist Rwithp(R) =.
4.1.2 Half-maximal regret
The above shows that the reward function cannot be constrained by observation of the human, but
what about the expected long-term value? Suppose that an agent is unsure what the actual human
reward function is; if the agent itself is acting in an MDP/R, can it follow a policy that minimises the
possible downside of its ignorance?
This is prevented by a recent No Free Lunch theorem. Being ignorant of the reward function one
should maximise is equivalent of having a corrupted reward channel with arbitrary corruption. In that
case, Everitt et al. [2017] demonstrated that whatever policy the agent follows, there is a R2R
for whichis half as bad as the worst policy the agent could have followed. Speciﬁcally, let V
R(s)
be the expected return of reward function Rfrom states, given that the agent follows policy . If
was the optimal policy for R, then this can be written as V
R(s). The regret of forRatsis given by
the difference:
Reg(;R)(s) =V
R(s)V
R(s):
Then Everitt et al. [2017] demonstrates that for any ,
max
R2RReg(;R)(s)1
2
max
02;R2RReg(0;R)(s)
:
So for any compatible (p;R) = \_, we cannot rule out that maximizing Rleads to at least half of the
worst-case regret.
7This is the ‘indifferent’ planner pof subsubsection 5.1.1.
4
5 Simplicity of degenerate decompositions
Like many No Free Lunch theorems, the result of the previous section is not surprising given there are
no assumptions about the planning algorithm. No Free Lunch results are generally avoided by placing
a simplicity prior on the algorithm, dataset, function class or other object [Everitt et al., 2014]. This
amounts to saying algorithms can beneﬁt from regularisation. This section is dedicated to showing
that, surprisingly, simplicity does not solve the No Free Lunch result.
Our simplicity measure is minimum description length of an object, deﬁned as Kolmogorov com-
plexity [Kolmogorov, 1965], the length of the shortest program that outputs a string describing the
object. This is the most general formalization of Occam’s razor we know of [Vitanyi and Li, 1997].
Appendix A explores how the results extend to other measures of complexity, such as those that
include computation time. We start with informal versions of our main results.
Theorem 2 (Informal simplicity theorem) .Let( \_p;\_R)be a ‘reasonable’ planner-reward pair that
captures our judgements about the biases and rationality of a human with policy \_= \_p(\_R). Then
there are degenerate planner-reward pairs, compatible with \_, of lower complexity than ( \_p;\_R), and
a pair ( \_p0;\_R)of similar complexity to ( \_p;\_R), but with opposite reward function.
There are a few issues with this theorem as it stands. Firstly, simplicity in algorithmic information
theory is relative to the computer language (or equivalently Universal Turing Machine) Lused [Ming
and Vit ´anyi, 2014, Calude, 2002], and there exists languages in which the theorem is clearly false:
one could choose a degenerate language in which ( \_p;\_R)is encoded by the string ‘ 0’, for example,
and all other planner-reward pairs are of extremely long length. What constitutes a ‘reasonable’
language is a long-standing open problem, see Leike et al. [2017] and M ¨uller [2010]. For any pair of
languages, complexities differ only by a constant, the amount required for one language to describe
the other, but this constant can be arbitrarily large.
Nevertheless, this section will provide grounds for the following two semi-formal results:
Proposition 3. If\_is a human policy, and Lis a ‘reasonable’ computer language, then there exists
degenerate planner-reward pairs amongst the pairs of lowest complexity compatible with \_.
Proposition 4. If\_is a human policy, and Lis a ‘reasonable’ computer language with ( \_p;\_R)a
compatible planner-reward pair, then there exist a pair ( \_p0;\_R)of comparable complexity to ( \_p;\_R),
but opposite reward function.
The last part of Theorem 2, the fact that any ‘reasonable’ ( \_p;\_R)is expected to be of higher complexity,
will be addressed in Section 6.
5.1 Simple degenerate pairs
The argument in this subsection will be that 1) the complexity of \_is close to a lower bound on
any pair compatible with it and 2) degenerate decompositions are themselves close to this bound.
The ﬁrst statement follows because for any decomposition (p;R)compatible with \_, the map
(p;R)7!p(R) = \_will be a simple one, adding little complexity. And if a compatible pair (p0;R0)
can be from \_with little extra complexity, then it too will have a complexity close to the minimal
complexity of any other pair compatible with it. Therefore we will ﬁrst produce three degenerate
pairs that can be simply constructed from \_.
5.1.1 The degenerate pairs
We can deﬁne the trivial constant reward function 0, and the greedy planner pg. The greedy planner
pgacts by taking the action that maximises the immediate reward in the current state and the next
action. Thus8pg(R)(s) = argmaxaR(s;a). We can also deﬁne the anti-greedy planner pg, with
pg(R)(s) = argminaR(s;a). In general, it will be useful to deﬁne the negative of a planner:
Deﬁnition 5. Ifp:R! is a planner, the planner pis deﬁned byp(R) =p(R).
For any given policy , we can deﬁne the indifferent plannerp, which maps any reward function to
. We can also deﬁne the reward function R, so thatR(s;a) = 1 if(s) =a, andR(s;a) = 0
otherwise. The reward function Ris deﬁned to be the negative of R. Then:
8Recall thatpgis a planner, pg(R)is a policy, so pg(R)can be applied to states, and pg(R)(s)is an action.
5
Lemma 6. The pairs (p;0),(pg;R), and (pg;R)are all compatible with .
Proof. Since the image pis,p(0) =. Now,R(s;a)>0iff(s) =a, hence for all s:
pg(R)(s) = argmax
aR(s;a) =(s);
sopg(R) =. Thenpg(R) =pg((R)) =pg(R) =, by Deﬁnition 5.
5.1.2 Complexity of basic operations
We will look the operations that build the degenerate planner-reward pairs from any compatible pair:
1. For any planner p,f1(p) = (p;0)as a planner-reward pair.
2. For any reward function R,f2(R) = (pg;R).
3. For any planner-reward pair (p;R),f3(p;R) =p(R).
4. For any planner-reward pair (p;R),f4(p;R) = (p;R).
5. For any policy ,f5() =p.
6. For any policy ,f6() =R.
These will be called the basic operations, and there are strong arguments that reasonable computer
languages should be able to express them with short programs. The operation f1, for instance, is
simply appending the ﬂat trivial 0,f2appends a planner deﬁned by the simple9search operator
argmax ,f3applies a planner to the object — a reward function — that the planner naturally acts on,
f4is a double negation, while f5andf6are simply described in subsubsection 5.1.1.
From these basic operations, we can deﬁne three composite operations that map any compatible
planner-reward pair to one of the degenerate pairs (the element F4=f4is useful for later deﬁnitions).
Thus deﬁne
F=fF1=f1f5f3; F2=f2f6f3; F3=f4f2f6f3; F4=f4g:
For any \_-compatible pair (p;R)we haveF1(p;R) = (p\_;0),F2(p;R) = (pg;R\_), andF3(p;R) =
(pg;R\_)(see the proof of Proposition 7).
LetKLdenote Kolmogorov complexity in the language L: the shortest algorithm in Lthat generates
a particular object. We deﬁne the F-complexity of Las
max
(p;R);Fi2FKL(Fi(p;R))KL(p;R):
Thus theF-complexity of Lis how much the Fipotentially increase10the complexity of pairs.
For a constant c0, this allows us to formalise what we mean by Lbeing ac-reasonable language
forF: that theF-complexity of Lis at mostc. A reasonable language is a c-reasonable language for
acthat we feel is intuitively low enough.
5.1.3 Low complexity of degenerate planner-reward pairs
To formalise the concepts ‘of lowest complexity’, and ‘of comparable complexity’, choose a constant
c0, then (p;R)and(p0;R0)are of ‘comparable complexity’ if
jjKL(p;R)KL(p0;R0)jjc:
For a setSPR , the pair (p;R)2Sis amongst the lowest complexity in Sif
jjKL(p;R)min
(p0;R0)2SKL(p0;R0)jjc;
thusKLis within distance cof the minimum complexity element of S. Now formalize Proposition 3:
9In most standard computer languages, argmax just requires a for-loop, a reference to R, a comparison
with a previously stored value, and possibly the storage of a new value and the current action.
10F-complexity is non-negative: F4F4is the identity, so that KL(F4(p;R))KL(p;R) =
(KL(F4(F4(p;R))KL(F4(p;R)), meaning that max (p;R);F4KL(F4(p;R))KF(p;R)must be non-
negative; this is a reason to include F4in the deﬁnition of F.
6
Proposition 7. If\_is the human policy, cdeﬁnes a reasonable measure of comparable complexity,
andLis ac-reasonable language for F, then the degenerate planner-reward pairs (p\_;0),(pg;R\_),
and(pg;R\_)are amongst the pairs of lowest complexity among the pairs compatible with \_.
Proof. By Lemma 6, (p\_;0),(pg;R\_), and (pg;R\_)are compatible with \_. By the deﬁnitions
of thefiandFi, for(p;R)compatible with \_,f3((p;R)) =p(R) = \_and hence
F1(p;R) =f1f5( \_) =f1(p\_) = (p\_;0);
F2(p;R) =f2f6( \_) =f2(R\_) = (pg;R\_);
F3(p;R) =f4F2(p;R) = (pg;R\_):
Now pick (p;R)to be the simplest pair compatible with \_. SinceLisc-reasonable for F,
KL(p\_;0)c+KL(p;R). Hence (p\_;0)is of lowest complexity among the pairs compatible with
\_; the same argument applies for the other two degenerate pairs.
5.2 Negative reward
If( \_p;\_R)is compatible with \_, then so is (\_p;\_R) =f4( \_p;\_R) =F4( \_p;\_R). This immediately
implies the formalisation of Proposition 4:
Proposition 8. If\_is a human policy, cdeﬁnes a reasonable measure of comparable complexity, L
is ac-reasonable language for F, and ( \_p;\_R)is compatible with \_, then (\_p;\_R)is of comparable
complexity to ( \_p;\_R).
So complexity fails to distinguish between a reasonable human reward function and its negative.
6 The high complexity of the genuine human reward function
Section 5 demonstrated that there are degenerate planner-reward pairs close to the minimum com-
plexity among all pairs compatible with \_. This section will argue that any reasonable pair ( \_p;\_R)
is unlikely to be close to this minimum, and is therefore of higher complexity than the degenerate
pairs. Unlike simplicity, reasonable decomposition cannot easily be formalised. Indeed, a formaliza-
tion would likely already solve the problem, yielding an algorithm to maximize it. Therefore, the
arguments in this section are mostly qualitative.
We use reasonable to mean ‘compatible with human judgements about rationality’. Since we do
not have direct access to such a decomposition, the complexity argument will be about showing the
complexity of these human judgements. This argument will proceed in three stages:
1. Any reasonable ( \_p;\_R)is of high complexity, higher than it may intuitively seem to us.
2.Even given \_, any reasonable ( \_p;\_R)involves a high number of contingent choices. Hence
any given ( \_p;\_R)has high information (and thus high complexity), even given \_.
3. Past failures to ﬁnd a simple ( \_p;\_R)derived from \_are evidence that this is tricky.
6.1 The complexity of human (ir)rationality
Humans make noisy and biased decisions all the time. Though noise is important [Kahneman et al.,
2016], many biases, such as anchoring bias, overconﬁdence, planning fallacies, and so on, affect
humans in a highly systematic way; see Kahneman and Egan [2011] for many examples.
Many people may feel that they have a good understanding of rationality, and therefore assume that
assessing the (ir)rationality of any particular decision is not a complicated process. But an intuition
for bias does not translate into a process for establishing a ( \_p;\_R).
Consider the anchoring bias deﬁned in Ariely et al. [2004], where irrelevant information — the last
digits of social security numbers — changed how much people were willing to pay for goods. When
deﬁning a reasonable ( \_p;\_R), it does not sufﬁce to be aware of the existence of anchoring bias11, but
11The fact that many cognitive biases have only been discovered recently argue against people having a good
intuitive grasp of bias and rationality, as do people’s persistent bias blind spots [Scopelliti et al., 2015].
7
one has to precisely quantify the extent of the bias — why does anchoring bias seem to be stronger
for chocolate than for wine, for instance? And why these precise percentages and correlations, and
not others? And can people’s judgment tell which people are more or less susceptible to anchoring
bias? And can one quantify the bias for a single individual, rather than over a sample?
Any given ( \_p;\_R)can quantify the form and extent of these biases by computing objects like the regret
function Reg( \_p;\_R)(s) := Reg( \_p(\_R);\_R)(s) =V
\_R(s)V\_p(\_R)
\_R(s), which measures the divergence
between the expected value of the actual and optimal human policies12. Thus any given ( \_p;\_R)
— which contains the information to compute quantities like Reg( \_p;\_R)(s)or similar measures of
bias13, in every state — carries a high amount of numerical information about bias, and hence a high
complexity.
Since humans do not easily have access to this information, this implies that human judgement of
irrationality is subject to Moravec’s paradox [Moravec, 1988]. It is similar to, for example, social
skills: though it seems intuitively simple to us, it is highly complex to deﬁne in algorithmic terms.
Other authors have argued directly for the complexity of human values, from ﬁelds as diverse as
computer science, philosophy, neuroscience, and economics [Minsky, 1984, Bostrom, 2014, Glimcher
et al., 2009, Muehlhauser and Helm, 2012, Yudkowsky, 2011].
6.2 The contingency of human judgement
The previous section showed that reasonable ( \_p;\_R)carry large amounts of information/complexity,
but the key question is whether it requires information additional to that in \_. This section will show
that even when \_is known, there are many contingent choices that need to be made to deﬁne any
speciﬁc reasonable ( \_p;\_R). Hence any given ( \_p;\_R)contains a large amount of information beyond
that in \_, and hence is of higher complexity.
Reasons to believe that human judgement about reasonable ( \_p;\_R)contains many contingent choices:
There is a variability of human judgement between cultures. When Miller [1984] compared
American and Indian assessments of the same behaviours, they found systematically different
explanations for them14Basic intuitions about rationality also vary between cultures [Nisbett
et al., 2001, Br ¨uck, 1999].
There is a variability of human judgement within a single culture. When Slovic and Tversky
[1974] analysed the “Allais Paradox”, they found that different people gave different answers
as to what the rational behaviour was in their experiments.
There is evidence of variability of human judgement within the same person. Slovic and
Tversky [1974] further attempted to argue for the rationality of one of the answers. This
sometimes resulted in the participant sometimes changing their minds, and contradicting
their previous assessment of rationality.
There is a variability of human judgement for the same person assessing their own values,
caused by differences as trivial as question ordering [Schuman and Ludwig, 1983]. So
human meta-judgement, of own values and rationality, is also contingent and variable.
People have partial bias blind spots around their own biases [Scopelliti et al., 2015].
Thus if a human is following policy \_, a decomposition ( \_p;\_R)would provide additional information
about the cultural background of the decomposer, their personality within their culture, and even
about the past history of the decomposer and how the issue is being presented to them. Those last
pieces prevents us from ‘simply’ using the human’s own assessment of their own rationality, as that
assessment is subject to change and re-interpretation depending on their possible histories.
12To exactly quantify the anchoring bias above, we could use a regret function that contrasts \_with the same
policy, but where the decision is optimal for one turn only (rather than for all turns, as in standard regret).
13In constrast, regret for the degenerate planner-reward pairs is trivial. Reg(p\_;0)andReg(pg;R\_)are
identically zero — in the second case, since pg(R\_)is actually optimal for R\_, getting the maximal possible
reward — while (pg;R\_)has a regret that is identically 1at each step.
14“Results show that there were cross-cultural and developmental differences related to contrasting cultural
conceptions of the person [...] rather than from cognitive, experiential, and informational differences [...].”
8
6.3 The search for human rationality models
One ﬁnal argument that there is no simple algorithm for going from \_to( \_p;\_R): many have tried
and failed to ﬁnd such an algorithm. Since the subject of human rationality has been a major one for
several thousands of years, the ongoing failure is indicative — though not a proof — of the difﬁculties
involved. There have been many suggested philosophical avenues for ﬁnding such a reward (such as
reﬂective equilibrium [Rawls, 1971]), but all have been underdeﬁned and disputed.
The economic concept of revealed preferences [Samuelson, 1948] is the most explicit, using the
assumption of rational behaviour to derive human preferences. This is an often acceptable approx-
imation, but can be taken too far: failure to take achieve an achievable goal does not imply that
failure was desired. Even within the conﬁnes of economics, it has been criticised by behavioural
economics approaches, such as prospect theory [Kahneman and Tversky, 2013] — and there are
counter-criticisms to these.
Using machine learning to deduce the intentions and preferences of humans is in its infancy, but we
can see non-trivial real-world examples, even in settings as simple as car-driving [Lazar et al., 2018].
Thus to date, neither humans nor machine learning have been able to ﬁnd simple ways of going from
\_to( \_p;\_R), nor any simple and explicit theory for how such a decomposition could be achieved. This
suggests that ( \_p;\_R)is a complicated object, even if \_is known. In conclusion:
Conjecture 9 (Informal complexity proposition) .If\_is a human policy, and Lis a ‘reasonable’
computer language with ( \_p;\_R)a ‘reasonable’ compatible planner-reward pair, then the complexity of
( \_p;\_R)is not close to minimal amongst the pairs compatible with \_.
7 Conclusion
We have shown that some degenerate planner-reward decompositions of a human policy have near-
minimal description length and argued that decompositions we would endorse do not. Hence, under
the Kolmogorov-complexity simplicity prior, a formalization of Occam’s Razor, the posterior would
endorse degenerate solutions. Previous work has shown that noisy rationality is too strong an
assumption as it does not account for bias; we tried the weaker assumption of simplicity, strong
enough to avoid typical No Free Lunch results, but it is insufﬁcient here.
This is no reason for despair: there is a large space to explore between these two extremes. Our hope
is that with some minimal assumptions about planner and reward we can infer the rest with enough
data. Staying close to agnostic is desirable in some settings: for example, a misspeciﬁed model of
the human reward function can lead to disastrous decisions with high conﬁdence [Milli et al., 2017].
Anonymous [2019] makes a promising ﬁrst try — a high-dimensional parametric planner is initialized
to noisy rationality and then adapts to ﬁt the behavior of a systematically irrational agent.
How can we reconcile our results with the fact that humans routinely make judgments about the
preferences and irrationality of others? And, that these judgments are often correlated from human
to human? After all, No Free Lunch applies to human as well as artiﬁcial agents. Our result shows
that they must be using shared priors, beyond simplicity, that are not learned from observations.
We call these normative assumptions because they encode beliefs about which reward functions are
more likely and what constitutes approximately rational behavior. Uncovering minimal normative
assumptions would be an ideal way to build on this paper; Appendix C shows one possible approach.
Acknowledgments.
We wish to thank Laurent Orseau, Xavier O’Rourke, Jan Leike, Shane Legg, Nick Bostrom, Owain
Evans, Jelena Luketina, Tom Everrit, Jessica Taylor, Paul Christiano, Eliezer Yudkowsky, Stuart
Russell, Dylan Hadﬁeld-Menell, and Anders Sandberg, Adam Gleave, Rohin Shah, among many
others. This work was supported by the Alexander Tamas programme on AI safety research, the
Leverhulme Trust, and the Machine Intelligence Research Institute.
9
References
Pieter Abbeel and Andrew Y Ng. Apprenticeship Learning via Inverse Reinforcement Learning.
2004.
Eric Allender. When worlds collide: Derandomization, lower bounds, and kolmogorov complexity.
InInternational Conference on Foundations of Software Technology and Theoretical Computer
Science , pages 1–15. Springer, 2001.
Kareem Amin and Satinder Singh. Towards Resolving Unidentiﬁability in Inverse Reinforcement
Learning. 2016.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man ´e.
Concrete Problems in AI Safety. 2016.
Anonymous. Inferring reward functions from demonstrators with unknown biases. In Submitted to
International Conference on Learning Representations , 2019. URL https://openreview.net/
forum?id=rkgqCiRqKQ . under review.
Dan Ariely, George Loewenstein, and Drazen Prelec. Arbitrarily coherent preferences. The psychology
of economic decisions , 2:131–161, 2004.
Nick Bostrom. Superintelligence: Paths, dangers, strategies . Oxford University Press, 2014.
Abdeslam Boularias, Jens Kober, and Jan Peters. Relative Entropy Inverse Reinforcement Learning,
2011.
Joanna Br ¨uck. Ritual and rationality: some problems of interpretation in european archaeology.
European journal of archaeology , 2(3):313–344, 1999.
Cristian Calude. Information and randomness : an algorithmic perspective . Springer, 2002.
Chris Cundy and Daniel Filan. Exploring hierarchy-aware inverse reinforcement learning. arXiv
preprint arXiv:1807.05037 , 2018.
Owain Evans, Andreas Stuhlmueller, and Noah D. Goodman. Learning the Preferences of Ignorant,
Inconsistent Agents. Thirtieth AAAI Conference on Artiﬁcial Intelligence , 2015a.
Owain Evans, Andreas Stuhlm ¨uller, and Noah D Goodman. Learning the preferences of bounded
agents. NIPS Workshop on Bounded Optimality , pages 16–22, 2015b.
Tom Everitt and Marcus Hutter. Avoiding wireheading with value reinforcement learning. In
International Conference on Artiﬁcial General Intelligence , pages 12–22. Springer, 2016.
Tom Everitt, Tor Lattimore, and Marcus Hutter. Free Lunch for optimisation under the universal
distribution. In Proceedings of the 2014 IEEE Congress on Evolutionary Computation, CEC 2014 ,
pages 167–174, 2014.
Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. Reinforcement
Learning with a Corrupted Reward Channel. 2017.
Paul W Glimcher, Colin F Camerer, Ernst Fehr, and Russell A Poldrack. Neuroeconomics: Decision
making and the brain, 2009.
Thomas L. Grifﬁths, Falk Lieder, and Noah D. Goodman. Rational Use of Cognitive Resources:
Levels of Analysis Between the Computational and the Algorithmic. Topics in Cognitive Science ,
7(2):217–229, 2015.
Dylan Hadﬁeld-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. Cooperative Inverse
Reinforcement Learning. arXiv:1606.03137 [cs] , 2016.
Dylan Hadﬁeld-Menell, Smitha Milli, Stuart J Russell, Pieter Abbeel, and Anca Dragan. Inverse
reward design. In Advances in Neural Information Processing Systems , pages 6749–6758, 2017.
10
Andreas Hula, P. Read Montague, and Peter Dayan. Monte Carlo Planning Method Estimates
Planning Horizons during Interactive Social Exchange. PLOS Computational Biology , 11(6):
e1004254, 2015.
David Hume. Treatise on Human Nature Ed Selby-bigge, L a . 1888.
Daniel Kahneman and Patrick Egan. Thinking, fast and slow , volume 1. Farrar, Straus and Giroux
New York, 2011.
Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. In
Handbook of the fundamentals of ﬁnancial decision making: Part I , pages 99–127. World Scientiﬁc,
2013.
Daniel Kahneman, Andrew M Rosenﬁeld, Linnea Gandhi, and Tom Blaser. Noise: How to overcome
the high, hidden cost of inconsistent decision making. Harvard business review , 94(10):38–46,
2016.
Andrei N Kolmogorov. Three approaches to the quantitative deﬁnition oﬁnformation’. Problems of
information transmission , 1(1):1–7, 1965.
Daniel A Lazar, Kabir Chandrasekher, Ramtin Pedarsani, and Dorsa Sadigh. Maximizing road
capacity using cars that inﬂuence people. arXiv preprint arXiv:1807.04414 , 2018.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro Ortega, Tom Everitt, Andrew Lefrancq, Laurent
Orseau, and Shane Legg. Ai safety gridworlds. arXiv preprint arXiv:1711.09883 , 2017.
Leonid A Levin. Randomness conservation inequalities; information and independence in mathemat-
ical theories. Information and Control , 61(1):15–37, 1984.
Richard L Lewis, Andrew Howes, and Satinder Singh. Computational Rationality: Linking Mech-
anism and Behavior Through Bounded Utility Maximization. Topics in Cognitive Science , 6:
279–311, 2014.
Joan G Miller. Culture and the development of everyday social explanation. Journal of personality
and social psychology , 46(5):961, 1984.
Smitha Milli, Dylan Hadﬁeld-Menell, Anca Dragan, and Stuart Russell. Should robots be obedient?
arXiv preprint arXiv:1705.09990 , 2017.
LI Ming and Paul MB Vit ´anyi. Kolmogorov complexity and its applications. Algorithms and
Complexity , 1:187, 2014.
Marvin Minsky. Afterword to Vernor Vinge’s novel, “True names. ” Unpublished manuscript . 1984.
URL http://web.media.mit.edu/ ~minsky/papers/TrueNames.Afterword.html .
Hans Moravec. Mind children: The future of robot and human intelligence . Harvard University Press,
1988.
Luke Muehlhauser and Louie Helm. The singularity and machine ethics. In Singularity Hypotheses ,
pages 101–126. Springer, 2012.
Markus M ¨uller. Stationary algorithmic probability. Theoretical Computer Science , 411(1):113–130,
2010.
Andrew Ng and Stuart Russell. Algorithms for inverse reinforcement learning. Proceedings of the
Seventeenth International Conference on Machine Learning , pages 663–670, 2000.
Richard E Nisbett, Kaiping Peng, Incheol Choi, and Ara Norenzayan. Culture and systems of thought:
holistic versus analytic cognition. Psychological review , 108(2):291, 2001.
Nathan D Ratliff, J Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In
Proceedings of the 23rd international conference on Machine learning - ICML ’06 , pages 729–736,
2006.
11
John Rawls. A Theory of Justice . Cambridge, Massachusetts: Belknap Press, 1971. ISBN 0-674-
00078-1.
Stuart Russell. Rationality and Intelligence: A Brief Update. In Fundamental Issues of Artiﬁcial
Intelligence , pages 7–28. 2016.
Stuart Russell, Daniel Dewey, and Max Tegmark. Research Priorities for Robust and Beneﬁcial
Artiﬁcial Intelligence. AI Magazine , 36(4):105, 2015.
Paul A Samuelson. Consumption theory in terms of revealed preference. Economica , 15(60):243–253,
1948.
J¨urgen Schmidhuber. The speed prior: a new simplicity measure yielding near-optimal computable
predictions. In International Conference on Computational Learning Theory , pages 216–228.
Springer, 2002.
Howard Schuman and Jacob Ludwig. The norm of even-handedness in surveys as in life. American
Sociological Review , pages 112–120, 1983.
Irene Scopelliti, Carey K Morewedge, Erin McCormick, H Lauren Min, Sophie Lebrecht, and
Karim S Kassam. Bias blind spot: Structure, measurement, and consequences. Management
Science , 61(10):2468–2486, 2015.
Kyriacos Shiarlis, Joao Messias, and Shimon Whiteson. Rapidly exploring learning trees. In
Proceedings - IEEE International Conference on Robotics and Automation , pages 1541–1548,
2017.
Paul Slovic and Amos Tversky. Who accepts savage’s axiom? Behavioral science , 19(6):368–373,
1974.
Amos Tversky and Daniel Kahneman. Judgment under Uncertainty: Heuristics and Biases. In Utility,
Probability, and Human Decision Making , pages 141–162. Springer Netherlands, Dordrecht, 1975.
Paul MB Vitanyi and Ming Li. An introduction to Kolmogorov complexity and its applications ,
volume 34. Springer Heidelberg, 1997.
David H. Wolpert and William G. Macready. No free lunch theorems for optimization. IEEE
Transactions on Evolutionary Computation , 1(1):67–82, 1997.
Eliezer Yudkowsky. Complex value systems in friendly ai. In International Conference on Artiﬁcial
General Intelligence , pages 388–393. Springer, 2011.
Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum Entropy Inverse
Reinforcement Learning. In AAAI Conference on Artiﬁcial Intelligence , pages 1433–1438, 2008.
12