Learning the Preferences of Ignorant, Inconsistent Agents
Owain Evans
University of OxfordAndreas Stuhlm ¨uller
Stanford UniversityNoah D. Goodman
Stanford University
Abstract
An important use of machine learning is to learn what
people value. What posts or photos should a user be
shown? Which jobs or activities would a person ﬁnd
rewarding? In each case, observations of people’s past
choices can inform our inferences about their likes and
preferences. If we assume that choices are approxi-
mately optimal according to some utility function, we
can treat preference inference as Bayesian inverse plan-
ning. That is, given a prior on utility functions and some
observed choices, we invert an optimal decision-making
process to infer a posterior distribution on utility func-
tions. However, people often deviate from approximate
optimality. They have false beliefs, their planning is
sub-optimal, and their choices may be temporally in-
consistent due to hyperbolic discounting and other bi-
ases. We demonstrate how to incorporate these devia-
tions into algorithms for preference inference by con-
structing generative models of planning for agents who
are subject to false beliefs and time inconsistency. We
explore the inferences these models make about pref-
erences, beliefs, and biases. We present a behavioral
experiment in which human subjects perform prefer-
ence inference given the same observations of choices
as our model. Results show that human subjects (like
our model) explain choices in terms of systematic de-
viations from optimal behavior and suggest that they
take such deviations into account when inferring pref-
erences.
Keywords: Bayesian learning, cognitive biases, preference
inference
Introduction
The problem of learning a person’s preferences from obser-
vations of their choices features prominently in economics
(Hausman 2011), in cognitive science (Baker, Saxe, and
Tenenbaum 2011; Ullman et al. 2009), and in applied ma-
chine learning (Jannach et al. 2010; Ermon et al. 2014). To
name just one example, social networking sites use a per-
son’s past behavior to select what stories, advertisements,
and potential contacts to display to them. A promising ap-
proach to learning preferences from observed choices is to
Copyright c
2016, Association for the Advancement of Artiﬁcial
Intelligence (www.aaai.org). All rights reserved.invert a model of rational choice based on sequential deci-
sion making given a real-valued utility function (Russell and
Norvig 1995). This approach is known as Inverse Reinforce-
ment Learning (Ng and Russell 2000) in an RL setting and
asBayesian Inverse Planning (Baker, Saxe, and Tenenbaum
2009) in the setting of probabilistic generative models.
This kind of approach usually assumes that the agent
makes optimal decisions up to “random noise” in action se-
lection (Kim et al. 2014; Zheng, Liu, and Ni 2014). How-
ever, human deviations from optimality are more systematic.
They result from persistent false beliefs, sub-optimal plan-
ning, and from biases such as time inconsistency and fram-
ing effects (Kahneman and Tversky 1979). If such devia-
tions are modeled as unstructured errors, we risk mistaken
preference inferences. For instance, if an agent repeatedly
fails to choose a preferred option due to a systematic bias,
we might conclude that the option is not preferred after all.
Consider someone who smokes every day while wishing to
quit and viewing their actions as regrettable. In this situa-
tion, a model that has good predictive performance might
nonetheless fail to identify what this person values.
In this paper, we explicitly take into account structured
deviations from optimality when inferring preferences. We
construct a model of sequential planning for agents with in-
accurate beliefs and time-inconsistent biases (in the form
of hyperbolic discounting). We then do Bayesian inference
over this model to jointly infer an agent’s preferences, be-
liefs and biases from sequences of actions in a simple
Gridworld-style domain.
To demonstrate that this algorithm supports accurate pref-
erence inferences, we ﬁrst exhibit a few simple cases where
our model licenses conclusions that differ from standard ap-
proaches, and argue that they are intuitively plausible. We
then test this intuition by asking impartial human subjects to
make preference inferences given the same data as our al-
gorithm. This is based on the assumption that people have
expertise in inferring the preferences of others when the do-
main is simple and familiar from everyday experience. We
ﬁnd that our algorithm is able to make the same kinds of
inferences as our human judges: variations in choice are ex-
plained as being due to systematic factors such as false be-
liefs and strong temptations, not unexplainable error.
The possibility of false beliefs and cognitive biases means
that observing only a few actions often fails to identify a
single set of preferences. We show that humans recognize
this ambiguity and provide a range of distinct explanations
for the observed actions. When preferences can’t be identi-
ﬁed uniquely, our model is still able to capture the range of
explanations that humans offer. Moreover, by computing a
Bayesian posterior over possible explanations, we can pre-
dict the plausibility of explanations for human subjects.
Computational Framework
Our goal is to infer an agent’s preferences from observa-
tions of their choices in sequential decision problems. The
key question for this project is: how are our observations of
behavior related to the agent’s preferences? In more techni-
cal terms, what generative model (Tenenbaum et al. 2011)
best describes the agent’s approximate sequential planning
given some utility function? Given such a model and a prior
on utility functions, we could “invert” it (by performing
full Bayesian inference) to compute a posterior on what the
agent values.
The following section describes the class of models we
explore in this paper. We ﬁrst take an informal look at the
speciﬁc deviations from optimality that our agent model in-
cludes. We then deﬁne the model formally and show our im-
plementation as a probabilistic program , an approach that
clariﬁes our assumptions and enables easy exploration of de-
viations from optimal planning.
Deviations from optimality
We consider two kinds of deviations from optimality:
False beliefs and uncertainty Agents can have false or
inaccurate beliefs. We represent beliefs as probability dis-
tributions over states and model belief updates as Bayesian
inference. Planning for such agents has been studied in work
on POMDPs (Kaelbling, Littman, and Cassandra 1998). In-
ferring the preferences of such agents was studied in re-
cent work (Baker and Tenenbaum 2014; Panella and Gmy-
trasiewicz 2014). Here, we are primarily interested in the in-
teraction of false beliefs with other kinds of sub-optimality.
Temporal inconsistency Agents can be time-inconsistent
(also called “dynamically inconsistent”). Time-inconsistent
agents make plans that they later abandon. This concept has
been used to explain human behaviors such as procrastina-
tion, temptation and pre-commitment (Ainslie 2001), and
has been studied extensively in psychology (Ainslie 2001)
and in economics (Laibson 1997; O’Donoghue and Rabin
2000).
A prominent formal model of human time inconsistency
is the model of hyperbolic discounting (Ainslie 2001). This
model holds that the utility or reward of future outcomes is
discounted relative to present outcomes according to a hy-
perbolic curve. For example, the discount for an outcome
occurring at delay dfrom the present might be modeled
as a multiplicative factor1
1+d. The shape of the hyperbola
means that the agent takes $100 now over $110 tomorrow,
but would prefer to take $110 after 31 days to $100 after
30 days. The inconsistency shows when the 30th day comes
D2#
D1#start#end#
D2#
D1#start#end#
Naive PlannerSophisticated Planner
Vegetarian)Café)Donut)Chain)Store)Noodle)Shop)
D2D1D2D1Figure 1: Agents with hyperbolic discounting exhibit differ-
ent behaviors depending on whether they model their future
discounting behavior in a manner that is (a) Naive (left) or
(b) Sophisticated (right).
around: now, the agent switches to preferring to take the
$100 immediately.
This discounting model does not (on its own) determine
how an agent plans sequentially. We consider two kinds
of time-inconsistent agents. These agents differ in terms of
whether they accurately model their future choices when
they construct plans. First, a Sophisticated agent has a fully
accurate model of its own future decisions. Second, a Naive
agent models its future self as assigning the same (dis-
counted) values to options as its present self. The Naive
agent fails to accurately model its own time inconsistency.1
We illustrate Naive and Sophisticated agents with a deci-
sion problem that we later re-use in our experiments. The
problem is a variant of Gridworld where an agent moves
around the grid to ﬁnd a place to eat (Figure 1).
In the left pane (Figure 1a), we see the path of an agent,
Alice, who moves along the shortest path to the Vegetarian
Cafe before going left and ending up eating at Donut Store
D2. This behavior is sub-optimal independent of whether her
preference is for the Vegetarian Cafe or the Donut Store, but
can be explained in terms of Naive time-inconsistent plan-
ning. From her starting point, Alice prefers to head for the
Vegetarian Cafe (as it has a higher undiscounted utility than
the Donut Store). She does not predict that when close to
the Donut Store (D2), she will prefer to stop there due to
hyperbolic discounting.
The right pane (Figure 1b) shows what Beth, a Sophisti-
cated agent with similar preferences to Alice, would do in
the same situation. Beth predicts that, if she took Alice’s
route, she would end up at the Donut Store D2. So she in-
stead takes a longer route in order to avoid temptation. If
the longer route wasn’t available, Beth could not get to the
Vegetarian Cafe without passing the Donut Store D2. In this
case, Beth would either go directly to Donut Store D1, which
1The distinction and formal deﬁnition of Naive and Sophisti-
cated agents is discussed in O’Donoghue and Rabin (1999).
is slightly closer than D2 to her starting point, or (if utility
for the Vegetarian Cafe is sufﬁciently high) she would cor-
rectly predict that she will be able to resist the temptation.
Formal model deﬁnition
We ﬁrst deﬁne an agent with full knowledge and no time in-
consistency,2and then generalize to agents that deviate from
optimality.
We will refer to states s2S, actionsa2A, a determinis-
tic utility function U:SA!R, a stochastic action choice
functionC:S!A, and a stochastic state transition func-
tionT:SA!S. To refer to the probability that C(s)
returnsa, we useC(a;s).
Optimal agent: full knowledge, no discounting Like all
agents we consider, this agent chooses actions in proportion
to exponentiated expected utility ( softmax ):
C(a;s)/eEUs[a]
The noise parameter modulates between random choice
(= 0) and perfect maximization ( =1). Expected util-
ity depends on both current and future utility:
EUs[a] =U(s;a) +E
s0;a0[EUs0[a0]]
withs0T(s;a)anda0C(s0). Note that expected future
utility recursively depends on C—that is, on what the agent
assumes about how it will make future choices.
Time-inconsistent agent Now the agent’s choice and ex-
pected utility function are parameterized by a delay d, which
together with a constant kcontrols how much to discount fu-
ture utility:
C(a;s;d)/eEUs;d[a]
EUs;d[a] =1
1 +kdU(s;a) +E
s0;a0[EUs0;d+1[a0]]
withs0T(s;a). For the Naive agent, a0C(s0;d+ 1) ,
whereas for the Sophisticated agent, a0C(s0;0). When
we compute what the agent actually does in state s, we set
dto0. As a consequence, only the Sophisticated agent cor-
rectly predicts its future actions.3An implementation of the
Naive agent as a probabilistic program is shown in Figure 2.
Time-inconsistent agent with uncertainty We now relax
the assumption that the agent knows the true world state.
Instead, we use a distribution p(s)to represent the agent’s
belief about which state holds. Using a likelihood function
p(ojs), the agent can update this belief:
p(sjo)/p(s)p(ojs)
The agent’s choice and expected utility functions are now
parameterized by the distribution p(s)and the current ob-
servationo:
C(a;p(s);o;d)/eEUp(s);o;d[a]
2This is the kind of agent assumed in the standard setup of an
MDP (Russell and Norvig 1995)
3This foresight allows the Sophisticated agent to avoid tempting
states when possible. If such states are unavoidable, the Sophisti-
cated agent will choose inconsistently.var agent =function (state, delay){
return Marginal(
function (){
var action = uniformDraw(actions)
var eu = expUtility (state, action, delay)
factor(alpha *eu)
return action
})
}
var expUtility =function (state, action, delay){
if(isFinal(state)){
return 0
}else {
var u = 1/(1 + k *delay) *utility(state, action)
return u + Expectation( function (){
var nextState = transition(state, action)
var nextAction = sample( agent (nextState, delay+1))
return expUtility (nextState, nextAction, delay+1)
})
}
}
Figure 2: We specify agents’ decision-making processes as
probabilistic programs. This makes it easy to encode ar-
bitrary biases and decision-making constraints. When au-
tomated inference procedures invert such programs to in-
fer utilities from choices, these constraints are automati-
cally taken into account. Note the mutual recursion between
agent andexpUtility : the agent’s reasoning about fu-
ture expected utility includes a (potentially biased) model of
its own decision-making.
To compute expected utility, we additionally take the expec-
tation over states. Now EUp(s);o;d[a]is deﬁned as:
E
sp(sjo)1
1 +kdU(s;a) +E
s0;o0;a0
EUp(sjo);o0;d+1[a0]
withs0T(s;a),o0p(ojs0)anda0C(p(sjo);o0;d+
1)(for the Naive agent) or a0C(p(sjo);o0;0)(for the
Sophisticated agent).
Inferring preferences We deﬁne a space of possible
agents based on the dimensions described above (utility
functionU, priorp(s), discount parameter k, noise param-
eter). We additionally let Ybe a variable for the agent’s
type, which ﬁxes whether the agent discounts at all, and if
so, whether the agent is Naive or Sophisticated. So, an agent
is deﬁned by a tuple := (p(s);U;Y;k; ), and we per-
form inference over this space given observed actions. The
posterior joint distribution on agents conditioned on action
sequencea0:Tis:
P(ja0:T)/P(a0:Tj)P() (1)
The likelihood function P(a0:Tj)is given by the multi-
step generalization of the choice function Ccorresponding
to. For the prior P(), we use independent uniform pri-
ors on bounded intervals for each of the components. In the
following, “the model” refers to the generative process that
0.02.55.07.510.0
0.02.55.07.510.0
Utility for DonutUtility for Vegetarian
Naive
012
481216
Utility for VegetarianDiscount strength (k)
Sophisticated
0.00670.01830.04980.13530.36791.0000
23456
Utility for NoodleP("Noodle shop open")
Sophisticated
Figure 3: Given data cor-
responding to Figure 1,
the model infers a joint
posterior distribution
on preferences, beliefs
and other agent prop-
erties (such as discount
strength) that reveals
relations between differ-
ent possible inferences
from the data. The darker
a cell, the higher its
posterior probability.
involves a prior on agents and a likelihood for choices given
an agent.
Agents as probabilistic programs
We implemented the model described above in the prob-
abilistic programming language WebPPL (Goodman and
Stuhlm ¨uller 2014). WebPPL provides automated inference
over functional programs that involve recursion. This means
that we can directly translate the recursions above into pro-
grams that represent an agent and the world simulation used
for expected utility calculations. All of the agents above can
be captured in a succinct functional program that can easily
be extended to capture other kinds of sub-optimal planning.
Figure 2 shows a simpliﬁed example (including hyperbolic
discounting but not uncertainty over state).
For the Bayesian inference corresponding to Equation 1
we use a discrete grid approximation for the continuous vari-
ables (i.e. for U,p(s),kand) and perform exact inference
using enumeration with dynamic programming.
Model inferences
We now demonstrate that the model described above can in-
fer preferences, false beliefs and time inconsistency jointly
from simple action sequences similar to those that occur fre-
quently in daily life. We later validate this intuition in our ex-
periments, where we show that human subjects make infer-
ences about the agent that are similar to those of our model.
Example 1: Inference with full knowledge We have pre-
viously seen how modeling agents as Naive and Sophisti-
cated might predict the action sequences shown in Figures1a and 1b respectively. We now consider the inference prob-
lem. Given that these sequences are observed, what can be
inferred about the agent? We assume for now that the agent
has accurate beliefs about the restaurants and that the two
Donut Stores D1 and D2 are identical (with D1 closer to the
starting point).4We model each restaurant as having an im-
mediate utility (received on arriving at the restaurant) and a
delayed utility (received one time-step after). This interacts
with hyperbolic discounting, allowing the model to repre-
sent options that are especially “tempting” when they can be
obtained with a short delay.
For the Naive episode (Figure 1a) our model infers that ei-
ther softmax noise is very high or that the agent is Naive (as
explained for Alice above). If the agent is Naive, the utility
of the Vegetarian Cafe must be higher than the Donut Store
(otherwise, the agent wouldn’t have attempted to go to the
Cafe), but not too much higher (or the agent wouldn’t give
in to temptation, which it in fact does). This relationship is
exhibited in Figure 3 (top left), which shows the model pos-
terior for the utilities of the Donut Store and Vegetarian Cafe
(holding ﬁxed the other agent components Y,k, and).
Example 2: Inference with uncertainty In realistic set-
tings, people do not have full knowledge of all facts rele-
vant to their choices. Moreover, an algorithm inferring pref-
erences will itself be uncertain about the agent’s uncer-
tainty. What can the model infer if it doesn’t assume that
the agent has full knowledge? Consider the Sophisticated
episode (Figure 1b). Suppose that the Noodle Shop is closed,
and that the agent may or may not know about this. This cre-
ates another possible inference, in addition to Sophisticated
avoidance of temptation and high noise: The agent might
prefer the Noodle Shop and might not know that it is closed.
This class of inferences is shown in Figure 3 (bottom): When
the agent has a strong prior belief that the shop is open, the
observations are most plausible if the agent also assigns high
utility to the Noodle Shop (since only then will the agent at-
tempt to go there). If the agent does not believe that the shop
is open, the Noodle Shop’s utility does not matter—the ob-
servations have the same plausibility either way.
In addition, the model can make inferences about the
agent’s discounting behavior (Figure 3 right): When utility
for the Vegetarian Cafe is low, the model can’t explain the
data well regardless of discount rate k(since, in this case,
the agent would just go to the Donut Store directly). The
data is best explained when utility for the Vegetarian Cafe
and discount rate are in balance—since, if the utility is very
high relative to k, the agent could have gone directly to the
Vegetarian Cafe, without danger of giving in to the Donut
Store’s temptation.
Example 3: Inference from multiple episodes Hyper-
bolic discounting leads to choices that differ systematically
from those of a rational agent with identical preferences. A
time-inconsistent agent might choose one restaurant more
often than another, even if the latter restaurant provides more
4In Experiment 2, we allow the utilities for D1 and D2 to be
different. See row 3 of Figure 6 and the “Preference” entry for So-
phisticated in Figure 7.
Vegetarian)Café)Donut)Chain)Store)Noodle)Shop)
!!D2!
D1!start!end!
ROAD!CLOSED!D2#
D1#end#
D2#
start#end#
D3#Episode 1Episode 2Episode 3D2D1D2D1D2D3Figure 4: The observations in Experiment 3 show the Donut Chain Store being chosen twice and the Vegetarian Cafe once.
utility in total. Our model is able to perform this kind of in-
ference. Figure 4 shows the same agent choosing in three
different episodes. While the agent chooses the Donut Store
two out of three times, our model assigns posterior probabil-
ity 0.59 (+/- 0.05 for 95% CI) that the agent prefers the Veg-
etarian Cafe over the Donut Store. As we decrease the prior
probability of high softmax noise, this posterior increases
beyond 0.59. By contrast, a model without time inconsis-
tency infers a preference for the Donut Store, and has to ex-
plain Episode 2 in Figure 4 in terms of noise, which leads to
high-entropy predictions of future choices.
Experiments with Human Subjects
We have shown that, given short action sequences, our
model can infer whether (and how) an agent is time-
inconsistent while jointly inferring appropriate utilities. We
claim that this kind of inference is familiar from every-
day life and hence intuitively plausible. This section pro-
vides support for this claim by collecting data on the infer-
ences of human subjects. In our ﬁrst two experiments, we
ask subjects to explain the behavior in Figures 1a and 1b.
This probes not just their inferences about preferences, but
also their inferences about biases and false beliefs that might
have inﬂuenced the agent’s choice.
Experiment 1: Inference with full knowledge
Experiment 1 corresponds to Example 1 in the previous sec-
tion (where the agent is assumed to have full knowledge).
Two groups of subjects were shown Figures 1a and 1b, hav-
ing already seen two prior episodes showing evidence of a
preference for the Vegetarian Cafe over the other restaurants.
People were then asked to judge the plausibility of different
explanations of the agent’s behavior in each episode.5
Results are shown in Figure 5. In both Naive (Figure 1a)
and Sophisticated (1b) conditions, subjects gave the high-
est ratings to explanations involving giving in to temptation
(Naive) or avoiding temptation (Sophisticated). Alternative
explanations suggested that the agent wanted variety (hav-
ing taking efﬁcient routes to the Vegetarian Cafe in previous
episodes) or that they acted purely based on a preference (for
a long walk or for the Donut Store).
5In a pilot study, we showed subjects the same stimuli and had
them write free-form explanations. In Experiment 1, subjects had
to judge four of the explanations that occurred most frequently in
this pilot.
1.01.52.02.53.0
Tempted by donut
Preference for varietyPrefers donut
Vegetarian too farAverage score for explanation
Naive
1.01.52.02.53.0
Avoid temptation
Preference for varietyPrefers walkTry noodle
Sophisticated
Figure 5: Explanations in Experiment 1 for the agent’s be-
havior in Figure 1a (Naive) and 1b (Sophisticated). Subjects
(n=120) knew that the agent has accurate knowledge, and
saw prior episodes providing evidence of a preference for
the Vegetarian Cafe. Subjects selected scores in f1;2;3g.
Experiment 2: Inference with uncertainty
Experiment 2 corresponds to Example 2 above. Subjects see
one of the two episodes in Figure 1 (with Figure 1a modiﬁed
so D1 and D2 can differ in utility and Figure 1b modiﬁed so
the Noodle Shop is closed). There is no prior information
about the agent’s preferences, and the agent is not known to
have accurate beliefs. We asked subjects to write explana-
tions for the agent’s behavior in the two episodes and coded
these explanations into four categories. Figure 6 speciﬁes
which formal agent properties correspond to which category.
While not all explanations correspond to something the
model can infer about the agent, the most common explana-
tions map cleanly onto the agent properties —few explana-
tions provided by people fall into the “Other” category (Fig-
ure 7). The model inferences in this ﬁgure show the marginal
likelihood of the observed actions given the corresponding
property of, normalized across the four property types. In
both the Naive and the Sophisticated case, the model and
people agree on what the three highest-scoring properties
are. Explanations involving false beliefs and preferences rate
more highly than those involving time inconsistency. This is
because, even if we specify whether the agent is Naive/So-
phisticated, the actions in Figure 1a/b are fairly unlikely—
Property Formalization Example explanation from our human subjects
Agent doesn’t know
Donut Store D1 is open.p(D1 = open)<0:15 “He decided he wanted to go to the Donut Store for lunch. He
did not know there was a closer location”
Agent falsely believes
Noodle Shop is open.p(N= open)>0:85 “He was heading towards the noodle shop ﬁrst, but when he got
there, it was closed, so he continued on the path and ended up
settling for ... vegetarian cafe.”
Agent prefers D2 to D1. U(D2)>U(D1) “He might also enjoy the second donut shop more than the ﬁrst”
Agent is Naive / Sophisti-
cated.Y= Naive=Soph: “He ... headed for the Vegetarian Cafe, but he had to pass by
the Donut shop on his way. The temptation was too much to
ﬁght, so he ended up going into the Donut Shop.”
Figure 6: Map from properties invoked in human explanations to formalizations in the model. The left column describes the
property. The center column shows how we formalized it in terms of the variables used to deﬁne an agent . The right column
gives an explanation (from our dataset of human subjects) that invokes this property.
Naive Sophisticated
0.00.10.20.30.40.5
(Avoid) temptationFalse beliefPreferenceOther
(Avoid) temptationFalse beliefPreferenceOtherFrequency/probability of explanation
SubjectType
People
Model
Figure 7: Explanations in Experiment 2 for the agent’s be-
havior in Figures 1a (Naive) and 1b (Sophisticated). Subjects
did not know whether the agent has accurate knowledge, and
did not see prior episodes. There were n=31 subjects (Naive)
and n=40 subjects (Sophisticated).
they require a narrow range of utility values, as illustrated in
Figure 3 (top left), which favors more speciﬁc explanations.
Experiment 3: Inference from multiple episodes
Following Example 3 above, subjects (n=50) saw the
episodes in Figure 4 and inferred whether the agent prefers
the Vegetarian Cafe or the Donut Store. Like the model, the
majority of subjects inferred that the agent prefers the Vege-
tarian Cafe. Overall, 54% (+/- 7 for 95% CI) inferred a pref-
erence of Vegetarian Cafe over the Donut Store, compared
to the 59% posterior probability assigned by the model.
Episode 2 (in which the agent does not choose the Donut
Store) is identical to the Sophisticated episode from Figure
1. Experiments 1 and 2 showed that subjects explain this
episode in terms of Sophisticated time-inconsistent plan-
ning. Together with Experiment 3, this suggests that sub-
jects use this inference about the agent’s planning to infer
the agent’s undiscounted preferences, despite having seen
the agent choose the Donut Store more frequently.Conclusion
AI systems have the potential to improve our lives by help-
ing us make choices that involve integrating vast amounts
of information or that require us to make long and elaborate
plans. For instance, such systems can recommend and ﬁlter
the information we see on social networks or music services
and can construct intricate plans for travel or logistics. For
these systems to live up to their promise, we must be willing
to delegate some of our choices to them—that is, we need
such systems to reliably act in accordance with our prefer-
ences and values. It can be difﬁcult to formally specify our
preferences in complex domains; instead, it is desirable to
have systems learn our preferences, just as learning in other
domains is frequently preferable to manual speciﬁcation.
This learning requires us to build in assumptions about
how our preferences relate to the observations the AI system
receives. As a starting point, we can assume that our choices
result from optimal rational planning given a latent utility
function. However, as our experiments with human subjects
show, this assumption doesn’t match people’s intuitions on
the relation between preferences and behavior, and we ﬁnd
little support for the simplistic model where what is chosen
most is inferred to be the most valued. We exhibited more re-
alistic models of human decision-making, which in turn sup-
ported more accurate preference inferences. By approaching
preference inference as probabilistic induction over a space
of such models, we can maintain uncertainty about prefer-
ences and other agent properties when the observed actions
are ambiguous.
This paper has only taken a ﬁrst step in the direction we
advocate. Two priorities for further work are applications to
more realistic domains and the development of alternatives
to using human preference inferences as a standard by which
to evaluate algorithms. The goal for this emerging subﬁeld
of AI is to make systems better able to support humans even
in domains where human values are complex and nuanced,
and where human choices may be far from optimal.
Acknowledgments
This work was supported by Future of Life Institute grant
2015-144846 (all authors) and by ONR grant N00014-13-1-
0788 (NG).
This material is based on research sponsored by DARPA
under agreement number FA8750-14-2-0009 (AS and NG).
The U.S. Government is authorized to reproduce and dis-
tribute reprints for Governmental purposes notwithstanding
any copyright notation thereon. The views and conclusions
contained herein are those of the authors and should not be
interpreted as necessarily representing the ofﬁcial policies
or endorsements, either expressed or implied, of DARPA or
the U.S. Government.
References
Ainslie, G. 2001. Breakdown of will . Cambridge University
Press.
Baker, C. L., and Tenenbaum, J. B. 2014. Modeling human
plan recognition using bayesian theory of mind.
Baker, C. L.; Saxe, R.; and Tenenbaum, J. B. 2009. Action
understanding as inverse planning. Cognition 113(3):329–
349.
Baker, C. L.; Saxe, R. R.; and Tenenbaum, J. B. 2011.
Bayesian theory of mind: Modeling joint belief-desire attri-
bution. In Proceedings of the thirty-second annual confer-
ence of the cognitive science society , 2469–2474.
Ermon, S.; Xue, Y .; Toth, R.; Dilkina, B.; Bernstein, R.;
Damoulas, T.; Clark, P.; DeGloria, S.; Mude, A.; Barrett, C.;
et al. 2014. Learning large-scale dynamic discrete choice
models of spatio-temporal preferences with application to
migratory pastoralism in east africa. In Meeting Abstract .
Goodman, N. D., and Stuhlm ¨uller, A. 2014. The Design and
Implementation of Probabilistic Programming Languages .
Hausman, D. M. 2011. Preference, value, choice, and wel-
fare. Cambridge University Press.
Jannach, D.; Zanker, M.; Felfernig, A.; and Friedrich, G.
2010. Recommender systems: an introduction . Cambridge
University Press.
Kaelbling, L. P.; Littman, M. L.; and Cassandra, A. R. 1998.
Planning and acting in partially observable stochastic do-
mains. Artiﬁcial intelligence 101(1):99–134.
Kahneman, D., and Tversky, A. 1979. Prospect theory: An
analysis of decision under risk. Econometrica: Journal of
the Econometric Society 263–291.
Kim, D.; Breslin, C.; Tsiakoulis, P.; Ga ˇsic, M.; Henderson,
M.; and Young, S. 2014. Inverse reinforcement learning
for micro-turn management. In Proceedings of the Annual
Conference of the International Speech Communication As-
sociation, INTERSPEECH , 328–332. International Speech
and Communication Association.
Laibson, D. 1997. Golden eggs and hyperbolic discounting.
The Quarterly Journal of Economics 443–477.
Ng, A. Y ., and Russell, S. J. 2000. Algorithms for inverse
reinforcement learning. In Icml, 663–670.
O’Donoghue, T., and Rabin, M. 1999. Doing it now or later.
American Economic Review 103–124.O’Donoghue, T., and Rabin, M. 2000. The economics of im-
mediate gratiﬁcation. Journal of Behavioral Decision Mak-
ing13(2):233.
Panella, A., and Gmytrasiewicz, P. 2014. Learning poli-
cies of agents in partially observable domains using bayesian
nonparametric methods. In AAMAS Workshop on Multiagent
Sequential Decision Making Under Uncertainty .
Russell, S., and Norvig, P. 1995. Artiﬁcial Intelligence: A
modern approach . Prentice-Hall.
Tenenbaum, J. B.; Kemp, C.; Grifﬁths, T. L.; and Goodman,
N. D. 2011. How to grow a mind: Statistics, structure, and
abstraction. science 331(6022):1279–1285.
Ullman, T.; Baker, C.; Macindoe, O.; Evans, O.; Goodman,
N.; and Tenenbaum, J. B. 2009. Help or hinder: Bayesian
models of social goal inference. In Advances in neural in-
formation processing systems , 1874–1882.
Zheng, J.; Liu, S.; and Ni, L. M. 2014. Robust bayesian in-
verse reinforcement learning with sparse behavior noise. In
Twenty-Eighth AAAI Conference on Artiﬁcial Intelligence .

Owain Evans
Andreas Stuhlmuller
Noah D. Goodman


Maximum Margin Planning

Maximum Margin Planning
Nathan D. Ratliﬀ ndr@ri.cmu.edu
J. Andrew Bagnell dbagnell@ri.cmu.edu
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA. 15213 USA
Martin A. Zinkevich maz@cs.ualberta.ca
Department of Computing Science, University of Alberta, Edmonton, AB T6G 2E1, Canada
Abstract
Imitation learning of sequential, goal-
directed behavior by standard supervised
techniques is often diﬃcult. We frame learn-
ing such behaviors as a maximum margin
structured prediction problem over a space
of policies. In this approach, we learn map-
pings from features to cost so an optimal pol-
icy in an MDP with these cost mimics the ex-
pert’s behavior. Further, we demonstrate a
simple, provably eﬃcient approach to struc-
tured maximum margin learning, based on
the subgradient method, that leverages ex-
isting fast algorithms for inference. Although
the technique is general, it is particularly rel-
evant in problems where A* and dynamic
programming approaches make learning poli-
cies tractable in problems beyond the limita-
tions of a QP formulation. We demonstrate
our approach applied to route planning for
outdoor mobile robots, where the behavior a
designer wishes a planner to execute is often
clear, while specifying cost functions that en-
gender this behavior is a much more diﬃcult
task.
1. Introduction
In “imitation learning” a learner attempts to mimic
an expert’s behavior or control strategy. In numerous
instances, notably within robotics (Pomerleau, 1989;
LeCun et al., 2006), supervised learning approaches
have been used to great eﬀect to learn mappings from
features to decisions. Long-range and goal-directed be-
havior, by contrast, has proven more diﬃcult to cap-
ture using these techniques.
Appearing in Proceedings of the 23rdInternational Con-
ference on Machine Learning , Pittsburgh, PA, 2006. Copy-
right 2006 by the author(s)/owner(s).In mobile robotics, the motivating application of our
imitation learning approaches, researchers often en-
courage long-horizon goal directed behavior by parti-
tioning an autonomy software into a “perception” sub-
system and a “planning” subsystem. The perception
system computes various models and features of an en-
vironment. For instance, perception might determine
the supporting surface, obstacles lying above this sur-
face, average color at various locations, density of ladar
returns, et cetera. Planning takes as input a cost-map
over the vehicle conﬁguration or state space (Hebert
et al., 1998) and computes a minimal risk (cost) path
through it that serves as a coherent sequence of deci-
sions for the robot over a long horizon. This approach
has proven powerful and lies at the heart of many au-
tonomous mobile robotic systems in both indoor and
outdoor environments.
Unfortunately, the leap from perception’s model to
costs for a planner is often a diﬃcult one. In prac-
tice, it is often done by hand-designed heuristics that
are painstakingly validated by observing the result-
ing robot behavior. In this work, we propose a novel
method whereby we attempt to automate the mapping
from perception features to costs. We do so by fram-
ing the problem as one of supervised learning to take
advantage of examples given by an expert describing
desired behavior. In essence, we leverage the fact that
the desired behavior is often quite clear to a human
designer while specifying costs that engender this be-
havior is a much more diﬃcult task, particularly when
it involves simultaneously tweaking a large set of knobs
that map features to costs.
Learning to plan so as to mimic a teacher’s behav-
ior may be cast as a structured prediction problem
over the space of policies. The learner’s goal is to
take example input features and example policies or
trajectories through the state space (e.g. paths) and
learn to predict the same sequence of decisions. De-
cisions at each state must be coordinated and their
Maximum Margin Planning
costs balanced to achieve a satisfactory global plan-
ning strategy that implements the desired behavior.
In this approach, our goal is to learn mappings from
features to cost functions so an optimal policy in a
Markov Decision Problem with this cost function im-
itates the expert’s behavior. We demonstrate that we
can learn such mappings in the structured large mar-
gin framework. (Taskar et al., 2005)
The key contributions of this work are three-fold.
First, we demonstrate a novel method for learning to
plan. Second, we demonstrate an eﬃcient, simple ap-
proach to structured maximum-margin classiﬁcation
(in both online and batch settings) that is applica-
ble whenever a fast specialized algorithm is available
to compute the minimum of the loss-augmented cost
function. This approach demonstrates linear conver-
gence when used in batch settings and is applicable to
large problems where other Quadratic Programming
techniques are not. We develop an online theory for
Structured Maximum Margin and show that our al-
gorithm achieves sublinear regret that scales inversely
with the margin and without a dependence on the size
of the policy we must learn. Finally, we demonstrate
empirically that our method is particularly applicable
to problems of relevance in mobile robotics. We out-
line our current research directions and other natural
applications of Max Margin Planning in the conclu-
sions (Section 5). We also discuss related approaches
to imitation learning and structured classiﬁcation in
Section 5.
2. Preliminaries
We model the planning problem with discrete Markov
Decision Processes. Let xandaindex the state and ac-
tion spacesXandA, respectively; and let p(y|x, a) and
sdenote, respectively, the transition probablities and
initial state distribution. A discount factor on rewards
(if any) is absorbed into the transition probabilities.
Our reward functions1are learned from supervised ex-
amples to produce policies that mimic demonstrated
behavior. We repeatedly make use of the linear pro-
gramming formulation of the MDP problem (Puter-
man, 1994), denoting by v∈Vthe primal variables of
the value function, and by µ∈Gthe dual state-action
frequency counts. We consider here only stationary
policies; the generalization is straightforward.
The input to our algorithm is a set of training instances
D={(Xi,Ai, pi, Fi, yi,Li)}n
i=1. A training instance
consists of an MDP with transition probabilities pi,
1In some deciplines is it more common and suitable to
describe problems in terms of costs rather than rewards.
In what follows, the term “cost” is used interchangeably to
mean “negative reward”.state-action pairs ( xi, ai)∈ X i×A iover which d-
dimensional feature vectors are place in the form of
ad×|X||A| feature matrix Fi.yidenotes the desired
trajectory (or full policy) that exempliﬁes behavior we
hope to match. We often consider the alternate repre-
sentation for such a trajectory in terms of µi, a vector
of state-action frequency counts. Note that since we
are encoding the policies in terms of the dual state-
action frequency counts, this formulation can attempt
to learn to match either trajectories or entire policies.
This will be described in detail below. We also discuss
the role of the loss vector libelow.
We use subscripts to denote indexing by training in-
stance, and reserve superscripts for indexing into vec-
tors. (E.g. µx,a
iis the expected state-action frequency
for state xand action aof example i.) Note that we
will sometimes write D={(Xi,Ai, pi, fi, yi,Li)}n
i=1≡
{(Xi,Ai,Gi, Fi, µi, li)}n
i=1, using fi(y) to denote vector
of expected feature counts Fiµof the ith example. It is
useful for some problems, such as robot path planning,
to imagine representing the features as a set of maps
and example paths through those maps. For instance,
one feature map might indicate the elevation at each
state, another the slope, and a third the presence of
vegetation.
The learner attempts to ﬁnd a linear mapping2of
these features to rewards so that for each problem in-
stance the best policy over the resulting reward func-
tion µ∗= arg maxµ∈G iwTFiµis “close” to the demon-
strated policy µi. The notion of closeness is deﬁned
by a loss-function L:Y×Y → R+between so-
lutions. In our work this function is of the form
L(y, yi) =Li(y) = lT
iµ. Intuitively, a loss vector
li∈ R|X||A|
+ is placed over state-action pairs that de-
ﬁnes for each state-action pair how much the learner
pays for failing to match the behavior of an example
policy yias the cumulative loss of the learned policy
through this MDP. (See Figure 4)
The maximum margin principle builds in a degree of
robustness where we attempt to make the supplied
solution yilook signiﬁcantly better than alternative
policies. In particular, we adopt the structured max-
margin framework and attempt to make yibetter than
any other solution ˆ yby a margin that scales with the
size of the loss of ˆ y; we wish to ensure that the cor-
rect policy looks much more attractive than very bad
policies.
2Although space doesn’t permit full details, both formu-
lations we use for learning later straightfowardly generalize
to handle nonlinearity through the use of kernelization.
Maximum Margin Planning
2.1. Loss-functions
The framework thus described admits any loss func-
tion that factors over state-action pairs. A natural
loss function in the case of deterministic acyclic path
planning is the count of the number of states that the
planner visits that the teacher did not. We have found
somewhat better performance by smoothing this loss
function so that nearby paths are also admissible. In a
general MDP, we might penalize choosing diﬀerent ac-
tions from the teacher at any states the teacher reaches
or penalize reaching states the teacher chooses not to
enter. We also assume that L(y, yi)≥0.
2.2. Quadratic Programming Formulation
Given a training set D={(Xi,Ai, pi, fi, yi,Li)}n
i=1,
the structured large margin criteria (Taskar et al.,
2005) implies we are solving the following quadratic
program:3
min
w,ζ i1
2/bardblw/bardbl2+γ
n/summationdisplay
iβiζq
i (1)
s.t.∀i wTfi(yi) +ζi≥max
y∈Y iwTfi(y) +Li(y) (2)
The intuition behind these constraints is that we al-
low only weight vectors for which the example policies
have higher expected reward than all other policies by
a margin that scales with the loss. The slack variables
ζipermit violations of these constraints for a penalty
that scales with the hyperparameter γ≥0.βi>0
are data dependent scalars that can be used for nor-
malization when the examples are of diﬀerent lengths.
q∈{1,2}distinguishes between L1andL2slack penal-
ties commonly found in the literature (Tsochantaridis
et al., 2005).
If we consider the case for which both fi(·) andLi(·)
are linear in the state-action frequencies µas described
above (i.e. they factor over state-action pairs), the
maximum margin problem becomes
min
w,ζ i1
2/bardblw/bardbl2+γ
n/summationdisplay
iβiζq
i (3)
s.t.∀i wTFiµi+ζi≥max
µ∈G iwTFiµ+lT
iµ(4)
where µ∈G iexpresses the Bellman-ﬂow constraints
for each MDP, namely that µ≥0 satisﬁes:
/summationdisplay
x,aµx,api(x/prime|x, a) +sx/prime
i=/summationdisplay
aµx/prime,a
The nonlinear, convex constraints in Equation 4 can
be transformed into a compact set of linear constraints
3Unless stated otherwise, /bardbl./bardbldenotes the L2norm.(Taskar et al., 2005; Taskar et al., 2003) by computing
the dual of the right hand side of each yielding:
∀i wTFiµi+ζi≥min
v∈VisT
iv (5)
where v∈Viare the value-functions that satisfy the
Bellman primal constraints:
∀x, a vx≥(wTFi+li)x,a+/summationdisplay
x/primepi(x/prime|x, a)vx/prime(6)
By combining the constraints together we can write
one compact quadratic program:
min
w,ζ i,vi1
2/bardblw/bardbl2+γ
n/summationdisplay
iβiζq
i (7)
s.t.∀i wTFiµi+ζi≥sT
ivi (8)
∀i, x, a vx
i≥(wTFi+li)x,a+/summationdisplay
x/primepi(x/prime|x, a)vx/prime
i(9)
This results provides a represenation of what we call
the Maximum Margin Planning (MMP) problem as a
compact quadratic program. Note that the number
of constraints scales linearly with state-action pairs
and training examples. While oﬀ-the-shelf quadratic
programming software can be applied at this point to
directly optimize this program, we provide an alter-
native formulation in Section 3 that allows us improve
signiﬁcantly over this via the utilization of subgradient
methods.
We consider additional useful loss functions in Section
4, where we detail examples relevant to path planning
problems.
3. Eﬃcient Optimization
In practice, solving the quadratic program in Equa-
tion 9 is at least as hard as solving the linear program-
ming formulation of a single MDP. While this can be
an appropriate strategy for a class of MDPs, it is gen-
erally appreciated that for many problems there exist
specially designed algorithms such as policy iteration
and A* that can solve particular classes of MDPs both
theoretically and empirically more rapidly. Developing
eﬃcient specialized algorithms, and especially lever-
aging existing inference algorithms, is an open prob-
lem for general structured maximum margin problems.
(Taskar et al., 2005). We present a simple approach
based on the subgradient method (Shor, 1985) that al-
lows the use of fast maximization algorithms (e.g. fast
planners) for a provably eﬃcient learning strategy.
The ﬁrst step is to transform the optimization pro-
gram into a “hinge-loss” form. The hinge-loss view
is a common way to understand the maximum mar-
gin problem and relate it to other methods like logis-
tic regression and AdaBoost. This view comes from
Maximum Margin Planning
noting that the slack variables ζiare tight and thus
equal max µ∈G i(wTFi+lT
i)µ−wTFiµi. We can there-
fore move these constraints into the objective function,
simplifying the problem into a single cost function:
cq(w) =1
nn/summationdisplay
i=1βi/parenleftbigg
max
µ∈G i(wTFi+lT
i)µ−wTFiµi/parenrightbiggq
+
λ
2/bardblw/bardbl2(10)
where we have multiplied through byλ
γto make a
regularized risk functional interpretation more clear
(Rifkin & Poggio, 2003). Again, q∈{1,2}deﬁnes the
slack penalization.
This objective function is convex, but nondiﬀeren-
tiable. We can optimize it by utilizing a generaliza-
tion of gradient descent called the subgradient method
(Shor, 1985). A subgradient of a convex function
c:W→ Ratwis deﬁned as a vector gsuch that
∀w/prime∈W , gT(w/prime−w)≤c(w/prime)−c(w) (11)
Note that subgradients need not be unique, though at
points of diﬀerentiability, they necessarily agree with
the gradient. We denote the set of all subgradients of
c(·) at point wby∂c(w).
To compute the subgradient of c(w), we make use
of the following four well known properties: (1) sub-
gradient operators are linear; (2) the gradient is the
unique subgradient of a diﬀerentiable function; (3) de-
noting y∗= argmaxy[f(x, y)] for diﬀerentiable f(., y),
∇xf(x, y∗) is a subgradient of the piecewise diﬀeren-
tiable convex function max y[f(x, y)]; (4) an analogous
chain rule holds as expected. We are now equipped
to compute a subgradient gq
w∈∂c(w) of our objective
function (10):
gq
w=1
nn/summationdisplay
i=1qβi/parenleftbig
(wTFi+lT
i)µ∗−wTFiµi/parenrightbigq−1·
Fi∆wµi+λw (12)
where µ∗= arg max µ∈G i(wTFi+lT
i)µand ∆wµi=
µ∗−µi. This latter expression points out that, intu-
itively, the subgradient compares the state-action vis-
itation frequency counts between the example policy
and the optimal policy with respect to current reward
function wTFi.
Note that computing the subgradient requires solving
the problem µ∗= arg max µ∈G i(wTFi+lT
i)µfor each
MDP. This is precisely the problem of solving the par-
ticular MDP with the reward function wTFi+lT
i, and
can be eﬃcient implemented via a myriad of special-
ized algorithms. Algorithm 1 details the application
of the subgradient method to the Maximum Margin
Planning problem.Algorithm 1 Max Margin Planning
1:procedure MMP (Training set{Fi, µi, li}N
i=1,
Regularization parameter λ > 0, Stepsize se-
quence{αt}(learning rate), Iterations T)
2: t←1
3: w←0
4: while t≤Tdo
5: Compute optimal policy and state action
visitation frequencies µ∗,ifor each in-
put map for loss augmented cost map
(wTFi+lT
i).
6: Compute g∈∂c(w) as in Equation 12.
7: w←w−αtg
8: (Optional): Project won to any additional
constraints.
9: t←t+ 1
10: end while
11: return w
12:end procedure
The basic iterative update given gt∈∂c(wt) and αtis
wt+1=PW[wt−αtgt] (13)
wherePprojects wonto any problem speciﬁc (convex)
constraints we may impose on w.4
3.1. Guarantees in the Batch Setting
In the batch setting, this algorithm is one of a well
studied class of algorithms forming the subgradient
method (Shor, 1985).5Crucial to this method is
the choice of stepsize sequence {αt}; and convergence
guarantees vary accordingly. Our results are devel-
oped from (Nedic & Bertsekas, 2000) which analyzes
incremental subgradient algorithms, of which the sub-
gradient method is a special case.
Our results require a strong convexity assumption to
hold for the objective function. Given W ⊆ Rd, a
function f:W→ Risη-strongly convex if there exists
g:W→ Rdsuch that for all w, w/prime∈W:
f(w/prime)≥f(w) + ( g(w))T(w/prime−w) +η/bardblw/prime−w/bardbl2(14)
Theorem 1. Linear convergence of constant
stepsize sequence. Let the stepsize sequence {αt}
4It is actually suﬃcient that P be an approximate
projection operator that need only satisfy the inequality
∀w/prime∈ W ,/bardblPW[w]−w/prime/bardbl ≤ /bardbl w−w/prime/bardbl.
5The term “subgradent method” is used in lieu of “sub-
gradient descent” because the method is not technically
a descent method. Since the stepsize sequence is chosen
in advance, the objective value per iterate can, and often
does, increase.
Maximum Margin Planning
of Algorithm (1) be chosen as αt=α≤1
λ. Fur-
thermore, assume for a particular region of radius
Raround the minimum, ∀w, g∈∂c(w),/bardblg/bardbl ≤ C.
Then the algorithm converges at a linear rate to a
region of some minimum point x∗ofcbounded by
/bardblxmin−x∗/bardbl≤/radicalBig
αC2
λ≤C
λ.
Proof. (Sketch) By the strong convexity of cq(w) and
Proposition 2.4 of (Nedic & Bertsekas, 2000) we have
/bardblwt+1−w∗/bardbl2≤(1−αλ)t+1/bardblw0−w∗/bardbl2+αC2
λ
−→
t→∞αC2
λ≤C2
λ2
2
This theorem shows that we attain a linear conver-
gence rate under a suﬃciently small constant stepsize,
but that convergence is only to a region around the
minimum. Alternatively, we can choose a diminishing
stepsize rule of the form αt=r
tfort≥1, where ris
some positive constant that can be thought of as the
learning rate. Under this rule, Algorithm 1 is guaran-
teed to converge to the minimum, but only at a sublin-
ear rate under the above strong convexity assumption
(see (Nedic & Bertsekas, 2000), Proposition 2.8).
3.2. Optimization in an Online Setting
In contrast with many optimization techniques, the
subgradient method naturally extends from the batch
setting (as presented) to an online one. In the online
setting one imagines seeing several planning problems
on closely related domains: in particular, one may ob-
serve a domain, be required to plan a path for it, and
then observe the “correct” path (or observe the cor-
rections of the suggested path). At each time step i:
(1) We observeGiand Fi, (2) Select a weight vector
wiand using this compute a resulting path (3) Finally
we observe the true policy yi. Thus, we can deﬁne
ci(w) =λ
2/bardblw/bardbl2+ max µ∈G i(wTFi+li)µ−wTFiµito
be the strongly convex cost function (see Equation 10)
at time i, which we can compute given yi,Gi, and
Fi. This is now an online convex programming
problem (Zinkevich, 2003), to which we will apply
the extension of Greedy Projection by (Hazan et al.,
2006) where the learning rate is 1 /(iλ).
The loss we truly care about on round iis the plan-
ning loss,L(µi, σi), where σiis the policy we choose.
Space doesn’t permit a proof,6but the following may
be derived using tools from (Hazan et al., 2006):
Theorem 2. Sublinear regret for subgradi-
ent MMP. Assume that the features in each state
are bounded in norm by 1. Further, assume that
6See appendix of extended version of paper at http://
(removed for review).there is a w∗that with hindsight achieves for all i,
max µ∈G iwTFi(µ−µi) +lT
iµ= 0, then:
/summationdisplay
iL(µi, σi)≤1
λ(1 + ln n) +nλ/bardblw∗/bardbl2(15)
Choosing λ=√1+ln n
/bardblw∗/bardbl√n, then:
/summationdisplay
iL(µi, σi)≤/bardblw∗/bardbl/radicalbig
n(1 + ln n) (16)
Thus, if we know nand the achievable margin, our loss
grows only sublinearly in the number of time steps.
2
Further, in the case when there is no perfect w∗, we
can instead achieve a competitive ratio that scales in-
versely with the margin. Observe that Theorem 2 is
a result about the additive loss function Lspeciﬁcally.
If we attempted instead to consider, for instance, a
setting where we did not require more margin from
higher loss paths (e.g. 0/1 loss on paths) our bound
would be much weaker and scale with the size of the
domain.
3.3. Modiﬁcations for Acyclic Positive Costs
For inﬁnite horizon problems in acyclic domains A*
and its variants are generally the most eﬃcient route
to ﬁnding a good plan. Such domains require rewards
to be strictly negative (equivalently, costs must be
strictly positive), otherwise inﬁnite reward paths may
result. The strictness of this negativity is to ensure
the existence of an admissable heuristic.
Assuming Fi≥0 (element-wise), this can be imple-
mented via component-wise negativity constraints on
wor a set of constraints enforcing the negativity of the
reward for each state-action pair individually. Exact
projection onto the former can be implemented sim-
ply by setting the violated components of wto 0, and
an approximate projection onto the latter can be im-
plemented eﬃciently by iteratively projecting onto the
most violated constraint.
3.4. Incorporating Prior Knowledge
It is often important to be able to build in prior knowl-
edge about cost-functions that may improve learning
performance. One useful technique is to regularize the
solution about a prior belief on winstead of the 0 vec-
tor. Another is to have our loss function mark certain
state-action pairs as being poor choices: this forces our
algorithm to have large margin with respect to them.
Finally, we may incorporate domain knowledge in the
form of constaints on w: e.g., we may require that
a certain area state have at least double the cost of
another state. All of these are powerful methods to
Maximum Margin Planning
transfer expert knowledge to the learner in addition
to the use of training examples.
4. Experimental Results
To validating these concepts we focused on the practi-
cal problem of path planning using the batch learning
algorithm presented in section 3. In this setting, the
MDP can be viewed as a two-dimensional map dis-
cretized uniformely into an array of cells. Each cell
represents a particular location in the world and typ-
ical actions include moving from a given cell to one
of the eight neighboring cells. In all experiments, we
used A∗as our specialized planning algorithm, set
βi= 1//bardblµi/bardblq
1, chose q= 2, and used reasonable values
for regularization.
We ﬁrst exhibit the versatility of our algorthm in learn-
ing distinct concepts within a single domain. Diﬀering
example trajectories, demonstrated in one region of
a map, lead to a signiﬁcantly diﬀerent behavior in a
separate holdout region after learning. Figure 1 shows
qualitatively the results of this experiment. The be-
havior presented in the top row suggests a desire to
stay on the road, while that portrayed in the bottom
row embodies more clandestine needs. By column,
from left to right, the images depict the training exam-
ple presented to the algorithm, the learned cost map
on a holdout region after training, and the resulting
behavior produced by A∗over this region.7
For our second experiment, the data derived entirely
from laser range readings (ladar) over the region of in-
terest collected during an overhead helicopter sweep.8
A visualization of the raw data is depicted in Figure 3.
Figure 2 shows typical results from a holdout region.
The learned behavior (green) often matches well the
desired behavior (red). Even when the learner failed
to match the desired trajectory exactly, the learned
behavior adheres to the primary rules set forth im-
plicitly by the examples. Namely, the learner ﬁnds an
eﬃcient path that avoids buildings (white) and grassy
areas (gray) in lieu of roads.
Notice that the loss-augmented path (blue) in this ﬁg-
ure performs generally worse than the ﬁnal learned tra-
jectory. This is because loss-augmentation makes areas
of high loss more desirable than they would be in the
7The features used in this experiment were derived en-
tirely from a single overhead satellite image. We dis-
cretized the image into ﬁve distinct color classes and added
smoothed versions of the resulting features to propegate
proximity information.
8Raw features were computed from mean and standard
deviations from each of elevation, signal reﬂectance, hue,
saturation, and local ladar shape information (Vandapel
et al., 2004). Again, we added smoothed versions of the
raw features to utilize proximity information.
Figure 1. Demonstration of learning to plan based on satel-
lite color imagery. For a particular training/holdout region
pair, the top row trains the learner to follow the road while
the bottom row trains the learner to “hide” in the trees.
From left to right, the columns depict the single training
example presented, the learned cost map over the holdout
region, and the corresponding learned behavior over that
region. Cost scales with intensity.
ﬁnal learned map. Intuitively, if the learner is able to
perform well with respect to the loss-augmented cost
map, then it should perform even better without the
loss-augmentation; that is, the concept is learned with
margin .
For comparison, we attempted to learn similar behav-
ior using two alternative approaches to MMP. First,
we tried the reactive approach of directly learning from
examples a mapping that takes state features to next
actions as in (LeCun et al., 2006).9Unfortunately, the
resulting paths were rather poor matches to the train-
ing data. See Figure 3 for a typical example of a path
learned by the classiﬁer.
A somewhat more successful attempt was to try to
learn costs directly by a hand labeling of regions. This
provides dramatically more explicit information to the
learner than MMP requires: a trainer provided exam-
ples regions of low, medium, and high costs, based
upon (1) expert knowledge of the planner, (2) iter-
ated training and observation, and (3) the trainer had
prior knowledge of the cost maps found under MMP
batch learning on this data set.10Although cost maps
9We used the same training data, training Regularized
Least Squares classiﬁers (Rifkin & Poggio, 2003) to predict
which nearby state to transition to. It proved diﬃcult to
engineer good features here; our best results come from us-
ing the same local state features as MMP augmented with
distance and orientation to the goal. The learner typically
achieved between 0.7-0.85 prediction accuracy.
10The low cost examples came from the example paths
and the medium/high cost examples were supplied sepa-
rately. Low cost and high cost examples were chosen as
Maximum Margin Planning
Figure 2. See Figure 3 for data-set. Data shown are MMP learned cost maps (dark low cost) with a teacher supplied path
(red), loss-augmented path (blue), and ﬁnal learned path (green). These are learned results on a holdout set.
Figure 3. Left: the result of a next-action classiﬁer applied
super-imposed on a visualization of the second data-set.
Right: a cost map learned by manual training of a regres-
sion. The learned paths (green) in both cases are poor ap-
proximations of the training examples (not shown on left,
red on right).
given this extra information looked qualitivately cor-
rect, Figure 3 and Figure 4 demonstrate that the plan-
ning performance was signiﬁcantly inferior.
5. Related and Future Work
Two strands of work are most directly related to Max-
imum Margin Planning. One is work on Inverse Re-
inforcement Learning (IRL). The goal in IRL is to
observe an agent acting in an MDP (with unknown
reward function) and extract from the agent’s behav-
ior the reinforcement function it is attempting to op-
timize. Unfortunately, it is well known that standard
IRL is ill-posed: the zero reward function, for instance,
admits all policies as optimal ones. Nevertheless, there
have been some heuristic attempts to use IRL ideas
to similar eﬀect as MMP. One useful heuristic is to
have a learning algorithm try to match expected fea-
ture counts (in our language/summationtext
iF µi) (Abbeel & Ng,
minimum and maximum values for A*, respectively. Mul-
tiple medium cost levels were tried.2004) between the learner’s policy and the demon-
strated examples. This variant of IRL diﬀers from
MMP in that MMP is designed to allow the demon-
stration of policies from more than a single MDP: we
typically demonstrate examples over multiple feature
maps and with diﬀerent start and goal states, with
the aim of the learner to extrapolate to entirely new
maps and goals. This leads to signiﬁcantly diﬀerent
algorithmic approaches. The relation between IRL
and MMP is reminiscent of the distinction between
generative and discriminative learning: IRL feature
matching is designed for learning when we believe an
agent is acting (near-optimally) in an MDP and fur-
ther that we can (nearly) match feature expectations.
These assumptions, like those of generative models, are
strong ones: the ability to match feature expectations,
for instance, means that the algorithm’s behavior will
be near-optimal for every cost function linear in the
features simultaneously (i.e. for any reward function
wTF). MMP makes the weaker assumption that our
goal is to directly mimic output behavior and is agnos-
tic about a real underlying MDP or reward function.
MDPs here structure the output decisions and provide
a potentially natural class of experts that we attempt
to compete with.
The other strand of closely related work is that of
Structured Max-Margin optimization techniques. The
subgradient optimization presented here makes prac-
tical learning in problems where straightforward QP
techniques are intractable. It is often the case, that
path planning problems are quickly solved with MMP
that would be too large to even represent for a generic
QP solver. Recently, a number of other techniques
have been proposed to solve problems of these kinds,
including cutting plane (Tsochantaridis et al., 2005)
and extra-gradient techniques. (Taskar et al., 2006)
The latter is applicable where inference may be writ-
ten as a linear program and is also able to achieve
linear convergence rates. The subgradient method
Maximum Margin Planning
Figure 4. (Left) Visualization of inverted loss function (1 −
l(x)) for a training example path. (Right) Comparison
of holdout loss of MMP (by number of iterations) to a
regression method where a teacher hand-labeled costs. The
next-action classiﬁer approach had a much larger loss than
either method.
has the advantage of being applicable to any prob-
lems where loss augmented inference may be quickly
solved including by combinatorial methods. Further,
our algorithm extends naturally to the online case,
where sublinear regret bounds are available. It will
be interesting to compare these methods on problems
where they are both applicable. In recent work, (Du-
ame et al., 2006) has considered reinforcement learning
based approaches to structured classiﬁcation. Subgra-
dient methods for (unstructured) margin linear class-
siﬁcation were considered in (Zhang, 2004). (LeCun
et al., 1998) considers the use of gradient methods for
learning using decoding methods such as Viterbi; our
approach (if applied to sequence labeling) extends such
methods to use notions of structured maximum mar-
gin.
Our current research eﬀort is to take advantage of
the online behavior and apply the algorithm in a re-
planning scenario where new features are being gen-
erated continuously from on-board perception. Our
algorithm has natural applications in a variety of non-
robotic domains where planning is essential. We are
currently applying improved versions of the subgradi-
ent method, also able to leverage existing specialized
algorithms inference algorithms, to a variety of maxi-
mum margin learning problems.
Acknowledgements
We thank Omead Amidi, Boris Sofman, Tony Stentz, and
Nicolas Vandapel for their generous help with the experi-
mental work as well as valuable conversations with Geoﬀ
Gordon. The ﬁrst two authors gratefully acknowledge the
partial support of this research by the DARPA Learning
for Locomotion contract.
References
Abbeel, P., & Ng, A. Y. (2004). Apprenticeship learning via
inverse reinforcement learning. ICML ’04: Proceedings
of the twenty-ﬁrst international conference on Machine
learning .Duame, H., Langford, J., & Marcu, D. (2006). Search-
based structured prediction. In Preparation.
Hazan, E., Kalai, A., Kale, S., & Agarwal, A. (2006). Loga-
rithmic regret algorithms for online convex optimization.
To appear in COLT 2006.
Hebert, M., Stentz, A. T., & Thorpe, C. (1998). Mo-
bility planning for autonomous navigation of multiple
robots in unstructured environments. Proceedings of
ISIC/CIRA/ISAS Joint Conference .
LeCun, Y., Bottou, L., Bengio, Y., & Haﬀner, P. (1998).
Gradient-based learning applied to document recogni-
tion. Proceedings of the IEEE (pp. 2278–2324).
LeCun, Y., Muller, U., Ben, J., Cosatto, E., & Flepp, B.
(2006). Oﬀ-road obstacle avoidance through end-to-end
learning. In Advances in neural information processing
systems 18 .
Nedic, A., & Bertsekas, D. (2000). Convergence rate of in-
cremental subgradient algorithms. Stochastic Optimiza-
tion: Algorithms and Applications .
Pomerleau, D. (1989). Alvinn: An autonomous land vehi-
cle in a neural network. Advances in Neural Information
Processing Systems 1 .
Puterman, M. (1994). Markov decision processes: Discrete
stochastic dynamic programming . Wiley.
Rifkin, Y., & Poggio (2003). Regularized least squares
classiﬁcation. Advances in Learning Theory: Methods,
Models and Applications . IOS Press.
Shor, N. Z. (1985). Minimization methods for non-
diﬀerentiable functions . Springer-Verlag.
Taskar, B., Chatalbashev, V., Guestrin, C., & Koller, D.
(2005). Learning structured prediction models: A large
margin approach. Twenty Second International Confer-
ence on Machine Learning (ICML05) .
Taskar, B., Guestrin, C., & Koller, D. (2003). Max mar-
gin markov networks. Advances in Neural Information
Processing Systems (NIPS-14) .
Taskar, B., Lacoste-Julien, S., & Jordan, M. (2006). Struc-
tured prediction via the extragradient method. In Ad-
vances in neural information processing systems 18 .
Tsochantaridis, I., Joachims, T., Hofmann, T., & Altun,
Y. (2005). Large margin methods for structured and
interdependent output variables. Journal of Machine
Learning Research , 1453–1484.
Vandapel, N., Huber, D., Kapuria, A., & Hebert, M.
(2004). Natural terrain classiﬁcation using 3-d ladar
data. IEEE International Conference on Robotics and
Automation .
Zhang, T. (2004). Solving large scale linear prediction
problems using stochastic gradient descent algorithms.
Proceedings of ICML .
Zinkevich, M. (2003). Online convex programming and
generalized inﬁnitesimal gradient ascent. Proceedings
of the Twentieth International Conference on Machine
Learning .

Nathan D. Ratliff
J. Andrew Bagnell
Martin A. Zinkevich


Ambitious vs. narrow value learning

Suppose I’m trying to build an AI system that “learns what I want” and helps me get it. I think that people sometimes use different interpretations of this goal. At two extremes of a spectrum of possible interpretations:

* The AI learns my preferences over (very) long-term outcomes. If I were to die tomorrow, it could continue pursuing my goals without me; if humanity were to disappear tomorrow, it could rebuild the kind of civilization we would want; *etc.* The AI might pursue radically different subgoals than I would on the scale of months and years, if it thinks that those subgoals better achieve what I really want.
* The AI learns the narrower subgoals and instrumental values I am pursuing. It learns that I am trying to schedule an appointment for Tuesday and that I want to avoid inconveniencing anyone, or that I am trying to fix a particular bug without introducing new problems, *etc.* It does not make any effort to pursue wildly different short-term goals than I would in order to better realize my long-term values, though it may help me correct some errors that I would be able to recognize as such.

I think that many researchers interested in AI safety per se mostly think about the former. I think that researchers with a more practical orientation mostly think about the latter.

The ambitious approach
----------------------

The maximally ambitious approach has a natural theoretical appeal, but it also seems quite hard. It requires understanding human preferences in domains where humans are typically very uncertain, and where our answers to simple questions are often inconsistent, like how we should balance our own welfare with the welfare of others, or what kinds of activities we really want to pursue vs. enjoying in the moment. (It seems unlikely to me that there is a unified notion of “what I want” in many of these cases.) It also requires extrapolation to radically unfamiliar domains, where we will need to make decisions about issues like population ethics, what kinds of creatures do we care about, and unforeseen new technologies.

I have written about this problem, pointing out that it is unclear how you would solve it [even with an unlimited amount of computing power](https://medium.com/ai-control/the-easy-goal-inference-problem-is-still-hard-fad030e0a876). My impression is that most practitioners don’t think of this problem even as a long-term research goal — it’s a qualitatively different project without direct relevance to the kinds of problems they want to solve.

The narrow approach
-------------------

The narrow approach looks relatively tractable and well-motivated by existing problems. We want to build machines that helps us do the things we want to do, and to that end they need to be able to understand what we are trying to do and what instrumental values guide our behavior. To the extent that our “preferences” are underdetermined or inconsistent, we are happy if our systems at least do as well as a human, and make the kinds of improvements that humans would reliably consider improvements.

But it’s not clear that anything short of the maximally ambitious approach can solve the problem we ultimately care about. A sufficiently clever machine will be able to make long-term plans that are significantly better than human plans. In the long run, we will want to be able to use AI abilities to make these improved plans, and to generally perform tasks in ways that humans would never think of perform them — going far beyond correcting simple errors that can be easily recognized as such.

In defense of the narrow approach
=================================

I think that the narrow approach probably takes us much further than it at first appears. I’ve written about these arguments before, which are for the most part similar to the reasons that [approval-directed agents](https://medium.com/ai-control/model-free-decisions-6e6609f5d99e) or directly [mimicking human behavior](https://medium.com/ai-control/against-mimicry-6002a472fc42) might work, but I’ll quickly summarize them again:

Instrumental goals
------------------

Humans have many clear instrumental goals like “remaining in effective control of the AI systems I deploy,” “acquiring resources and other influence in the world,” or “better understanding the world and what I want.” A value learner may able to learn robust preferences like these and pursue those instrumental goals using all of its ingenuity. Such AI’s would not necessarily be at a significant disadvantage with respect to normal competition, yet the resources they acquired would remain under meaningful human control (if that’s what their users would prefer).

This requires learning robust formulations of concepts like “meaningful control,” but it does not require making inferences about cases where humans have conflicting intuitions, nor considering cases which are radically different from those encountered in training — AI systems can continue to gather training data and query their users even as the nature of human-AI interactions changes (if that’s what their users would prefer).

Process
-------

Even if we can’t infer human preferences over very distant objects, we might be able to infer human preferences well enough to guide a process of deliberation (real or hypothetical). Using the inferred preferences of the human could help eliminate some of the errors that a human would traditionally make during deliberation. Presumably these errors run counter to a deliberator’s short-term objectives, if those objectives are properly understood, and this judgment doesn’t require a direct understanding of the deliberator’s big-picture values.

This kind of error-correction could be used as a complement to other kinds of idealization, like providing the human a lot of time, allowing them to consult a large community of advisors, or allowing them to use automated tools.

Such a process of error-corrected deliberation could itself be used to provide a more robust definition of values or a more forward looking criterion of action, such as “an outcome/action is valuable to the extent that I would/did judge it valuable after extensive deliberation.”

Bootstrapping
-------------

By interacting with AI assistants, humans can potentially form and execute very sophisticated plans; if so, simply helping them achieve their short-term goals may be all that is needed. For some discussion of this idea, see [these](https://medium.com/ai-control/safe-automated-assistants-aa69edd46a57) [three](https://medium.com/ai-control/implementing-our-considered-judgment-6c715a239b3e) [posts](https://medium.com/ai-control/approval-directed-bootstrapping-5d49e886c14f).

Conclusion
==========

I think that researchers interested in scalable AI control have been too quick to dismiss “narrow” value learning as unrelated to their core challenge. Overall I expect that the availability of effective narrow value learning would significantly simplify the AI control problem even for superintelligent systems, though at the moment we don’t understand the relationship very well.

(Thanks to Andreas Stuhlmüller and Owain Evans for helpful discussion.)