Active Preference-Based Learning of Reward Functions.

Active Preference-Based
Learning of Reward Functions
Dorsa Sadigh, Anca D. Dragan, Shankar Sastry, and Sanjit A. Seshia
University of California, Berkeley, {dsadigh, anca, sastry, sseshia}@eecs.berkeley.edu
Abstract —Our goal is to efﬁciently learn reward functions
encoding a human’s preferences for how a dynamical system
should act. There are two challenges with this. First, in many
problems it is difﬁcult for people to provide demonstrations
of the desired system trajectory (like a high-DOF robot arm
motion or an aggressive driving maneuver), or to even assign
how much numerical reward an action or trajectory should
get. We build on work in label ranking and propose to
learn from preferences (or comparisons) instead: the person
provides the system a relative preference between two trajec-
tories. Second, the learned reward function strongly depends
on what environments and trajectories were experienced
during the training phase. We thus take an active learning
approach, in which the system decides on what preference
queries to make. A novel aspect of our work is the complexity
and continuous nature of the queries: continuous trajectories
of a dynamical system in environments with other moving
agents (humans or robots). We contribute a method for
actively synthesizing queries that satisfy the dynamics of
the system. Further, we learn the reward function from
a continuous hypothesis space by maximizing the volume
removed from the hypothesis space by each query. We assign
weights to the hypothesis space in the form of a log-concave
distribution and provide a bound on the number of iterations
required to converge. We show that our algorithm converges
faster to the desired reward compared to approaches that are
not active or that do not synthesize queries in an autonomous
driving domain. We then run a user study to put our method
to the test with real people.
I. Introduction
Reward functions play a central role in specifying how
dynamical systems should act: how an end-user wants
their assistive robot arm to move, or how they want their
autonomous car to drive. For many systems, end-users
have difﬁculty providing demonstrations of what they
want. For instance, they cannot coordinate 7 degrees of
freedom (DOFs) at a time [2], and they can only show
the car how they drive, not how they want the car to
drive [5]. In such cases, another option is for the system
to regress a reward function from labeled state-action
pairs, but assigning precise numeric reward values to
observed robot actions is also difﬁcult.
In this paper, we propose a preference-based approach
to learning desired reward functions in a dynamical
system. Instead of asking for demonstrations, or for the
value of the reward function for a sample trajectory (e.g.,
“rate the safety of this driving maneuver from 1 to 10”),
we ask people for their relative preference between two
sample trajectories (e.g., “is x1more safe or less safe than
x2?”).Active preference-based learning has been successfully
used in many domains [1, 6, 7, 14], but what makes
applying it to learning reward functions difﬁcult is the
complexity of the queries, as well as the continuous
nature of the underlying hypothesis space of possible
reward functions. We focus on dynamical systems with
continuous or hybrid discrete-continuous state. In this
setting, queries consist of two candidate continuous state
and action space trajectories that satisfy the system’s dy-
namics, in an environment or scenario that the learning
algorithm also needs to decide on, consisting of an initial
state and the trajectories of other agents in the scene.
Consider again the example of autonomous driving. In
this situation, a query would consist of two trajectories
for the car from some starting state among other cars
following their own trajectories.
Typically in preference-based learning, the queries are
actively selected by searching some discrete or sampled
set (e.g., [3, 9, 12, 13, 20, 21]). Our ﬁrst hypothesis is
that in our setting, the continuous and high-dimensional
nature of the queries renders relying on a discrete set
ineffective. Preference-based work for such spaces has
thus far collected data passively [18, 22]. Our second
hypothesis is that active generation of queries leads to
better reward functions faster.
We contribute an algorithm for actively synthesizing
queries from scratch. We do continuous optimization in
query space to maximize the expected volume removed
from the hypothesis space. We use the human’s response
to assign weights to the hypothesis space in the form of a
log-concave distribution , which provides an approximation
of the objective via a Metropolis algorithm that makes it
differentiable w.r.t. the query parameters. We provide a
bound on the number of iterations required to converge.
We compare our algorithm to non-active and non-
synthesis approaches to test our hypotheses. We use an
experimental setup motivated by autonomous driving,
and show that our approach converges faster to a desired
reward function. Finally, we illustrate the performance
of our algorithm in terms of accuracy of the reward
function learned through an in-lab usability study.
II. Problem Statement
Modeling Choices. Our goal is to model the behavior
and preferences of a human for how a dynamical system
should act. We denote this dynamical system that should
match human preferences by H, and it interacts with
other systems (robots) in an environment. We model the
overall system including H, and all the other agents as a
fully-observable dynamical system. The continuous state
of the environment x2Xincludes the state of Hand
all the other agents. We let uHdenote the continuous
control input ofH. For simplicity, we assume that there
is only one other agent (robot) Rlocally visible in the
environment, and we let uRbe its continuous control
input. The dynamics of the state changes based on the
actions of both agents through fHR:
xt+1=fHR(xt,uR,uH). (1)
We deﬁne a trajectory x2X, where x=
(x0,u0
R,u0
H), . . . ,(xN,uN
R,uN
H)is a ﬁnite horizon sequence
of states and actions of all agents. Here, Xis a set of all
feasible continuous trajectories. A feasible trajectory is
one that satisﬁes the dynamics of HandR. We ﬁrst
parameterize the preference reward function as a linear
combination of a set of features:
rH(xt,ut
R,ut
H) =w>f(xt,ut
R,ut
H), (2)
where wis a vector of weights for the feature func-
tion f(xt,ut
R,ut
H)evaluated at every state and action
pair. We assume a d-dimensional feature function, so
f(xt,ut
R,ut
H)2Rd. Further, for a ﬁnite horizon N, we
letx= ( x0, . . . , xN)>be a sequence of states, uH=
(u0
H, . . . , uN
H)>a sequence of human actions, and uR=
(u0
R, . . . , uN
R)>a sequence of robot actions. We deﬁne RH
to be the human’s expected reward over horizon N:
RH(x0,uR,uH) =N
å
t=0rH(xt,ut,ut
H). (3)
For simpler notation, we combine the N+1 elements
offsoF=åN
t=0f(xt,ut
R,ut
H). Therefore, F(x)evaluates
over a trajectory x. We ﬁnally reformulate the reward
function as an inner product:
RH(x) =wF(x). (4)
Our goal is to learn RH.
Approach Overview. Inverse Reinforcement Learning
(IRL) [15, 19, 23] enables us to learn RHthrough demon-
strated trajectories. However, IRL requires the human to
show demonstrations of the optimal sequence of actions.
Providing demonstrations can be challenging for many
human-robot tasks. Furthermore, generating interesting
training data that actively explores RHthrough atypical,
interesting environments (which is necessary in many
cases for resolving ambiguities) works well [8, 16] but in
practice can make (good) demonstrations infeasible: the
algorithm cannot physically manufacture environments,
and therefore relies on simulation, which makes demon-
strations only possible through teleoperation.
For these reasons, we assume demonstrations are not
available in our work. Instead, we propose to leveragepreference-based learning, which queries Hto provide
comparisons between two candidate trajectories. We pro-
pose a new approach for active learning of a reward
function for human preferences through comparisons.
We split trajectories xinto two parts: a scenario and the
trajectory of the agent Hwhose reward function we are
learning. We formalize a scenario to be the initial state
of the environment as well as the sequence of actions for
the other agent(s) R,t= (x0,u0
R, . . . , uN
R). Given an en-
vironment speciﬁed by scenario t2T, the human agent
will provide a ﬁnite sequence of actions uH=u0
H, . . . , uN
H
in response to scenario t. This response, along with the
scenario tdeﬁnes a trajectory xshowing the evolution
of the two systems together in the environment.
We iteratively synthesize queries where we ask the
human to compare between two trajectories xAand xB
deﬁned over the same ﬁxed scenario tas shown in Fig. 1
(a). Their answer provides us information about w. In
what follows, we discuss how to update our probability
distribution over wgiven the answer to a query, and
how to actively synthesize queries in order to efﬁciently
converge to the right w.
III. L earning Reward Weights
from Preferences of Synthesized Queries
In this section, we describe how we update a distribu-
tion over reward parameters (weights win equation (4))
based on the answer of one query. We ﬁrst assume that
we are at iteration tof the algorithm and an already
synthesized pair of trajectories xAand xBin a common
scenario is given (we discuss in the next section how to
generate such a query). We also assume Hhas provided
her preference for this speciﬁc pair of trajectories at
iteration t. Let her answer be I, with It= + 1 if she
prefers the former, and It=1 if she prefers the latter.
This answer gives us information about w: she is more
likely to say +1 ifxAhas higher reward than xB, and
vice-versa, but she might not be exact in determining
It. We thus model the probability p(Ijw)as noisily
capturing the preference w.r.t. RH:
p(Itjw) =8
>><
>>:exp(RH(xA))
exp(RH(xA))+exp(RH(xB))It= + 1
exp(RH(xB))
exp(RH(xA))+exp(RH(xB))It=1(5)
We start with a prior over the space of all w, i.e., w
is uniformly distributed on the unit ball. Note that the
scale of wdoes not change the preference It, so we can
constrainjjwjj 1 to lie in this unit ball. After receiving
the input of the human It, we propose using a Bayesian
update to ﬁnd the new distribution of w:
p(wjIt)µp(w)p(Itjw). (6)
Let
j=F(xA)F(xB) (7)
Then our update function that multiplies the current
distribution at every step is:
fj(w) =p(Itjw) =1
1+exp(Itw>j)(8)
Updating the distribution of wallows us to reduce the
probability of the undesired parts of the space of w, and
maintain the current probability of the preferred regions.
IV . S ynthesizing Queries
through Active Volume Removal
The previous section showed how to update the dis-
tribution over wafter getting the response to a query.
Here, we show how to synthesize the query in the ﬁrst
place: we want to ﬁnd the next query such that it will
help us remove as much volume (the integral of the
unnormalized pdf over w) as possible from the space
of possible rewards.
Formulating Query Selection as Constrained Opti-
mization. We synthesize experiments by maximizing the
volume removed under the feasibility constraints of j:
max
jminfE[1fj(w)],E[1fj(w)]g
subject to j2F(9)
The constraint in this optimization requires jto be in
the feasible set F:
F=fj:j=F(xA)F(xB),xA,xB2X,
t= (x0,uA
R) = ( x0,uB
R)g(10)
which is a set of the difference of features over feasible
trajectories xA,xB2Xdeﬁned over the same scenario t.
In this optimization, we maximize the minimum of the
split between the two spaces preferred by either choices
of the human agent. Each term in the minimum is the
volume removed depending on the input of the human
It, i.e., the preference. The expectation is taken w.r.t. to
distribution over w.Solution. We ﬁrst reformulate our
problem as an unconstrained optimization, where we en-
force the feasibility of trajectories by directly optimizing
over the query components, x0,uR,uA
H,uB
H, as opposed
to optimizing in the desired feature difference j:
max
x0,uR,uA
H,uB
HminfE[1fj(w)],E[1fj(w)]g(11)
Here, jremains a function of x0,uR,uA
H,uB
H. We will
solve this by local optimization using a Quasi-Newton
method (L-BFGS [4]). To do so, we need a differentiable
objective.
The distribution p(w)can become very complex and
there is no simple way to compute the volume removed
by a Bayesian update, let alone differentiate through it.
We therefore resort to sampling, and optimize an ap-
proximation of the objective obtained via samples, where
weights are sampled in proportion to their probability.
Assume we sample w1, . . . , wMindependently from thedistribution p(w). Then we can approximate p(w)by the
empirical distribution composed of point masses at wi’s:
p(w)1
MM
å
i=1d(wi). (12)
Then the volume removed by an update fj(w)can be
approximated by:
E[1fj(w)]'1
MM
å
i=1(1fj(wi)). (13)
Given such samples, the objective is now differentiable
w.r.t. j, which is differentiable w.r.t. the starting state
and controls – the ingredients of the query which are
the variables in (11). What remains is to get the actual
samples. To do so, we take advantage of the fact that
p(w)is a log-concave function, and the update function
fj(w)deﬁned here is also log-concave in was shown in
Fig. 1 (c); therefore, the posterior distribution of wstays
log-concave. Note we do not need to renormalize the dis-
tribution p(w)after a Bayesian update, i.e. divide p(w)
by its integral. Instead, we use Metropolis Markov Chain
methods to sample from p(w)without normalization.
Log-concavity is useful because we can take advantage
of efﬁcient polynomial time algorithms for sampling
from the current p(w)[17]1. In practice, we use an
adaptive Metropolis algorithm, where we initialize with
a warm start by computing the mode of the distribution,
and perform a Markov walk [11].
We could ﬁnd the mode of fjfrom (8), but it requires a
convex optimization. We instead speed this up by choos-
ing a similar log-concave function whose mode evaluates
to zero always, and which reduces the probability of
undesired wby a factor of exp (Itw>j):
fj(w) =min(1, exp (Itw>j)) (14)
Fig. 1 (c) shows this simpler choice of update function
in black with respect to p(Itjw)in gray. The shape is
similar, but enables us to start from zero, and a Markov
walk will efﬁciently sample from the space of w.
In Fig. 1 (b), we show a simple diagram demonstrating
our approach. Here, wﬁrst uniformly lies on a unit d-
dimensional ball. For simplicity, here we show a 3D ball.
The result of a query at every step is a state and trajec-
tories that result in a feature difference vector j, normal
to the hyperplane fw:w>j=0g, whose direction
represents the preference of the human, and its value lies
inFdeﬁned in equation (10). We reduce the volume of w
by reducing the probability distribution of the samples
ofwon the rejected side of the hyperplane through the
update function fj(w)that takes into account noise in
the comparison.
1Note that computing the actual volume removed can be converted
to the task of integrating a log-concave function, for which efﬁcient
algorithms do exist. The purpose of using samples instead is to have
an expression suitable for maximization of volume removed, i.e. an
expression differentiable w.r.t. the query.
φ"#"$"%ℋℋℛℛ#$#%(a) Preference query.(b) Query response effect.(c) log-concaveupdate ofw.#$or #%→'(Fig. 1: In (a) two candidate trajectories are provided for comparison to the human oracle. xAshows a smoother trajectory without
any collisions. In (b) the unit ball is representing the space of w. We synthesize experiments that correspond to a separating
hyperplanefw:wj=0g, and reweigh samples of won each side of the hyperplane in order to update the distribution
ofw. In (c) we show the two choices of log-concave update functions. Here, f1j(w) = p(Itjw)is the Bayesian update, and
f2j(w) =min(1, exp (Itw>j))is our choice of simpler update function.
V . A lgorithm and Performance Guarantees
Armed with a way of generating queries and a way of
updating the reward distribution based on the human’s
answer to each query, we now present our method for
actively learning a reward function in Alg. 1. The inputs
to the algorithm are a set of features f, the desired
horizon N, the dynamics of the system fHR, and the
number of iterations iter. The goal is to converge to the
true distribution of w, which is equivalent to ﬁnding the
preference reward function RH(x) =wF(x).
We ﬁrst initialize this distribution to be uniform over a
unit ball in line 3. Then, for Miterations, the algorithm
repeats these steps: volume estimation, synthesizing a
feasible query, querying the human, and updating the
distribution.
We sample the space of win line 5. Using these
samples, and the dynamics of the system, SynthExps
solves the optimization in equation (11): it synthesizes
a feasible pair of trajectory that maximizes the expected
volume removed from the distribution of p(w). The
answer to the query is received in line 7. We compute
fj(w), and update the distribution in line 10.
Algorithm 1 Preference-Based Learning of
Reward Functions
1:Input: Features f, horizon N, dynamics f,iter
2:Output: Distribution of w:p(w)
3:Initialize p(w)Uniform (B), for a unit ball B
4:While t<iter:
5: W Msamples from AdaptiveMetropolis( p(w))
6:(x0,uR,uA
H,uB
H) SynthExps( W,f)
7: It QueryHuman (x0,uR,uA
H,uB
H)
8: j=F(x0,uR,uA
H)F(x0,uR,uB
H)
9: fj(w) =min(1,Itexp(w>j))
10: p(w) p(w)fj(w)
11: t t+1
12:End forRegarding the convergence of Alg. 1, one cannot gen-
erally make any strict claims for several reasons: We
replace the distribution p(w)by an empirical distribu-
tion, which could introduce errors. The maximization
in line 6 is via non-convex optimization which does
not necessarily ﬁnd the optimum, and even if it was
guaranteed that the global optimum is found, it could
potentially make very little progress in terms of volume
removal, since the set Fcan be arbitrary.
Putting aside the issues of sampling and global opti-
mization, we can compare what Alg. 1 does to the best
one could do with the set F. Alg. 1 can be thought of as
agreedy volume removal.
Theorem V .1. Under the following assumptions:
The update function is f jas deﬁned in equation (8),
The human inputs are noisy similar to equation (5),
The errors introduced by sampling and non-convex opti-
mization are ignored,
Alg. 1 removes at least 1etimes as much volume as
removed by the best adaptive strategy after ln(1
e)times as
many iterations.
Proof: Removed volume can be seen to be an adap-
tive submodular function, deﬁned in terms of the choices
jand the human input It, as deﬁned in [10]. It is
also adaptive monotone; thus, the results of [10] imply
that greedy volume removal for lsteps in expectation
removes at least (1exp(l/k))OPT kwhere OPT kis
the best solution any adaptive strategy can achieve after
ksteps. Setting l=kln(1
e)gives us the desired result.
One caveat is that in equation (8) the human input
Itis treated as worst case, i.e., in the synthesis step,
one maximizes the minimum removed volume (over the
possible It). Namely maximizing the following quantity:
minfEw[1Pr[It= + 1jw]]),Ew[1Pr[It=1jw]])g.
(15)
Normally the greedy strategy in adaptive submodular
maximization should treat Itas probabilistic. In other
words instead of the minimum one should typically
maximize the following quantity:
Pr[It= + 1]Ew[1Pr[It= + 1jw]]+
Pr[It=1]Ew[1Pr[It=1jw]].(16)
However, note Ew[1Pr[It= + 1jw]]is simply Pr [It=
1]and similarly Ew[1Pr[It= + 1jw]]is Pr[It= + 1].
Therefore Alg. 1 is maximizing min (Pr[It=1], Pr[It=
+1]), whereas greedy submodular maximization should
be maximizing 2 Pr [It=1]Pr[It= + 1]. It is easy to see
that these two maximizations are equivalent, since the
sum Pr [It=1] +Pr[It= + 1] =1 is ﬁxed.
VI. S imulation Experiment
In this section, we evaluate our theory using a semiau-
tonomous driving example. Our goal is to learn people’s
preferred reward function for driving. In this context,
our approach being preference-based is useful since it
allows the users to only compare two candidate trajec-
tories in various scenarios instead of requiring the user
to demonstrate a full trajectory (of how they would like
to drive, not how they actually drive). In addition, our
approach being active enables choosing informative test
cases that are otherwise difﬁcult to encounter in driving
scenarios. For example, we can address the preference of
drivers in moral dilemma situations (deciding between
two undesirable outcomes), which are very unlikely to
arise in standard collected driving data. Finally, our
approach synthesizes these test cases from scratch, which
should help better exploit the continuous and high-
dimensional space of queries. We put these advantages
to the test in what follows.
Experimental Setup. We assume a human driven vehicle
Hliving in an environment with another vehicle R,
and we synthesize trajectories containing two candidate
sequence of actions for the human driven car, while for
every comparison we ﬁx the synthesized scenario (i.e.,
the initial state of the environment and the sequence of
actions ofR). Fig. 1 (a) shows an example of this com-
parison. The white vehicle is R, and the orange vehicle
corresponds toH. The white and orange lines show the
path taken by the human and robot, respectively, in the
two cases over a horizon of N=5. We assume a simple
point-mass dynamics model for both of the vehicles:
[˙x˙y˙q˙v] =
vcos(q)vsin(q)vu1u2av
.
(17)
Here, the state x=
˙x˙y˙q˙v
includes the coor-
dinates of the robot xand y, the heading q, and the
velocity v. The control input of this dynamical system
isu=
u1u2
, where u1is the steering input, and u2
is the acceleration. We also use aas a friction coefﬁcient.
We learn the reward function of the human’s prefer-
ences based on queries similar to Fig. 1 (a). We deﬁne a
set of features that allow representing this cost function.First, f1µc1exp(c2d2)corresponds to penalizing
getting close to the boundaries of the road, where dis
the distance between the vehicle and these boundaries,
and c1and c2are appropriate scaling factors. We use a
similar feature f2for enforcing staying within a single
lane by penalizing leaving the boundaries of the lane.
We also encourage higher speed for moving forward
through f3= (vvmax)2, where vis the velocity of the
vehicle, and vmaxis the speed limit. Also, we would like
the vehicle to have a heading along with the road using
a feature f4=qH~n, where qHis the heading of H, and
~nis a normal vector along the road. Our last feature
f5corresponds to collision avoidance, and is a non-
spherical Gaussian over the distance of HandR, whose
major axis is along the robot’s heading. Then, we aim to
learn a distribution over the weights corresponding to
these features w= [w1,w2,w3,w4,w5]soRH=wF(x)
best represents the preference reward function.
Conditions. We compare our algorithm with two base-
lines: non-active and non-synthesis.
First, we compare it to a non-active version. Instead
of actively generating queries that will remove the most
volume, we uniformly sample a scenario. We do not use
totally random trajectories for the cars as this would be a
weak baseline, instead we sample two candidate weight
vectors wAand wBfrom the current distribution on w.
We then maximize the reward functions wAF(xA)and
wBF(xB)to solve for optimal uA
Hand uB
H. That creates
our query, comparing two reward functions, but no
longer optimized to efﬁciently learn w. The trajectories
generated through this other approach are then used
to query the human, and update the distribution of
p(w)similar to Alg.1. Second, we compare it against
anon-synthesis (discrete) version. Instead of solving a
continuous optimization problem to generate a query,
we do a discrete search over a sampled set of queries.
Metric. We evaluate our results using a hidden reward
function Rtrue=wtrueF(x). We query an ideal user
who knows Rtrue and uses it to compare pairs of tra-
jectories, and show the wcomputed by our algorithm
efﬁciently converges to wtrue.
At every step of the iteration, we compute the follow-
ing measure of convergence:
m=E[wwtrue
jwjjwtruej]. (18)
Here, mcomputes the average heading of the current
distribution of wwith respect to wtrue – how similar
the learned reward is. Since the prior distribution of w
is symmetric (uniformly distributed on a unit ball), this
expectation starts at 0, and moves closer to 1 at every
step of the iteration.
Hypotheses.
H1. The reward function learned through our algorithm is
closer to the true reward compared to the non-active baseline.
1.0
 0.5
 0.0 0.5 1.0
w4 for Heading0.00.51.01.52.02.5PDF
1.0
 0.5
 0.0 0.5 1.0
w1 for Road Boundary1.0
0.5
0.00.51.0w4 for Heading
1.0
 0.5
 0.0 0.5 1.0
w2 for Staying within Lanes1.0
0.5
0.00.51.0w4 for Heading
1.0
 0.5
 0.0 0.5 1.0
w3 for Keeping Speed1.0
0.5
0.00.51.0w4 for Heading
1.0
 0.5
 0.0 0.5 1.0
w5 for Collision Avoidance1.0
0.5
0.00.51.0w4 for Heading
1.0
 0.5
 0.0 0.5 1.0
w4 for Heading0.00.51.01.52.02.5PDF
1.0
 0.5
 0.0 0.5 1.0
w1 for Road Boundary1.0
0.5
0.00.51.0w4 for Heading
1.0
 0.5
 0.0 0.5 1.0
w2 for Staying within Lanes1.0
0.5
0.00.51.0w4 for Heading
1.0
 0.5
 0.0 0.5 1.0
w3 for Keeping Speed1.0
0.5
0.00.51.0w4 for Heading
1.0
 0.5
 0.0 0.5 1.0
w5 for Collision Avoidance1.0
0.5
0.00.51.0w4 for HeadingFig. 2: Distribution of w4, the weight for the heading feature, relative to the other features. The top plots shows the starting
distribution, and the bottom plot shows the distribution at convergence. The orange dot and dotted line show the ground truth
for the weights.
1.0
 0.5
 0.0 0.5 1.00.00.51.01.52.02.53.03.54.0PDFw1 for Road Boundary
w2 for Staying within Lanes
w3 for Keeping Speed
w4 for Heading
w5 for Collision Avoidance
1.0
 0.5
 0.0 0.5 1.00.00.51.01.52.02.53.03.54.0PDF
Fig. 3: Distribution over all weights before/after convergence.
The dotted lines show the ground truth of the weights.
H2. The reward function learned through our algorithm
is closer to the true reward compared to the non-synthesis
baseline.
Results. We run a paired t-test for each hypothesis.
Supporting H1, we ﬁnd that our algorithm signiﬁcantly
outperforms the non-active version ( t(1999) = 122.379,
p<.0001), suggesting that it is important to be doing ac-
tive learning in these continuous and high-dimensional
query spaces. Supporting H2, we ﬁnd that our algo-
rithm signiﬁcantly outperforms the non-synthesis ver-
sion ( t(1999) =35.39, p<.0001), suggesting the impor-
tance of synthesizing queries instead of relying on a dis-
crete set. Note these results hold even with conservative
Bonferroni corrections for multiple comparisons.
Fig. 3 shows the distribution of wfor our algorithm
corresponding to the ﬁve features after 200 iterations,
showing convergence to close to the true weights (in
dotted lines). The mode of the distribution of w1has a
negative weight enforcing staying within the roads, and
w2has a positive weight enforcing staying within your
own lane. w3also shows a slight preference for keepingspeed, and w4shows a signiﬁcant preference for keeping
heading. w5also shows the distribution for the weight
of collision avoidance.
Further, we show the initial and ﬁnal two dimensional
projection of the density of w4, the weight of the heading
feature, with respect to all the other w’s in Fig. 2. This
shift and convergence of the distribution is clear from the
top plot to the bottom plot. The orange dot in all plots,
as well as the orange dotted line show the ground truth
ofwtrue. Fig. 2 shows our algorithm clearly converges to
close to the ground truth for all features.
We show the convergence of the distribution p(w)
in Fig 4. The dark red line shows the mean result of
Alg. 1. It outperforms, according to metric m, both the
non-active condition (black), as well as the non-synthesis
(discretized) condition (dark blue). The results are the
average of 10 runs of the algorithms (since sampling
is nondeterministc). The bar graph in the middle also
shows the converged value of mafter 200 queries.
In a thought experiment, we also investigated to what
extent we see these effects because we are generating real
trajectories for dynamical systems. To test this, we used
the three algorithms to produce abstract queries that a
simulation could answer, consisting of only a feature
difference jas opposed to two feasible real trajectories.
The value of mafter 200 iterations for these versions of
the algorithms are shown in the bar graph with lighter
colors. Because these approaches are unencumbered by
the need for feasibility of the queries (or even producing
actual concrete queries), they overall perform better than
when the algorithms need to generate real queries. In
this case, being active is still important, but the ability
Queries are real trajectoriesQueries are abstract feature difference.Fig. 4: A comparison of our algorithm (red) with a non-
active (black) and a non-synthesis (blue) version. Both deciding
on queries actively and synthesizing them instead of relying
on a predeﬁned set improve performance by a statistically
signiﬁcant margin. On the right, we run an experiment to
argue that these things are really important when synthesizing
real queries for continuous high-dimensional and constrained
systems: the lighter colors are for a version that only generates
a query in feature diference space, without rendering it into
actual (feasible) trajectories – the algorithm is allowed to make
a fake query. There, being active and especially doing synthesis
don’t matter (synthesis even hurts, likely because it is a local
search and a discrete set of queries can better cover the space).
to perform synthesis is no longer useful compared to
relying on a discrete set. Without needing to produce
trajectories, a discrete set covers the much lower dimen-
sional space of feature differences, and discrete search
is not bottlenecked by local optima. This suggests that
indeed, synthesis is important when we are dealing
with learning reward functions for dynamical systems,
requiring queries in a continuous, high-dimensional, and
dynamically constrained space.
Overall, the results suggest that doing active query
synthesis is important in reward learning applications of
preference-based learning, with our results supporting
both of our central hypotheses and demonstrating an
improvement over prior methods which are either non-
active or non-synthesis.
VII. U sability Study
Our experiments so far supported the value of our
contribution: active preference-based learning that syn-
thesizes queries from scratch outperformed both non-
active and non-synthesis learning. Next, we ran a usabil-
ity study with human subjects. The purpose of this study
is not to compare our algorithm once again with other
methods. Instead, we wanted to test whether a person
can interact with our algorithm to recover a reasonable
reward function for dynamical systems with continuous
state and action spaces.
A. Experimental Design
In the learning phase of the experiments, we jumpstart
the weights wwith a reference starting distribution, and
only update that based on the data collected for each
user. We ask for 10 query inputs to personalize the
distribution of wfor each user.
Fig. 5: Car trajectories (orange) in a scenario obtained by
optimizing different weights. On the left, we see the trajectories
optimized based on the learned weight w, and the middle
and right plots correspond to optimizing with w1(slightly per-
turbed w), and w2(highly perturbed w). The perturbations
result in safe lane changes as well, but slower trajectories in
this particular scenario.
Unlike our simulation experiments, here we do not
have access to the ground truth reward function. Still,
we need to evaluate the learned reward function. We
thus evaluate what the users think subjectively about the
behavior produced on a 7 point Likert scale.
Manipulated Factors. In order to calibrate the scale,
we need to compare the reward’s rating with some
baseline. We choose to perturb the learned reward for
this calibration. Thus we only manipulate one factor:
perturbation from the learned weights . We ask users to
compare trajectories obtained by optimizing for reward
with three different values for the weights: i) w=E[w]
providing the mode of the learned distribution from
Alg. 1, ii) w2alarge perturbation of wwhich enables
us to sanity check that the learned reward is better
than a substantially different one, and iii) w1aslight
perturbation of wwhich is a harder baseline to beat
and enables us to test that the learned reward is a
local optimum. The weights thus vary in distance or
alignment with the learned weight, from identical to very
different. We simply add a zero mean Gaussian noise to
perturb w. For w1, we add a Gaussian with standard
deviation of 0.1jwj, and similarly with a standard
deviation ofjwjforw2. Our choice of perturbation is
also affected by the test scenarios. In practice, we sample
from possible w1and w2until the distance between
the human trajectories is signiﬁcant enough to create
interesting scenarios that differentiate the weights (some
simple scenarios have the car drive almost the same
regardless of the reward function).
The 3 weights lead to 3 conditions. Fig. 5 shows one
of the test environments where the robot (white car)
decides to change lanes, and the human driver can take
any of the three depicted orange trajectories. Here the
trajectories correspond to the optimal human actions
based on w,w1, and w2from left to right. The learned
reward function results in a longer and faster trajectory
where the human driver changes lane. The perturbations
also result in a safe lane change; however, they result in
much slower trajectories.
Dependent Measures. For 10 predeﬁned scenarios (ini-
Fig. 6: Here we show the human trajectories of all users for
a speciﬁc scenario based on optimizing the learned reward
function w. We plot the value of weights wfor the ﬁve
features for all users. This ﬁgure uses the same legend as Fig. 3.
tial state and actions of the other car R), we generate
the 3 trajectories corresponding to each condition. We
ask users to rate each trajectory on a 7 point Likert scale
in terms of how closely it matches their preference.
Hypothesis.
H3. Perturbation from the learned weights negatively im-
pacts user rating: the learned weights outperform the per-
turbed weights, with the larger perturbation result in the
smallest rating.
Subject Allocation. We recruited 10 participants (6 fe-
male, 4 male) in the age range of 26 65. All owned
drivers license with at least 8 years of driving experience.
We ran our experiments using a 2D driving simulator,
where we asked preference queries similar to Fig. 6.
B. Analysis
We ran an ANOVA with perturbation amount (mea-
sured via our metric mrelative to the learned weights)
as a factor and user rating as a dependent measure.
Supporting H3, we found a signiﬁcant negative effect of
perturbation (positive effect of similarity m) on the user
rating ( F=278.2, p<.0001). This suggests the learned
weights are useful in capturing desired driving behavior,
and going away from them decreases performance.
In Fig. 7, we show the aggregate result of the 1 7
rating across all scenarios and users. As shown in this
ﬁgure, the highest rating in orange corresponds to the
learned reward weights w, and our users preferred the
slightly perturbed trajectories over the highly perturbed
trajectories since the rating for w1is higher than w2.
We found that, perhaps due to the fairly simple
features we used, users tended to converge to similar
weights, likely representing good driving in general.
In Fig. 6, we show the learned weight distribution for
every feature is shown for all 10 users, and the resulting
optimal trajectories for all our users. These have very
high overlap, further suggesting that users converged
to similar reward functions. In future work, we aim
to look at a more expressive feature set that enables
0 1 2 3 4 5 6 7
User rating of ξ∗
ww∗w1w2Fig. 7: User ratings. The orange bar corresponds to the learned
weights w, and the gray bars correspond to the perturbations
ofw. Our users preferred wover the slightly perturbed ones
w1, and preferred w1over the highly perturbed ones w2.
people to customize their car to their individual desired
driving style, as opposed to getting the car to just do the
one reasonable behavior possible (which is what might
have happened in this case). Nonetheless, the result is
encouraging because it shows that people were able to
get to a behavior that they and other users liked.
VIII. D iscussion
Summary. We introduce an algorithm that can efﬁciently
learn reward functions representing human preferences.
While prior work relies on a discrete predeﬁned set
of queries or on passively learning the reward, our
algorithm actively synthesizes preference queries: it op-
timizes in the continuous space of scenario parameters
and trajectory pairs to identify a comparison to present
to the human that will maximize the amount of ex-
pected volume that the answer will remove in the space
of possible reward functions. We leverage continuous
optimization to synthesize feasible queries, and assign
a log-concave distribution over reward parameters to
formulate an optimization-friendly criterion. We provide
convergence guarantees for our algorithm, and compare
it to a non-active and non-synthesis based approaches.
Our results show that both active learning and synthesis
are important. A user study suggests that the algorithm
helps real end-users attain useful reward functions.
Limitations and Future Work. Our work is limited in
many ways: we use a local optimization method for
query synthesis which converges to local optima, we use
an approximately rational model of the human when
in fact real people might act differently, we assume
that the robot has access to the right features, and we
only consider one other agent in the world rather than
multiple. Furthermore, our study shows that people can
arrive at useful reward functions, such alignment with
the reward leads to higher preference for the emerging
trajectory, but it does not yet show that people can use
this to customize their behavior – most users end up
with very similar learned weights, and we believe this
is because of the limitations of our features and our
simulator. We plan to address these in the future.
Acknowledgments. This work was supported in part by
the VeHICaL project (NSF grant #1545126).
References
[1] Nir Ailon and Mehryar Mohri. Preference-based
learning to rank. Machine Learning , 80(2-3):189–211,
2010.
[2] Baris Akgun, Maya Cakmak, Karl Jiang, and
Andrea L Thomaz. Keyframe-based learning
from demonstration. International Journal of Social
Robotics , 4(4):343–355, 2012.
[3] Riad Akrour, Marc Schoenauer, and Michèle Sebag.
April: Active preference learning-based reinforce-
ment learning. In Joint European Conference on Ma-
chine Learning and Knowledge Discovery in Databases ,
pages 116–131. Springer, 2012.
[4] Galen Andrew and Jianfeng Gao. Scalable training
of l 1-regularized log-linear models. In Proceedings of
the 24th international conference on Machine learning ,
pages 33–40. ACM, 2007.
[5] Chandrayee Basu, Qian Yang, David Hungerman,
Anca Dragan, and Mukesh Singhal. Do you want
your autonomous car to drive like you? In 2017 12th
ACM/IEEE International Conference on Human-Robot
Interaction (HRI) . IEEE, 2017.
[6] Darius Braziunas. Computational approaches to
preference elicitation. Department of Computer Sci-
ence, University of Toronto, Tech. Rep , 2006.
[7] Klaus Brinker, Johannes Fürnkranz, and Eyke
Hüllermeier. Label ranking by learning pairwise
preferences. Technical report, Technical Report
TUD-KE-2007-01, Knowledge Engineering Group,
TU Darmstadt, 2007.
[8] Christian Daniel, Malte Viering, Jan Metz, Oliver
Kroemer, and Jan Peters. Active reward learning.
InRobotics: Science and Systems , 2014.
[9] Johannes Fürnkranz, Eyke Hüllermeier, Weiwei
Cheng, and Sang-Hyeun Park. Preference-based
reinforcement learning: a formal framework and a
policy iteration algorithm. Machine learning , 89(1-2):
123–156, 2012.
[10] Daniel Golovin and Andreas Krause. Adaptive
submodularity: Theory and applications in active
learning and stochastic optimization. Journal of
Artiﬁcial Intelligence Research , 42:427–486, 2011.
[11] Heikki Haario, Eero Saksman, and Johanna Tammi-
nen. An adaptive metropolis algorithm. Bernoulli ,
pages 223–242, 2001.
[12] Rachel Holladay, Shervin Javdani, Anca Dragan,
and Siddhartha Srinivasa. Active comparison basedlearning incorporating user uncertainty and noise.
[13] Ashesh Jain, Shikhar Sharma, Thorsten Joachims,
and Ashutosh Saxena. Learning preferences for
manipulation tasks from online coactive feedback.
The International Journal of Robotics Research , 2015.
[14] Amin Karbasi, Stratis Ioannidis, et al. Comparison-
based learning with rank nets. arXiv preprint
arXiv:1206.4674 , 2012.
[15] Sergey Levine and Vladlen Koltun. Continuous
inverse optimal control with locally optimal exam-
ples. arXiv preprint arXiv:1206.4617 , 2012.
[16] Manuel Lopes, Francisco Melo, and Luis Monte-
sano. Active learning for reward estimation in
inverse reinforcement learning. In Joint European
Conference on Machine Learning and Knowledge Dis-
covery in Databases , pages 31–46. Springer, 2009.
[17] László Lovász and Santosh Vempala. Fast algo-
rithms for logconcave functions: Sampling, round-
ing, integration and optimization. In 2006 47th
Annual IEEE Symposium on Foundations of Computer
Science (FOCS’06) , pages 57–68. IEEE, 2006.
[18] Constantin Rothkopf and Christos Dimitrakakis.
Preference elicitation and inverse reinforcement
learning. Machine Learning and Knowledge Discovery
in Databases , pages 34–48, 2011.
[19] Dorsa Sadigh, Shankar Sastry, Sanjit Seshia, and
Anca D. Dragan. Planning for autonomous cars that
leverages effects on human actions. In Proceedings
of the Robotics: Science and Systems Conference (RSS) ,
June 2016.
[20] Hiroaki Sugiyama, Toyomi Meguro, and Yasuhiro
Minami. Preference-learning based inverse rein-
forcement learning for dialog control. In INTER-
SPEECH , pages 222–225, 2012.
[21] Aaron Wilson, Alan Fern, and Prasad Tadepalli. A
bayesian approach for policy learning from trajec-
tory preference queries. In Advances in neural infor-
mation processing systems , pages 1133–1141, 2012.
[22] Christian Wirth, Johannes Fürnkranz, and Gerhard
Neumann. Model-free preference-based reinforce-
ment learning. In Thirtieth AAAI Conference on
Artiﬁcial Intelligence , 2016.
[23] Brian D Ziebart, Andrew L Maas, J Andrew Bag-
nell, and Anind K Dey. Maximum entropy inverse
reinforcement learning. In AAAI , pages 1433–1438,
2008.

Dorsa Sadigh
Anca Dragan
S
Shankar Sastry
Sanjit Seshia


Shared Autonomy for Robotic Manipulation with Language Corrections.

Shared Autonomy for Robotic Manipulation with Language Corrections
Siddharth Karamcheti∗Raj Palleti∗Yuchen Cui Percy Liang Dorsa Sadigh
Department of Computer Science, Stanford University
{skaramcheti, rpalleti, yuchenc, pliang, dorsa}@stanford.edu
Abstract
Traditional end-to-end instruction following ap-
proaches for robotic manipulation are notori-
ously sample inefficient and lack adaptivity ;
for most single-turn methods, there is no way
to provide additional language supervision to
adapt robot behavior online – a property criti-
cal to deploying robots in collaborative, safety-
critical environments. In this work, we present
a method for incorporating language correc-
tions, built on the insight that an initial instruc-
tion and subsequent corrections differ mainly
in the amount of grounded context needed. To
focus on manipulation domains where the sam-
ple efficiency of existing work is prohibitive,
we incorporate our method into a shared auton-
omy system. Shared autonomy splits agency
between the human and robot; rather than spec-
ifying a goal the robot needs to achieve alone,
language informs the control space provided to
the human. Splitting agency this way allows
the robot to learn the coarse, high-level parts
of a task, offloading more involved decisions –
such as when to execute a grasp, or if a grasp is
solid – to humans. Our user study on a Franka
Emika Panda arm shows that our correction-
aware system is sample-efficient and obtains
significant gains over non-adaptive baselines.
1 Introduction
Research at the intersection of natural language
and robotics has focused on dyadic interactions be-
tween humans and robots, often in the single-turn
instruction following regime (Tellex et al., 2011;
Artzi and Zettlemoyer, 2013; Thomason et al.,
2015; Arumugam et al., 2017). In this paradigm,
a human gives an instruction, and the robot exe-
cutes behavior in the world, autonomously – simul-
taneously resolving the human’s goal as well as
planning a course of actions to execute in the envi-
ronment. While impactful, building systems with
this explicit division of agency between humans
∗denotes equal contribution
“Grab the thing on the white table, and place it in the basket”
“No, on the left!” Shared Autonomy Regime
Incorrectly interprets original instruction21
3Figure 1: Our proposed system. Whereas prior work
only allows for issuing a single language utterance held
constant during execution (solid line), our approach
allows users to provide language corrections during
execution (left window – dashed line).
and robots is nontrivial; many existing systems
either make strong assumptions about the environ-
ment in order to use motion planners (Matuszek
et al., 2012; Kollar et al., 2013), or require extreme
amounts of language-aligned data to learn general
policies (Chevalier-Boisvert et al., 2019; Stepputtis
et al., 2020; Lynch and Sermanet, 2020).
Coupled with the severe sample inefficiency of
existing approaches is their lack of adaptivity . Con-
sider the robot in Fig. 1, trying to execute “grab
the thing on the white table and place it in the
basket.” This instruction is ambiguous and as a
result, it is not clear what should happen. One
natural option is for the human to provide a set
of streaming corrections to the robot, changing its
behavior on the fly. While recent work tries to
get at the spirit of this idea by learning from dia-
logue (Thomason et al., 2019a,b), post-hoc correc-
tions (Co-Reyes et al., 2019), or implicit feedback
(Karamcheti et al., 2020), none of these approaches
=
[Train-Only] 6-DoF Action
“Pour the blue cup into  the cereal bowl”State Encoder
Distil-RoBERTaTraining UtterancesDistil-RoBERTa“Put the banana away” “Towards the cup”“Empty the blue cup’s contents into the bowl”Similarity  SearchInstruction & Correction  “Gating” Function
hstateProprioceptive StateObject Poses+=hlanguageα1××ProjectionBasis 1 — (6-DoF)Basis 2 — (6-DoF)z1z2⋅Inference Human controls!
Action Encoder
Predicted  Robot Action…Basis n — (6-DoF)…znLanguageActionState“Decoder”  (Dot-Product)Figure 2: Our proposed LILAC model. Central to LILAC is the “gating” module (orange) which controls the
amount of state-context for a given language input, allowing us to handle corrections. We use GPT-3 in lieu of a
heuristic to provide α(see Appendix A for discussion), though we plan to learn αfrom user feedback in the future.
Solid lines represent the inference pipeline, while dashed lines indicate training-only steps.
work online . Real systems for language-driven
human-robot interaction must be able to handle
streaming corrections in a manner that is both nat-
ural andsample efficient .
One answer to sample efficiency lies in leverag-
ing existing methods in shared autonomy (Dragan
and Srinivasa, 2013; Argall, 2018; Javdani et al.,
2018). This class of approaches splits agency be-
tween the human and robot; during execution, both
parties influence the ultimate actions of the robot,
sharing the burden of reasoning over actions. By
factoring the difficulty of the problem across the
human and the robot, shared autonomy approaches
see large gains in sample efficiency; the robot can
learn coarse, high-level features, off-loading the
short, fine-grained manipulation to the human, thus
playing off the strenghts of both parties. The con-
crete instance of shared autonomy that we focus
on in this work is learned latent actions for assis-
tive teleoperation (Herlant et al., 2016; Losey et al.,
2021), where we learn low-dimensional control
spaces that humans can use via joysticks, sip-and-
puff devices, or other assistive tools to maneuver
high-dimensional robots. Because humans are ac-
tively involved in controlling the robot – especially
in critical states, such as when aligning the robot
gripper to lift a cup – these approaches are able
to operate at the scale of 5 - 10 examples per task.
This is in contrast to the 10K - 100K demonstra-
tions required by modern instruction following sys-
tems learned via imitation learning, in the fully
autonomous setting (Luketina et al., 2019).
The learned latent actions paradigm operational-
izes this division of agency by formulating a set
of approaches that use small datasets of demon-
strations to learn task-specific assistive controllers(Losey et al., 2020; Jeon et al., 2020; Karamcheti
et al., 2021b; Li et al., 2020). While powerful and
sample-efficient, a key failing of these approaches
is their inability to handle multiple objectives – and
more importantly – provide a natural interface for
users to specify their goals. To address this chal-
lenge, Karamcheti et al. (2021a) introduce LILA
(Language-Informed Latent Actions), by using lan-
guage in a manner similar to single-turn instruction-
following approaches. However, while accumu-
lating gains in sample efficiency, LILA, like most
single-turn systems, lacks adaptivity, a critical com-
ponent for any real-time, user controlled system.
Looking to Fig. 1, we see that LILA misinterprets
the ambiguous instruction “grab the thing on the
white table, and place it in the basket” (1), moving
towards the cup (2), rather than the banana as the
user intended (3). Problems of ambiguity, misspec-
ification, or underspecification are pervasive in any
real-world, user-facing language system, and as
such, we need an approach for handling streaming,
online corrections – in this case, simple utterances
like “stop,” or “no, on the left!”
In this work, we introduce LILAC (Language-
Informed Latent Actions with Corrections), an
adaptive system for real-world robotic manipula-
tion, that – unlike LILA – can effectively interpret
streaming language corrections. Critically, our key
insight is in realizing language utterances vary in
the amount of object state-dependence they require.
Aninstruction like that in Fig. 1 – “grab the thing
on the white table, and place it in the basket” – re-
quires dense grounded information about object
positioning while a correction – “no, on the left!” –
can be expressed as a function of the user’s (static)
reference frame, without proprioceptive state or ob-
ject information. This insight allows us to decouple
interpreting corrections from grounding: we can
learn corrections from even fewer examples , boost-
ing the robustness andgeneralization potential of
our approach.
The following sections introduce LILAC, includ-
ing how we identify the “state-dependence” of an
utterance. We layout our experiments, and culmi-
nate with the results of a user study (small-scale,
n= 5) conducted on a physical Franka Emika
Panda (a 7-DoF fixed arm manipulator), with a dis-
cussion of future work on extending LILAC, and
natural language supervision for shared autonomy.
2 Motivating Example
To gain a picture of LILAC’s latent actions pipeline,
we present a motivating example in Fig. 1. A
user first gives an instruction to “grab the thing
on the white table and place it in the basket,” which
presents them a 2-DoF control space they can use
via the joystick in their hands. With this new con-
trol space, pushing upon the joystick may bring
the arm closer to the table, while right might twist
the gripper to align with the objects on the table.
Unfortunately, the initial model’s control space pre-
diction is not perfect, and the arm skews towards
the wrong object!
With LILAC, the user is able to issue real-time
language corrections, updating their control space .
Specifically, the new user utterance is fed to our
learned model that parses the utterance, and pro-
duces a new control mapping reflecting the user’s
intent. Now, pushing left on the joystick brings the
arm left, allowing the user to grasp the banana, as
they intended. Finally, after the correction has been
satisfied, the user is able to denote termination with
a button on their controller, dropping back into the
control space for the original task they provided
– in this case, returning to the control space for
placing the object in the basket.
3LILAC: Natural Language Corrections
LILAC builds off of LILA as introduced by Karam-
cheti et al. (2021a) by adding a gating module to
handle streaming corrections. The architecture is
depicted in Fig. 2; solid lines denote the inference
logic, while dashed lines denote training logic.
Overview. LILAC is a conditional autoencoder
with some extra structural elements. The encoder
takes in (language, state, action) triples – the greenboxed components in Fig. 2 – and factors the model-
ing of the control space into two subproblems. The
first subproblem is identifying a set of basis vectors
b1. . . b nfor low-dimensional control conditioned
on the current state and language utterance, where
nis the dimensionality of the latent space. These
basis vectors have the same dimensionality as the
robot’s native action space (e.g., 6/7-DoF). The sec-
ond subproblem is finding a set of scalar weights
(z1. . . z n) of the recovered basis vectors, optimized
such that the convex combinationPn
i=1zi·bire-
constructs the original action.
At train time, we assume a dataset of training ut-
terances (either instructions orcorrections), paired
with robot trajectories comprised of (state, action)
pairs. For this work, we assume the action space
is the 6-DoF end-effector velocity of the robot (ob-
tained via forward kinematics), and the state space
is the combination of the robot’s proprioceptive
state, containing information about its joint states
and end-effector pose (also in 6D), as well as the
coordinates of each object in the scene. Because
we are interested in intuitive, low-dimensional con-
trol, we set n= 2, so that we only produce 2 basis
vectors and weights; this way, a human can operate
the robot using any 2-DoF interface, like a joystick
or computer mouse.
Exactly as in LILA, we use a pretrained Distil-
Roberta model from Sentence-BERT (Reimers and
Gurevych, 2019) to encode language utterances,
in tandem with an “unnatural-language processing”
nearest neighbors index (Marzoev et al., 2020); be-
cause we are in the small data regime (2 hours
of demonstrations), projecting inference-time ut-
terances onto existing training exemplars prevents
the LILAC model from generalizing poorly as lan-
guage embeddings drift, which could lead to prac-
tical issues of user safety.
Gating Instructions vs. Corrections. The key
insight of this work is that language instructions
and corrections differ in their amount of object
state-dependence – but what does this mean? From
a linguistic perspective, one might categorize dif-
ferent utterances based on the number of referents
present; an utterance like “grab the thing on the
white table and place it in the basket” as in Fig. 1
has 3 referents, indicating a large degree of state
dependence; the robot must ground the utterance
in the objects of the environment to resolve the cor-
rect behavior. However, an utterance like “no, to
the left!” has no explicit referents; one can resolve
End-Eﬀector ControlCorrection-Aware ImitationLILA (No Corrections)LILAC (Ours)Goal: “Place the fruit basket on the tray”
✅
✅
❌
❌
“Go forward”“Lower it”Figure 3: Qualitative trajectories from 4 different control strategies, operated by an “oracle.” Note the dashed lines
indicate when language corrections were used. In general, both LILAC and the End-Effector control methods are
able to solve tasks, with LILAC able to do so more efficiently. Imitation Learning (green) fails to fit the tasks – even
with corrections – and LILA is unable to progress without finer-grained correction information.
the utterance without object or proprioceptive state
information, instead relying solely on the user’s
static reference frame and induced coordinates.
To operationalize this idea whilst remaining sam-
ple efficient1, we use a gating function (orange, in
Fig. 2) that given language, predicts a value αfrom
0–1. A value of 0 signifies a correction . Appropri-
ately, in our architecture, this zeroes out any state-
dependent information (see the αterm in Fig. 2),
and predicts an action solely based on the provided
language. In this work, we construct a prompt har-
ness with GPT-3 (Brown et al., 2020) to output
α= 0,0.5,1– the prompt text and motivation for
this decision can be found in Appendix A.
4 Experiments & User Study
Our experiments consist of a targeted evaluation
with an expert “oracle” of LILAC and 3 different
baselines (results in Fig. 3), as well as a real-world
user study ( n= 5) on a complex manipulation
workspace with a Franka Emika Panda arm (results
in Fig. 4). We reuse all the publicly released data
from Karamcheti et al. (2021a), and for correction
data, collect a handful of corrections for moving in
the 6 cardinal directions, as well as two mixed cor-
rections that require some level of state grounding
(“lower the bowl” and “tilt the cup”).
Targeted Expert Evaluation. The goal of our
expert-controlled evaluation was to evaluate the
corrections-module in LILAC, and get a point of
reference to LILA and traditional methods for robot
control – specifically, using a control scheme that
1One question is why not treat all utterances as equal; the
answer is rooted in the small data regime we operate in. We’d
need to collect several instances of the same correction “to the
left” in different states to generalize, whereas with the LILAC
approach, we realistically only need one.uses inverse kinematics to control the end-effector
of the robot, two axes at a given time (e.g. [X,
Y], [Z, Roll], [Pitch, Yaw] ). Fig. 3 shows
the results; LILAC and End-Effector control are
both able to accomplish the two tasks, but LILA
struggles to recover from overshooting problems.
We also evaluate one other strategy – Language &
Correction-Aware Imitation, which doesn’t have
enough data to fit a reasonable policy (see discus-
sion in Karamcheti et al. (2021a) for more infor-
mation). We additionally evaluate a “no-language”
variant of latent actions in Appendix C.
User Study. Given the results from the automatic
evaluation, we run a within-subjects user study
(n= 5 , 3 male, 2 female, 3 users with prior
teleoperation experience), with the three operation
schemes – LILAC, LILA, and End-Effector control
– as the three conditions. Each user was randomly
assigned one of the five original tasks from the
LILA work, and asked to complete it with each
control strategy.2Fig. 4 shows the general quan-
titative (left) and subjective (right) study results.
Users provide linguistic feedback verbally; in this
work, we rely on an expert proctor to manually
type the verbal instruction into the computer run-
ning the LILAC model – future work will adopt
off-the-shelf ASR technologies, such as the Google
Speech-to-Text API.
On the Strength of End-Effector Control.
LILAC strictly dominates LILA, showing the ben-
efit of adding even simple real-time correction han-
dling. However, compared to End-Effector con-
trol, the success is less clear. While obtaining a
slightly higher success rate, the qualitative results
2More information about the structure of the study can be
found in Appendix B
Figure 4: User study results ( n= 5) with a Franka Emika Robotic arm. LILAC outperforms the non-adaptive LILA
model across the board, obtaining higher success rates, and is preferred by users. While LILAC and End-Effector
control obtain similar success rates (though LILAC is slightly higher), plotting the “jerk” (2nd derivative of input
velocity) paints a different picture: controlling the end-effector is very jerky, requiring high user load.
are mixed. One confound is the limited pool of
participants, many of whom already had prior tele-
operation experience (due to COVID restrictions,
we could not widen the pool). Another possible
confound is the structure of the user study itself; to
better allow for users to adjust to the LILAC cor-
rections procedure, the amount of practice time
afforded each user is larger than in prior work,
which allows users to get more acquainted with end-
effector control. Ultimately, this points to the tasks
in this work being on the simpler side, able to be
solved (mostly) without complex, mixed angular-
linear control of the robot’s end-effector; if we
were to account for more real-world manipulation
tasks such as sweeping, wiping, or feeding – all
of which require handling contact and controlling
3+ degrees-of-freedom – we would see degraded
End-Effector performance.
That being said, for additional insight on user
cognitive load when using the various control
schemes, we plotted the 2nd-derivative of accel-
eration – jerk – of the input controls; we see here
that End-Effector approaches require significantly
more fast movement compared to LILAC. Not only
is this more taxing on the user, but is potentially
unsafe depending on the application – another axis
we will explore in future work.
5 Discussion
The union of natural language and shared auton-
omy for real-world robot manipulation is a rich and
vibrant research area. Moving beyond strict dyadic
interactions towards the shared autonomy setting
opens the door to rich work in language supervi-
sion for robotic manipulation – work that has so
far been limited by the steep data requirements of
training language-conditioned policies (Stepputtis
et al., 2020; Shridhar et al., 2021). As shown in
this work, shared autonomy approaches are able tobenefit greatly from sharing agency with a human-
in-the-loop, leading to gains in data efficiency.
Specifically, in this work we introduced LILAC,
an adaptive shared autonomy system for handing
streaming language corrections, provided while a
user completes a task . The entire LILAC model
was trained with 2 hours of data collected by a sin-
gle person, vs. the multiple days of data that would
be required if using a fully autonomous, imitation
learning approach. While the sample efficiency
wins are clear, LILAC remains limited; the current
evaluation shows that hand-coded control schemes
that let users directly manipulate the end-effector
can be similarly effective in some cases. Further-
more, LILAC is heavily tied to the latent actions
paradigm for shared autonomy, which is only a
small slice of the different types of solutions for
human-in-the-loop robotic manipulation.
Future work in language and shared autonomy
will allow for learning newutterances online, from
user feedback; for example, developing methods
for learning control strategies for novel language,
like “flip over the cup,” with minimal user feedback
(teaching demonstrations, language corrections,
etc.). More broadly, we hope to generalize our
correction module to other versions of language-
informed robotics – for example, to policy blending
(Dragan and Srinivasa, 2013), guided planning, and
interactive imitation learning (Kelly et al., 2019) –
for more complex, real-world manipulation tasks.
Acknowledgments
Toyota Research Institute (“TRI”) provided funds
to support this work. Siddharth Karamcheti is
grateful to be supported by the Open Philanthropy
Project AI Fellowship. We would additionally like
to thank the participants of our user study, as well
as our anonymous reviewers.
References
Brenna D Argall. 2018. Autonomy in rehabilitation
robotics: an intersection. Annual Review of Control,
Robotics, and Autonomous Systems , 1:441–463.
Yoav Artzi and Luke Zettlemoyer. 2013. Weakly su-
pervised learning of semantic parsers for mapping
instructions to actions. Transactions of the Associa-
tion for Computational Linguistics (TACL) , 1:49–62.
Dilip Arumugam, Siddharth Karamcheti, Nakul
Gopalan, Lawson L. S. Wong, and Stefanie Tellex.
2017. Accurately and efficiently interpreting human-
robot instructions of varying granularities. In
Robotics: Science and Systems (RSS) .
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie
Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-V oss,
Gretchen Krueger, Tom Henighan, Rewon Child,
Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu,
Clemens Winter, Christopher Hesse, Mark Chen, Eric
Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,
Jack Clark, Christopher Berner, Sam McCandlish,
Alec Radford, Ilya Sutskever, and Dario Amodei.
2020. Language models are few-shot learners. arXiv
preprint arXiv:2005.14165 .
Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem
Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu
Nguyen, and Yoshua Bengio. 2019. Babyai: A plat-
form to study the sample efficiency of grounded
language learning. In International Conference on
Learning Representations (ICLR) .
John D. Co-Reyes, Abhishek Gupta, Suvansh Sanjeev,
Nick Altieri, John DeNero, Pieter Abbeel, and Sergey
Levine. 2019. Guiding policies with language via
meta-learning. In International Conference on Learn-
ing Representations (ICLR) .
Anca D Dragan and Siddhartha S Srinivasa. 2013. A
policy-blending formalism for shared control. In-
ternational Journal of Robotics Research (IJRR) ,
32:790–805.
Laura V Herlant, Rachel M Holladay, and Siddhartha S
Srinivasa. 2016. Assistive teleoperation of robot
arms via automatic time-optimal mode switching.
InACM/IEEE International Conference on Human
Robot Interaction (HRI) , pages 35–42.
Shervin Javdani, Henny Admoni, Stefania Pellegrinelli,
Siddhartha S Srinivasa, and J Andrew Bagnell. 2018.
Shared autonomy via hindsight optimization for tele-
operation and teaming. International Journal of
Robotics Research (IJRR) , 37:717–742.
Hong Jun Jeon, Dylan P. Losey, and Dorsa Sadigh. 2020.
Shared autonomy with learned latent actions. In
Robotics: Science and Systems (RSS) .
Siddharth Karamcheti, Dorsa Sadigh, and Percy Liang.
2020. Learning adaptive language interfaces throughdecomposition. In EMNLP Workshop for Interactive
and Executable Semantic Parsing (IntEx-SemPar) .
Siddharth Karamcheti, Megha Srivastava, Percy Liang,
and Dorsa Sadigh. 2021a. LILA: Language-informed
latent actions. In Conference on Robot Learning
(CoRL) .
Siddharth Karamcheti, A. Zhai, Dylan P. Losey, and
Dorsa Sadigh. 2021b. Learning visually guided latent
actions for assistive teleoperation. In Learning for
Dynamics & Control Conference (L4DC) .
Michael Kelly, Chelsea Sidrane, K. Driggs-Campbell,
and Mykel J. Kochenderfer. 2019. HG-DAgger: In-
teractive imitation learning with human experts. In
International Conference on Robotics and Automa-
tion (ICRA) , pages 8077–8083.
T. Kollar, J. Krishnamurthy, and Grant P. Strimel. 2013.
Toward interactive grounded language acqusition. In
Robotics: Science and Systems (RSS) .
Mengxi Li, Dylan P. Losey, Jeannette Bohg, and Dorsa
Sadigh. 2020. Learning user-preferred mappings for
intuitive robot control. In International Conference
on Intelligent Robots and Systems (IROS) .
Dylan P. Losey, Hong Jun Jeon, Mengxi Li, Kr-
ishna Parasuram Srinivasan, Ajay Mandlekar, Ani-
mesh Garg, Jeannette Bohg, and Dorsa Sadigh. 2021.
Learning latent actions to control assistive robots.
Autonomous Robots (AURO) , pages 1–33.
Dylan P. Losey, Krishnan Srinivasan, Ajay Mandlekar,
Animesh Garg, and Dorsa Sadigh. 2020. Controlling
assistive robots with learned latent actions. In In-
ternational Conference on Robotics and Automation
(ICRA) , pages 378–384.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar,
Jakob Foerster, Jacob Andreas, Edward Grefenstette,
Shimon Whiteson, and Tim Rocktäschel. 2019. A
survey of reinforcement learning informed by natu-
ral language. In International Joint Conference on
Artificial Intelligence (IJCAI) .
Corey Lynch and Pierre Sermanet. 2020. Grounding
language in play. arXiv preprint arXiv:2005.07648 .
Alana Marzoev, S. Madden, M. Kaashoek, Michael J.
Cafarella, and Jacob Andreas. 2020. Unnatural lan-
guage processing: Bridging the gap between syn-
thetic and natural language data. arXiv preprint
arXiv:2004.13645 .
Cynthia Matuszek, Nicholas FitzGerald, Luke Zettle-
moyer, Liefeng Bo, and Dieter Fox. 2012. A joint
model of language and perception for grounded at-
tribute learning. In International Conference on Ma-
chine Learning (ICML) , pages 1671–1678.
Nils Reimers and Iryna Gurevych. 2019. Sentence-
BERT: Sentence embeddings using siamese BERT-
networks. In Empirical Methods in Natural Lan-
guage Processing (EMNLP) .
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. 2021.
Cliport: What and where pathways for robotic manip-
ulation. In Conference on Robot Learning (CoRL) .
Simon Stepputtis, J. Campbell, Mariano Phielipp, Ste-
fan Lee, Chitta Baral, and H. B. Amor. 2020.
Language-conditioned imitation learning for robot
manipulation tasks. In Advances in Neural Informa-
tion Processing Systems (NeurIPS) .
Stefanie Tellex, Thomas Kollar, Steven Dickerson,
Matthew R Walter, Ashis Gopal Banerjee, Seth J
Teller, and Nicholas Roy. 2011. Understanding nat-
ural language commands for robotic navigation and
mobile manipulation. In Association for the Advance-
ment of Artificial Intelligence (AAAI) .
Jesse Thomason, Michael Murray, Maya Cakmak, and
Luke Zettlemoyer. 2019a. Vision-and-dialog naviga-
tion. In Conference on Robot Learning (CoRL) .
Jesse Thomason, Aishwarya Padmakumar, Jivko
Sinapov, Nick Walker, Yuqian Jiang, Harel Yedid-
sion, Justin W. Hart, Peter Stone, and Raymond J.
Mooney. 2019b. Improving grounded natural lan-
guage understanding through human-robot dialog. In
International Conference on Robotics and Automa-
tion (ICRA) .
Jesse Thomason, Shiqi Zhang, Raymond J. Mooney,
and Peter Stone. 2015. Learning to interpret natural
language commands through human-robot dialog. In
International Joint Conference on Artificial Intelli-
gence (IJCAI) .A Using GPT-3 to Identify Corrections
A core component of LILAC is the choice of gating
function, for producing the object state dependence
weight αfor a given language utterance. Critically,
αdictates whether an utterance is an instruction
(α= 1) that depends on the current environment
context, a correction (α= 0) that does not, and
can be interpreted without additional grounded in-
formation, or something in-between (α= 0.5).
We made the realization early on that identify-
ing whether a language utterance fell into one of
the above categories could (at least heuristically)
be decoupled from any environment information;
that is to say, we could predict αdirectly from the
language utterance alone. Given this hypothesis,
and the fact that we were not sure whether the α-
gating would even work in our small-data regime,
we found it difficult to defend the choice to collect
data to learn αupfront, prior to running through the
whole system. Therefore, our choices were to ei-
ther hardcode a series of αvalues for a small, fixed
set of correction language, or come up with some
heuristic (e.g., referent-counting) that could break
or generalize poorly to “in-between” utterances. In-
stead, we chose to try a prompt construction based
approach, leveraging GPT-3 (Brown et al., 2020).
The prompt we specified is shown in Fig. 5; we
did minimal prompt-tuning, only reordering the ex-
amples shown, and turning the temperature down
to 0 (as we wanted this to be deterministic). We
found GPT-3 to work incredibly well out of the
box – a phenomenon that cannot be overstated .
With a straightforward procedure, we were able
to use GPT-3 as a drop-in replacement for what
otherwise would have been a brittle heuristic, or a
limited set of language corrections. It is incredibly
exciting to be able to prototype these systems via
GPT-3 quickly, and we hope that this type of us-
age becomes prevalent throughout not only human-
robot interaction, but widespread NLP pipelines as
a whole. Given the results with the “drop-in” GPT-
3 model, we have a good idea as to where we need
to focus future work with respect to learning α–
specifically, for handling the nuanced utterances
that are “in-between” corrections and instructions –
and are excited to tackle this moving forward.
B User Study Details & Tasks
As mentioned in §4, we ran a within-subjects study
with a small number of participants (n= 5) . The
study consisted of each user using the following
Figure 5: The concrete prompt used for GPT-3 davinci-instruct , visualized in the OpenAI API Playground. We
primed GPT-3 with 3 handcrafted examples, without much other thought, and used the corresponding outputs as our
αgating values (“instruction” = 1, “in-between” = 0.5, “correction” = 0).
strategies to solve a given task: End-Effector con-
trol, LILA (no corrections), and LILAC (our pro-
posed approach). The order of strategies was ran-
domized across users. The tasks were as follows:
1.Pick & Insert Banana – Grasp the banana,
and place it in the plastic fruit basket, turning
the gripper appropriately to insert the banana.
2.Pick & Place Basket – Grasp the basket by
the handles, and lower it onto the tray.
3.Pick & Place Cereal Bowl – Grasp the
green cereal bowl by the lip of the bowl, lift
it off the pedestal, and lower it onto the tray
without collision.
4.Pick & Pour into Cereal Bowl – Grasp
the blue cup with marbles by either the lip of
the cup or the handle on the side, then lift it
over the cereal bowl, tilting the cup to pour
the marbles into the bowl.
5.Pick & Pour into Mug – Grasp the blue
cup with marbles by either the lip of the cup
or the handle on the side, then lift it over the
black mug – avoiding collisions – then pour
the contents into the mug.
Prior to executing the task in a given control
mode, each user was given 3 minutes to practice
using that control mode. During practice, the user
could experiment with various patterns of joystick
input to better understand how they translated to
movement of the robot arm. We found that theamount of “naturalization” time was critical in get-
ting users to adapt to the interfaces provided by
LILA, and LILAC. Furthermore, for users not al-
ready experienced in robot teleoperation, we found
this practice period important as well. However, we
found that it takes longer to naturalize to LILAC
vs. End-Effector control – this also explains the
slight difference in results between LILAC and
End-Effector control. Future work will explore the
impact of this “naturalization period” under various
conditions.
C Additional Visualizations & Baselines
To supplement our experiments from §4, we
present two additional sets of trajectory visualiza-
tions in Fig. 6 and Fig. 7. Fig. 6 shows the same 4
strategies as in Fig. 3, except for a more complex
pouring task; in general, the pattern of behavior is
the same – both LILAC and End-Effector control
are able to solve the task, whereas LILA stalls out
due to a lack of a corrective signal, with Correction-
Aware Imitation Learning failing to fit a decent
policy given limited data.
One other baseline we ran was the no-language
variant of latent actions ; we ran this to show the im-
pact that language has on providing intuitive, use-
ful control spaces. Fig. 7 shows the results of this
baseline vs. LILAC – at a glance, the no-language
variant completely fails, reducing to random, oscil-
latory behavior, because it cannot learn to provide
a good control space for all tasks, without an extra
conditioning signal. This is the same observation
made in Karamcheti et al. (2021a).
End-Eﬀector ControlCorrection-Aware ImitationLILA (No Corrections)LILAC (Ours)Goal: “Pour the contents of the blue cup into the mug”
✅
✅
❌
❌
“Move down”“Head right”Figure 6: Additional trajectory visualizations for the four strategies from §4, this time for a more complex “pouring”
task; we see similar results as in Fig. 3.
“Place the fruit basket on the tray”No-Language Latent Actions
“Pour the contents of the blue cup into the mug”
LILAC (Ours)
✅“Lower it”
❌No-Language Latent Actions
❌
“Head right”
LILAC (Ours)
✅
Figure 7: Results visualizing trajectories for LILAC and an additional baseline strategy – “No-Language Latent
Actions” – the language-free variant of our approach. Note that this approach trivially fails, as its overloaded,
unable to find a satisfying control scheme that would allow users to perform all 5 tasks without extra conditioning
information. This translates to aimless, oscillatory behavior during execution.

Siddharth Karamcheti*
Raj Palleti*
Yuchen Cui
Percy Liang
Dorsa Sadigh