Planning for Autonomous Cars that Leverage Effects on Human Actions.

Planning for Autonomous Cars
that Leverage Effects on Human Actions
Dorsa Sadigh, Shankar Sastry, Sanjit A. Seshia, and Anca D. Dragan
University of California, Berkeley, {dsadigh, sastry, sseshia, anca}@eecs.berkeley.edu
Abstract —Traditionally, autonomous cars make predic-
tions about other drivers’ future trajectories, and plan to
stay out of their way. This tends to result in defensive and
opaque behaviors. Our key insight is that an autonomous
car’s actions will actually affect what other cars will do in
response, whether the car is aware of it or not. Our thesis is
that we can leverage these responses to plan more efﬁcient
and communicative behaviors. We model the interaction
between an autonomous car and a human driver as a dy-
namical system, in which the robot’s actions have immediate
consequences on the state of the car, but also on human
actions. We model these consequences by approximating the
human as an optimal planner, with a reward function that
we acquire through Inverse Reinforcement Learning. When
the robot plans with this reward function in this dynamical
system, it comes up with actions that purposefully change
human state: it merges in front of a human to get them to
slow down or to reach its own goal faster; it blocks two
lanes to get them to switch to a third lane; or it backs up
slightly at an intersection to get them to proceed ﬁrst. Such
behaviors arise from the optimization, without relying on
hand-coded signaling strategies and without ever explicitly
modeling communication. Our user study results suggest that
the robot is indeed capable of eliciting desired changes in
human state by planning using this dynamical system.
I.introduction
Currently, autonomous cars tend to be overly defensive
and obliviously opaque . When needing to merge into an-
other lane, they will patiently wait for another driver to
pass ﬁrst. When stopped at an intersection and waiting
for the driver on the right to go, they will sit there unable
to wave them by. They are very capable when it comes
to obstacle avoidance, lane keeping, localization, active
steering and braking [5–8, 14, 16, 25]. But when it comes
to other human drivers, they tend to rely on simplistic
models: for example, assuming that other drivers will be
bounded disturbances [9, 21], they will keep moving at
the same velocity [17, 22, 27], or they will approximately
follow one of a set of known trajectories [10, 26].
These simplistic models lead to predictions about what
other cars are going to do, and the autonomous car’s task
is to do its best to stay out of their way. It will not cut
in front of another driver when it is in a rush. It will
also be restricted to functional actions, and not execute
actions that are communicative.
Our goal is to enable autonomous cars to be more
efﬁcient, and better at coordinating with human drivers.
Our key insight is that other drivers do not oper-
ate in isolation: an autonomous car’s actions will
(a) Car merges aheadof human; anticipates human braking(b) Car backs up at 4way stop;anticipates human proceeding
(c) User drives human car
Fig. 1: We enable cars to plan with a model of how human drivers
would react to the car’s actions. We test the planner in a user study,
where the car ﬁgures out that (a) it can merge in front of a human
and that will slow them down, or (b) it can back up slightly at an
intersection and that will make the human go ﬁrst.
actually have effects on what other drivers will
do. Leveraging these effects during planning will
generate behaviors for autonomous cars that are
more efﬁcient and communicative.
In this work, we develop an optimization-based
method for planning an autonomous vehicle’s behavior
in a manner that is cognizant of the effects it will have on
human driver actions. This optimization leads to plans
like the ones in Fig.1.
In the top left, the yellow (autonomous) car decides to
cut in front of a human driver in order to more efﬁciently
reach its goal. It arrives at this plan by anticipating that
taking this action will cause the human to brake and
make space for it.
In the top right, the yellow car wants to let the
human driver go ﬁrst through the intersection, and it
autonomously plans to back up slightly before going, an-
ticipating that this will encourage the human to proceed.
These can be interpreted as signaling behaviors, but
they emerge out of optimizing to affect human actions,
without ever explicitly modeling human inferences.
Our contributions are three-fold:
1. Formalizing interaction with drivers as a dynamical
system. We model driving in an environment with a
human driven car as a dynamical system with both
autonomous and human agents. In this model, the
autonomous car’s actions do not just have immediate
effects on the car’s state; instead, they also affect human
actions. These, in turn, affect the state of the world. We
propose a dynamics model for this system by modeling
the human as optimizing some reward function, which
we learn through Inverse Reinforcement Learning.
This builds on work in social navigation which ac-
counts for interaction potentials with human trajecto-
ries [11, 24]: the human and the robot trajectories are
jointly planned as the optimum of some reward function
in order for everyone to reach their goals and avoid each
other. More generally, these works instantiate collabora-
tive planning [20]. In contrast, our work allows for the
human and the robot to have different reward functions:
the human is optimizing their own reward function, and
the robot is leveraging this to better optimize its own.
The practical implications of allowing different reward
functions are that the robot now has the ability to
decide to be more aggressive (or not overly-defensive)
in pursuing its functional goals, as well as to speciﬁcally
target desired human states/responses.
2. Deriving an approximate optimization solution. We
introduce an approximation to the human model, and
derive a symbolic representation of the gradient of the
robot’s reward function with respect to its actions in
order to enable efﬁcient optimization.
3. Analyzing planning in the human-autonomous car
system. We present the consequences of planning in this
dynamical system, showcasing behaviors that emerge
when rewarding the robot for certain effects on human
state, like making the human slow down, change lanes,
or go ﬁrst through an intersection. We also show that
such behaviors can emerge from simply rewarding the
robot for reaching its goal state fast – the robot be-
comes more aggressive by leveraging its possible effects
on human actions. Finally, we test our hypothesis that
the planner is actually capable of affecting real human
actions in the desired way though an in-lab user study.
Overall, this paper takes a ﬁrst step towards enabling
cars to be aware of (and even leverage) the consequences
that their actions have on other drivers. Even though
admittedly more work is needed to put these ideas in
the ﬁeld, we are encouraged to see planners generate
actions that affect humans in a desired way without the
need for any hand-coded strategies or heuristics.II. Problem Statement
We focus on a human-robot system consisting of an
autonomous (robot) car interacting in an environment
with other human driven vehicles on the road. Our goal
is for the autonomous car to plan its actions in a manner
that is cognizant of their effects on the human driver
actions. We restrict ourselves to the two agent case in
this work, we have an autonomous car Rsharing the
road with a human driver H.
We model the problem as a fully observable dynamical
system, but one in which the robot actions have conse-
quences beyond their immediate effects on the car: they
will also affect human actions which in turn will affect
state.
A state x2Xin our system is continuous, and
includes the positions and velocities of the human and
autonomous (robot) car. The robot can apply continuous
controls uR, which affect state immediately through a
dynamics model fR:
x0=fR(x,uR) (1)
However, the next state the system reaches also depends
on the control the human chooses, uH. This control
affects the intermediate state through a dynamics model
fH:
x00=fH(x0,uH) (2)
The overall dynamics of the system combines the two:
xt+1=fH(fR(xt,ut
R),ut
H) (3)
The robot’s reward function depends on the current
state, the robot’s action, as well as the action that the
human takes at that step in response, rR(xt,ut
R,ut
H).
The key aspect of this formulation is that the robot will
have a model for what u Hwill be, and use that in planning
to optimize its reward.
The robot will use Model Predictive Control
(MPC) [18] at every iteration, it will compute a
ﬁnite horizon sequence of actions to maximize its
reward. It will then execute the ﬁrst one, and replan.
Let x= ( x1, . . . , xN)>denote a ﬁnite horizon se-
quence of states, uH= (u1
H, . . . , uN
H)>denote a ﬁnite
sequence of human’s continuous control inputs, and
uR= (u1
R, . . . , uN
R)>denote a ﬁnite sequence of robot’s
continuous control inputs. Let RRbe the reward over
the MPC time horizon:
RR(x0,uR,uH) =N
å
t=1rR(xt,ut
R,ut
H) (4)
where x0is the current state (the state at the current
iteration), and each state thereafter is obtained through
the dynamics model in (3) from the previous and the
robot and human controls.
At every iteration, the robot needs to ﬁnd the uRthat
maximizes this reward:
u
R=arg max
uRRR(x0,uR,u
H(x0,uR)) (5)
Here, u
H(x0,uR)is what the human would do over the
next Nsteps if the robot were to execute uR.
The robot does not actually know u
H, but in the next
section we propose a model for the human behavior that
the robot can use, along with an approximation to make
(5) tractable.
III. P lanning While Cognizant of Effects
onHuman Action
In order for the robot to solve the ﬁnite horizon
problem from (5) at every iteration, it needs access to
u
H(x0,uR). This would require the robot to have access
to the human’s brain, able to simulate what the human
would do in various scenarios. And yet, autonomous
cars do exist. Typically, we get around this problem by
assuming that u
H(x0,uR) =u
H(x0), e.g. that the human
will maintain their current velocity [13]. In this work, we
break that assumption.
We embrace that the human will take different actions
depending on what actions the robot will choose. To do
this, we model the human as maximizing their own reward
function rH(xt,ut
R,ut
H).
A. General Model
If the robot were to perform uRstarting from x0for
the next Ntime steps, the human would be planning at
every step to maximize their reward for a ﬁnite time
horizon based on the state xtthat would be reached
and the control the robot would apply at that state. For
instance, the robot would execute the ﬁrst control u0
R,
and the human would plan for a ﬁnite time horizon
based on x0and u0
R. The human would then execute
the ﬁrst control in the planned sequence, reaching a new
state x1, where they would observe the robot control u1
R,
and replan. In general, in this model we have:
ut
H(x0,uR) =ut
H(x0,u0:t
R,u0:t1
H) (6)
=arg max
ut:t+N1
HrH(xt,ut
R,ut
H)+ (7)
å
i=t+1:t+N1rH(xi,˜ui
R,ui
H) (8)
Here, ˜uRis the human’s prediction of what the robot will
do, which the human needs in order to be able to plan
for the next few steps. This could be a simple prediction,
like the robot maintaining its velocity, or it could be a
complex prediction, relying on the robot also computing
the optimal plan, moving us to the full game-theoretic
formulation.B. Simplifying Assumption
We simplify this model with an approximation: we
give the human model access to uRfrom the start,
compute the best response for the human, and assume
that to be u
H.
LetRHbe the human reward over the time horizon:
RH(x0,uH,uR) =N
å
t=1rH(xt,ut
R,ut
H) (9)
Our approximation is:
u
H(x0,uR) =arg max
uHRH(x0,uR,uH) (10)
This approximation is motivated by the short time
horizon, meaning we are not assuming the human has
access to the overall plan of the robot, just to the ﬁrst few
time steps – this is easier for a human to predict than a
full sequence of controls, e.g. that the robot will merge
into the human’s lane after a certain amount of time.
The general formulation is a two-player game, but this
avoids the problem of inﬁnite regress by allowing the
robot to play ﬁrst and force a best response from the
human.1
C. Solution
Assuming a known human reward function rH(which
we will obtain later through Inverse Reinforcement
Learning (IRL) [1, 15, 19, 29], see below), we can solve the
optimization in (5) using L-BFGS [2], which is a quasi-
Newton method that stores an approximate inverse Hes-
sian implicitly.
To apply L-BFGS, we need the gradient of (5) with
respect to uR:
¶RR
¶uR=¶RR
¶uH¶u
H
¶uR+¶RR
¶uR(11)
¶RR
¶uHand¶RR
¶uRcan both be computed symbolically
through backward propogation, as we have a represen-
tation of RRin terms of uHand uR. For¶u
H
¶uR, we use
that u
His the minimum from (10), which means that the
gradient of RHevaluated at u
His 0:
¶RH
¶uH(x0,uR,u
H(x0,uR)) = 0 (12)
Now, we can differentiate the expression in equa-
tion (12) with respect to uR:
¶2RH
¶u2
H¶u
H
¶uR+¶2RH
¶uH¶uR¶uR
¶uR=0 (13)
Finally, we can solve for a symbolic expression for¶u
H
¶uR:
1We enforce turn-taking for convenience, and it is justiﬁed in cases
where the robot response is immediate and the human response takes
longer (thus the human accounts for the robot). However, controls
could also be synchronous: the robot would still force a best response
for the human, but starting with the next time step.
¶u
H
¶uR= [¶2RH
¶uH¶uR][¶2RH
¶u2
H]1(14)
and plug it into (11).
D. Implementation Details
In our implementation, we used the software package
Theano [3, 4] to symbolically compute all Jacobians and
Hessians. Theano optimizes the computation graph into
efﬁcient C code, which is crucial for real-time applica-
tions. In our implementation, each step of our optimiza-
tion is solved in approximately 0.3 seconds for horizon
length N=5 on a 2.3 GHz Intel Core i7 processor with
16 GB RAM. Future work will focus on achieving better
computation time and a longer planning horizon.
E. Human Driver Reward
Thus far, we have assumed access to rH(xt,ut
R,ut
H). In
our implementation, we learn this reward function from
human data. We collect demonstrations of a driver in a
simulation environment, and use Inverse Reinforcement
Learning [1, 12, 15, 19, 23, 29] to recover a reward
function that explains the demonstrations.
To handle continuous states and actions, and the fact
that the demonstrations are noisy and possible locally
optimal, we use Continuous Inverse Optimal Control
with Locally Optimal Examples [15]. In what follows, we
recap the algorithm, and present the features we used in
our implementation.
IRL. We parametrize the human reward function as a
linear combination of features:
rH(xt,ut
R,ut
H) =qTf(xt,ut
R,ut
H) (15)
and apply the principle of maximum entropy [28, 29] to
deﬁne a probability distribution over human demonstra-
tions uH, with trajectories that have higher reward being
more probable:
P(uHjx0,q) =exp(RH(x0,uR,uH))R
exp(RH(x0,uR,˜uH))d˜uH(16)
We then do an optimization over the weights qin the
reward function that make the human demonstrations
the most likely:
max
qP(uHjx0,q) (17)
We approximate the partition function in (16) follow-
ing [15], by computing a second order Taylor approxi-
mation around the demonstration:
RH(x0,uR,˜uH)'RH(x0,uR,uH) + ( ˜uHuH)>¶RH
¶uH+
(˜uHuH)>¶2RH
¶u2
H(˜uHuH),
(18)
(a) Features for the boundaries of the road(b) Feature for staying inside the lanes.(c) Features for avoiding other vehicles.
Fig. 2: Features used in IRL for the human driven vehicle. In the heat
map, the warmer colors correspond to higher reward. In (a), we show
the features corresponding to staying within road boundaries, in (b),
we show the features for staying within each lane, and in (c) we show
non-spherical gaussian features corresponding to avoiding collisions.
which makes the integral in (16) a Gaussian integral,
with a closed form solution. See [15] for more details.
Features. Fig.2 shows the heat map of our features. The
heat map of features we have used are shown in Figure 2.
The warmer colors correspond to higher rewards. In
Fig. 2(a), we show the features corresponding to staying
within the boundaries of the roads. In Fig. 2(b), we have
features corresponding to staying within each lane, and
in Fig. 2, we have features corresponding to collision
avoidance, which are non-spherical Gaussians, and their
major axis is along the vehicle’s heading. In addition to
the features shown in the ﬁgure, we include a quadratic
function of the speed to capture efﬁciency as an objective.
Demonstrations. We collected demonstrations of a single
human driver in an environment with multiple au-
tonomous cars, which followed precomputed routes.
Despite the simplicity of our features and robot actions
during the demonstrations, the learned human model
is enough for the planner to produce behavior that is
human-interpretable (case studies in Sec. IV), and that
can affect human action in the desired way (user study
in Sec. V).
IV . C aseStudies
In this section, we introduce 3 driving scenarios, and
show the result of our planner assuming a simulated hu-
man driver, highlighting the behavior that emerges from
different robot reward functions. In the next section, we
put the planner to the test with real users and measure
the effects of the robot’s plan. Fig.3 illustrates our three
scenarios, and contains images from the actual user
study data showcasing not just robot actions but also
their real effects. Here, the yellow car is the autonomous
vehicle, and the red car is the human driven vehicle.
A. Conditions for Analysis Across Scenarios
In all three scenarios, we start from an initial position
of the vehicles on the road, as shown in Fig.3. In the
control condition, we give the car the reward function to
avoid collisions and have high velocity. We refer to this as
Rcontrol . In the experimental condition, we augment this
Speed: 0.05
Speed: 0.60
Speed: 0.30
Speed: 0.25
Speed: 0.54
Speed: 0.30
Speed: 0.99
Speed: 0.00
Speed: 0.00
Speed: 0.33
(a) Scenario 1: make human slow down(b) Scenario 2: make human go left/right (c) Scenario 3: make human go first
Autonomous Vehicle
Human Driven VehicleA void HumanAffect HumanFig. 3: Driving scenarios. In (a), the car plans to merge in front of the human in order to make them slow down. In (b), the car plans to direct
the human to another lane, and uses its heading to choose which lane the human will go to. In (c), the car plans to back up slightly in order
to make the human proceed ﬁrst at the intersection. None of these plans use any hand coded strategies. They emerge out of optimizing with
a learned model of how humans react to robot actions. In the training data for this model, the learned was never exposed to situations where
another car stopped at an orientation as in (b), or backed up as in (c). However, by capturing human behavior in the form of a reward, the
model is able to generalize to these situations, enabling the planner to ﬁnd creative ways of achieving the desired effects.
reward function with a speciﬁc desired human action
(e.g. low speed, lateral position, etc.). We refer to this as
Rcontrol +Raffect. Sections IV-C through IV-E contrast the
two plans for each of our three scenarios. Sec. IV-F shows
what happens when instead of explicitly giving the robot
a reward function designed to trigger certain effects on
the human, we simply task the robot with reaching a
destination as quickly as possible.
B. Driving Simulator
We model the dynamics of the vehicles as a simple
point-mass model. Let the state of the system be x=
[x y qv]>, where x,yare the coordinates of the vehicle,
qis the heading, and vis the speed. We let u= [u1u2]>
represent the control input, where u1is the steering input
and u2is the acceleration. We also use aas the friction
coefﬁcient, then the dynamics model of the vehicle is:
[˙x˙y˙q˙v] = [ vcos(q)vsin(q)vu1u2av].
(19)
C. Scenario 1: Make Human Slow Down
In this scenario, we show how an autonomous vehicle
can plan to make a human driver slow down in a
highway driving setting. The vehicles start at the initial
conditions depicted on left in Fig. 3 (a), in separate lanes.
In the experimental condition, we augment the robot’s
reward with the negative of the square of the human
velocity, which encourages the robot to slow the human
down.
Fig.3(a) contrasts our two conditions. In the control
condition, the human moves forward uninterrupted. In
(a) Reward for Scenario 2, making the human to turn left.(b) Reward for Scenario 2, making the human to turn right.(c) Reward for Scenario 3, making the human to cross first.
Fig. 4: Heat map of the reward functions in scenarios 2 and 3. The
warmer colors show higher reward values. In (a), (b), the reward
function of the autonomous vehicle is plotted, which is a function of
the human driven vehicle’s position. In order to affect the driver to go
left, the reward is higher on the left side of the road in (a), and to affect
the human to go right in (b), the rewards are higher on the right side
of the road. In (c), the reward of the autonomous vehicle is plotted for
scenario 3 with respect to the position of the human driven car. Higher
rewards correspond to making the human cross the intersection.
the experimental condition, however, the robot plans to move
in front of the person, expecting that this will make them slow
down.
D. Scenario 2: Make Human Go Left/Right
In this scenario, we show how an autonomous vehicle
can plan to change the human’s lateral location or lane.
The vehicles start at the initial conditions depicted on left
in Fig. 3 (b), in the same lane, with the robot in front of
the human. In the experimental condition, we augment
the robot’s reward with the lateral position of the human,
in two ways, to encourage the robot to make the human
go either left (orange border image) or right (blue border
image). The two reward additions are shown in Fig.4(a)
and (b).
Fig.3 (b) contrasts our two conditions. In the control
condition, the human moves forward, and might decide
to change lanes. In the experimental condition, however, the
robot plans to purposefully occupy two lanes (using either
a positive or negative heading), expecting this will make the
human move around it by using the unoccupied lane.
E. Scenario 3: Make Human Go First
In this scenario, we show how an autonomous vehicle
can plan to make the human proceed ﬁrst at an intersec-
tion. The vehicles start at the initial conditions depicted
on left in Fig. 3 (c), with both human and robot stopped
at the 4-way intersection. In the experimental condition,
we augment the robot’s reward with a feature based
on the yposition of the human car yHrelative to the
middle of the intersection y0. In particular, we used the
hyperbolic tangent of the difference, tanh (yHy0). The
reward addition is shown in Fig.4 (c).
Fig.3 (c) contrasts our two conditions. In the control
condition, the car goes in front of the human. In the ex-
perimental condition, however, the robot plans to purposefully
back up slightly, expecting this will make the human cross
ﬁrst. Note that this could be interpreted as a communica-
tive behavior, but communication was never explicitly
encouraged in the reward function. Instead, this behavior
emerged out of the goal of affecting human actions.
This is perhaps the most surprising behavior of the
three scenarios, because it is not something human
drivers do. However, our user study suggests that human
drivers to respond to this in the expected way. Further,
pedestrians exhibit this behavior at times, stepping back
away from an intersection to let a car go by ﬁrst.
F. Behaviors Also Emerge from Efﬁciency
Thus far, we explicitly encoded a desired effect on
human actions in the reward we gave the robot to
optimize. We have also found, however, that behaviors
like the ones we have seen so far can emerge out of the
need for efﬁciency.
Fig.5 (bottom) shows the generated plan for when the
robot is given the goal to reach a point in the left lane as
quickly as possible (reward shown in Fig.6). By modeling
the effects its actions have on the human actions, the
robot plans to merge in front of the person, expecting
that they will slow down.
In contrast, the top of the ﬁgure shows the generated
plan for when the robot uses a simple (constant velocity)
model of the person. In this case, the robot assumes that
merging in front of the person can lead to a collision,
and defensively waits for the person to pass, merging
behind them.
We hear about this behavior often in autonomous cars
today: they are defensive. Enabling them to plan in a
manner that is cognizant that they can affect other driver
actions can make them more efﬁcient at achieving their
goals.V . U serStudy
The previous section showed the robot’s plans when
interacting with a simulated user that perfectly ﬁts the
robot’s model of the human. Next, we present the results
of a user study that evaluates whether the robot can
successfully have the desired effects on real users.
A. Experimental Design
We use the same 3 scenarios as in the previous section.
Manipulated Factors. We manipulate a single factor: the
reward that the robot is optimizing, as described in Sec.
IV-A. This leads to two conditions: the experimental con-
dition where the robot is encouraged to have a particular
effect on human state though the reward Rcontrol +Raffect,
and the control condition where that aspect is left out
of the reward function and the robot is optimizing only
Rcontrol (three conditions for Scenario 2, where we have
two experimental conditions, one for the left case and
one for the right case).
Dependent Measures. For each scenario, we measure the
value along the user trajectory of the feature added to
the reward function for that scenario, Raffect. Speciﬁcally,
we measure the human’s negative squared velocity in
Scenario 1, the human’s xaxis location relative to center
in scenario 2 in Scenario 2, and whether the human went
ﬁrst or not through the intersection in Scenario 3 (i.e. a
ﬁltering of the feature that normalizes for difference in
timing among users and measures the desired objective
directly).
Hypothesis. We hypothesize that our method enables
the robot to achieve the effects it desires not only in
simulation, but also when interacting with real users:
The reward function that the robot is optimizing has
a signiﬁcant effect on the measured reward during
interaction. Speciﬁcally, R affect is higher, as planned,
when the robot is optimizing for it.
Subject Allocation. We recruited 10 participants (2 fe-
male, 8 male). All the participants owned drivers li-
cense with at least 2 years of driving experience. We
ran our experiments using a 2D driving simulator, we
have developed with the driver input provided through
driving simulator steering wheel and pedals as shown
in Figure 1. We used a within-subjects design and coun-
terbalanced the order of the conditions.
B. Analysis
Scenario 1: A repeated measures ANOVA showed the
square speed to be signiﬁcantly lower in the experimen-
tal condition than in the control condition ( F(1, 160 ) =
228.54, p<0.0001). This supports or hypothesis: the
human moved slower when the robot planned to have
this effect on the human.
We plot the speed and latitude proﬁle of the human
driven vehicle over time for all trajectories in Fig.7.
Fig.7(a) shows the speed proﬁle of the control condition
Initial
InitialRobot Lets Human Pass
Robot Merges in FrontFig. 5: A time lapse for Sec. IV-F, where the autonomous vehicle’s goal is to reach a ﬁnal point in the left lane. In the top scenario, the autonomous
vehicle has a simple model of the human driver that does not account for the inﬂuence of its actions on the human actions, so it acts more
defensively, waiting for the human to pass ﬁrst. In the bottom, the autonomous vehicle uses the learned model of the human driver, so it acts
more aggressively and reaches its goal faster.
(a) Single feature corresponding to distance to goal on the top left.(b) All features present for autonomous vehicle’s reward function.
Fig. 6: Heat map of reward function for reaching a ﬁnal goal at the
top left of the road. As shown in the ﬁgure, the goal position is darker
showing more reward for reaching that point.
trajectories in gray, and of the experimental condition
trajectories in orange. Fig.7(b) shows the mean and
standard error for each condition. In the control con-
dition, human squared speed keeps increasing. In the
experimental condition however, by merging in front of
the human, the robot is triggering the human to brake
and reduce speed, as planned. The purple trajectory
represents a simulated user that perfectly matches the
robot’s model, showing the ideal case for the robot.
The real interaction moves signiﬁcantly in the desired
direction, but does not perfectly match the ideal model,
since real users do not act exactly as the model would
predict.
The ﬁgure also plots the yposition of the vehicles
along time, showing that the human has not travelled
as far forward in the experimental condition.
Scenario 2: A repeated measures ANOVA showed a
signiﬁcant effect for the reward factor ( F(2, 227 ) =55.58,
Fig. 7: Speed proﬁle and latitude of human driven vehicle for Scenario
1. The ﬁrst column shows the speed of all trajectories with its mean
and standard errors in the bottom graph. The second column shows the
latitude of the vehicle over time; similarly, with the mean and standard
errors. The grey trajectories correspond to the control condition, and
the orange trajectories correspond to the experimental condition: the
robot decides to merge in front of the users and succeeds at slowing
them down. The purple plot corresponds to a simulated user that
perfectly matches the model that the robot is using.
p<0.0001). A post-hoc analysis with Tukey HSD
showed that both experimental conditions were signiﬁ-
cantly different from the control condition, with the user
car going more to the left than in the control condition
when Raffect rewards left user positions ( p<0.0001), and
more to the right in the other case ( p<0.001). This
supports our hypothesis.
Fig. 8: Trajectories of human driven vehicle for Scenario 2 (a) with
mean and standard error (right). Orange (blue) indicates conditions
where the reward encouraged the robot to affect the user to go left
(right).
We plot all the trajectories collected from the users
in Fig.8. Fig.8(a) shows the control condition trajectories
in grey, while the experimental conditions trajectories
are shown in orange (for left) and blue (for right). By
occupying two lanes, the robot triggers an avoid behavior
from the users in the third lane. Here again, purple
curves show a simulated user, i.e. the ideal case for the
robot.
Scenario 3: An ordinal logistic regression with user as a
random factor showed that signiﬁcantly more users went
ﬁrst in the intersection in the experimental condition
than in the baseline ( c2(1, 129 ) = 106.41, p<.0001).
This supports our hypothesis.
Fig.9 plots the yposition of the human driven vehicle
with respect to the xposition of the autonomous ve-
hicle. For trajectories that have a higher yposition for
the human vehicle than the xposition for the robot,
the human car has crossed the intersection before the
autonomous vehicle. The lines corresponding to these
trajectories travel above the origin, which is shown with
a blue square in this ﬁgure. The mean of the orange
lines travel above the origin, which means that the au-
tonomous vehicle has successfully affected the humans
to cross ﬁrst. The grey lines travel below the origin, i.e.
the human crossed second.
Overall, our results suggest that the robot was able to
affect the human state in the desired way, even though
it does not have a perfect model of the human.
VI.discussion
Summary. In this paper, we formalized the interaction
between an autonomous (robot) vehicle and a human
Fig. 9: Plot of yHwith respect to xR. The orange curves correspond
to when the autonomous vehicle affects the human to cross the
intersection ﬁrst. The grey curves correspond to when the nominal
setting.
driver as a dynamical system, in which the actions of
the robot affect those of the human and vice-versa. We
introduced an approximate solution that enables the
robot to optimize its own reward within this system.
The resulting plans can purposefully modify human be-
havior, and can achieve the robot’s goal more efﬁciently.
Our user study suggests that this is not only true in
simulation, but also true when tested with real users.
Limitations. All this work happened in a simple driving
simulator. To put this on the road, we will need more
emphasis on safety, as well as a longer planning horizon.
The former involves the use of formal methods and
safe control as well as better models of users: not all
drivers act the same and replanning is not the end-
solution to address this. Using a probabilistic dynamics
model as opposed to planning with the most probable
human actions, as well as estimating driving style, will
be important next steps.
An even bigger limitation is that we currently focus on
a single human driver. Looking to the interaction among
multiple vehicles is not just a computational challenge,
but also a modeling one – it is not immediately clear how
to formulate the problem when multiple human-driven
vehicles are interacting and reacting to each other.
Conclusion. Despite these limitations, we are en-
couraged to see autonomous cars generate human-
interpretable behaviors though optimization, without re-
lying on hand-coded heuristics. We also look forward to
applications of these ideas beyond autonomous driving,
to mobile robots, UAVs, and in general to human-robot
interactive scenarios where robot actions can inﬂuence
human actions.
VII. A cknowledgments
This work was partially supported by Berkeley Deep-
Drive, NSF grants CCF-1139138 and CCF-1116993, ONR
N00014-09-1-0230, and an NDSEG Fellowship.
References
[1] Pieter Abbeel and Andrew Y Ng. Exploration and appren-
ticeship learning in reinforcement learning. In Proceedings
of the 22nd international conference on Machine learning ,
pages 1–8. ACM, 2005.
[2] Galen Andrew and Jianfeng Gao. Scalable training of L1-
regularized log-linear models. In Proceedings of the 24th
international conference on Machine learning , pages 33–40.
ACM, 2007.
[3] Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James
Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas
Bouchard, and Yoshua Bengio. Theano: new features and
speed improvements. Deep Learning and Unsupervised
Feature Learning NIPS 2012 Workshop, 2012.
[4] James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal
Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph
Turian, David Warde-Farley, and Yoshua Bengio. Theano:
a CPU and GPU math expression compiler. In Proceedings
of the Python for Scientiﬁc Computing Conference (SciPy) ,
June 2010. Oral Presentation.
[5] MWMG Dissanayake, Paul Newman, Steven Clark,
Hugh F Durrant-Whyte, and Michael Csorba. A solu-
tion to the simultaneous localization and map building
(SLAM) problem. Robotics and Automation, IEEE Transac-
tions on , 17(3):229–241, 2001.
[6] Paolo Falcone, Francesco Borrelli, Jahan Asgari,
Hongtei Eric Tseng, and Davor Hrovat. Predictive
active steering control for autonomous vehicle systems.
IEEE Transactions on Control Systems Technology , 15(3):
566–580, May 2007.
[7] Paolo Falcone, Francesco Borrelli, H Eric Tseng, Jahan
Asgari, and Davor Hrovat. Integrated braking and steer-
ing model predictive control approach in autonomous
vehicles. In Advances in Automotive Control , volume 5,
pages 273–278, 2007.
[8] Paolo Falcone, H. Eric Tseng, Francesco Borrelli, Jahan
Asgari, and Davor Hrovat. MPC-based yaw and lateral
stabilisation via active front steering and braking. Vehicle
System Dynamics , 46(sup1):611–628, September 2008.
[9] Alison Gray, Yiqi Gao, J Karl Hedrick, and Francesco
Borrelli. Robust predictive control for semi-autonomous
vehicles with an uncertain driver model. In Intelligent
Vehicles Symposium (IV), 2013 IEEE , pages 208–213. IEEE,
2013.
[10] Christoph Hermes, Christian Wohler, Kurt Schenk, and
Franz Kummert. Long-term vehicle motion prediction. In
2009 IEEE Intelligent Vehicles Symposium , pages 652–657,
2009.
[11] Markus Kuderer, Henrik Kretzschmar, Christoph Sprunk,
and Wolfram Burgard. Feature-based prediction of trajec-
tories for socially compliant navigation. In Proceedings of
Robotics: Science and Systems , Sydney, Australia, July 2012.
doi: 10.15607/RSS.2012.VIII.025.
[12] Markus Kuderer, Shilpa Gulati, and Wolfram Burgard.
Learning driving styles for autonomous vehicles from
demonstration. In Proceedings of the IEEE International
Conference on Robotics & Automation (ICRA), Seattle, USA ,
volume 134, 2015.
[13] Stéphanie Lefèvre, Ashwin Carvalho, Yiqi Gao, H Eric
Tseng, and Francesco Borrelli. Driver models for person-
alised driving assistance. Vehicle System Dynamics , 53(12):
1705–1720, 2015.
[14] John Leonard, Jonathan How, Seth Teller, Mitch Berger,Stefan Campbell, Gaston Fiore, Luke Fletcher, Emilio Fraz-
zoli, Albert Huang, Sertac Karaman, et al. A perception-
driven autonomous urban vehicle. Journal of Field Robotics ,
25(10):727–774, 2008.
[15] Sergey Levine and Vladlen Koltun. Continuous inverse
optimal control with locally optimal examples. arXiv
preprint arXiv:1206.4617 , 2012.
[16] Jesse Levinson, Jake Askeland, Jan Becker, Jennifer Dol-
son, David Held, Soeren Kammel, J Zico Kolter, Dirk
Langer, Oliver Pink, Vaughan Pratt, et al. Towards fully
autonomous driving: Systems and algorithms. In 2011
IEEE Intelligent Vehicles Symposium (IV) , pages 163–168.
[17] Brandon Luders, Mangal Kothari, and Jonathan P How.
Chance constrained rrt for probabilistic robustness to
environmental uncertainty. In AIAA guidance, navigation,
and control conference (GNC), Toronto, Canada , 2010.
[18] Manfred Morari, CE Garcia, JH Lee, and DM Prett. Model
predictive control . Prentice Hall Englewood Cliffs, NJ, 1993.
[19] Andrew Y Ng, Stuart J Russell, et al. Algorithms for
inverse reinforcement learning. In Proceedings of the 17th
international conference on Machine learning , pages 663–670,
2000.
[20] Stefanos Nikolaidis, Ramya Ramakrishnan, Keren Gu, and
Julie Shah. Efﬁcient model learning from joint-action
demonstrations for human-robot collaborative tasks. In
Proceedings of the Tenth Annual ACM/IEEE International
Conference on Human-Robot Interaction , pages 189–196.
ACM, 2015.
[21] Vasumathi Raman, Alexandre Donzé, Dorsa Sadigh,
Richard M Murray, and Sanjit A Seshia. Reactive synthesis
from signal temporal logic speciﬁcations. In Proceedings of
the 18th International Conference on Hybrid Systems: Compu-
tation and Control , pages 239–248. ACM, 2015.
[22] Dorsa Sadigh and Ashish Kapoor. Safe control under
uncertainty. arXiv preprint arXiv:1510.07313 , 2015.
[23] Masamichi Shimosaka, Tetsuya Kaneko, and Kentaro
Nishi. Modeling risk anticipation and defensive driving
on residential roads with inverse reinforcement learning.
In2014 IEEE 17th International Conference on Intelligent
Transportation Systems (ITSC) , pages 1694–1700. IEEE, 2014.
[24] Peter Trautman and Andreas Krause. Unfreezing the
robot: Navigation in dense, interacting crowds. In 2010
IEEE/RSJ International Conference on Intelligent Robots and
Systems (IROS) , pages 797–803.
[25] Chris Urmson, Joshua Anhalt, Drew Bagnell, Christopher
Baker, Robert Bittner, MN Clark, John Dolan, Dave Dug-
gins, Tugrul Galatali, Chris Geyer, et al. Autonomous driv-
ing in urban environments: Boss and the urban challenge.
Journal of Field Robotics , 25(8):425–466, 2008.
[26] Ramanarayan Vasudevan, Victor Shia, Yiqi Gao, Ricardo
Cervera-Navarro, Ruzena Bajcsy, and Francesco Borrelli.
Safe semi-autonomous control with enhanced driver mod-
eling. In American Control Conference (ACC), 2012 , pages
2896–2903. IEEE, 2012.
[27] Michael P Vitus and Claire J Tomlin. A probabilistic
approach to planning and control in autonomous urban
driving. In 2013 IEEE 52nd Annual Conference on Decision
and Control (CDC) , pages 2459–2464.
[28] Brian D Ziebart. Modeling purposeful adaptive behavior
with the principle of maximum causal entropy. 2010.
[29] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and
Anind K Dey. Maximum entropy inverse reinforcement
learning. In AAAI , pages 1433–1438, 2008.

Dorsa Sadigh
Shankar Sastry
Sanjit Seshia
Anca Dragan