Autonomous Robots
https://doi.org/10.1007/s10514-018-9746-1
Planning for cars that coordinate with people: leveraging effects on
human actions for planning and active information gathering over
human internal state
Dorsa Sadigh1·Nick Landolﬁ1·Shankar S. Sastry1·Sanjit A. Seshia1·Anca D. Dragan1
Received: 16 February 2017 / Accepted: 2 April 2018
© Springer Science+Business Media, LLC, part of Springer Nature 2018
Abstract
Traditionally, autonomous cars treat human-driven vehicles like moving obstacles. They predict their future trajectories and
plan to stay out of their way. While physically safe, this results in defensive and opaque behaviors. In reality, an autonomous
car’s actions will actually affect what other cars will do in response, creating an opportunity for coordination. Our thesis is
that we can leverage these responses to plan more efﬁcient and communicative behaviors. We introduce a formulation of
interaction with human-driven vehicles as an underactuated dynamical system, in which the robot’s actions have consequences
on the state of the autonomous car, but also on the human actions and thus the state of the human-driven car. We model these
consequences by approximating the human’s actions as (noisily) optimal with respect to some utility function. The robot uses
the human actions as observations of her underlying utility function parameters. We ﬁrst explore learning these parameters
ofﬂine, and show that a robot planning in the resulting underactuated system is more efﬁcient than when treating the person
as a moving obstacle. We also show that the robot can target speciﬁc desired effects, like getting the person to switch lanes or
to proceed ﬁrst through an intersection. We then explore estimating these parameters online, and enable the robot to perform
active information gathering: generating actions that purposefully probe the human in order to clarify their underlying utility
parameters, like driving style or attention level. We show that this signiﬁcantly outperforms passive estimation and improves
efﬁciency. Planning in our model results in coordination behaviors: the robot inches forward at an intersection to see if can
go through, or it reverses to make the other car proceed ﬁrst. These behaviors result from the optimization, without relying
on hand-coded signaling strategies. Our user studies support the utility of our model when interacting with real users.
Keywords Planning for human–robot interaction ·Mathematical models of human behavior ·Autonomous driving
This is one of several papers published in Autonomous Robots compris-
ing the “Special Issue on Robotics Science and Systems”.
This paper combines work from Sadigh et al. ( 2016b ,c). It adds a
general formulation of the problem as a game, discusses its
limitations, and lays out the assumptions we make to reduce it to a
tractable problem. On the experimental side, it adds an analysis of the
adaptivity of the behaviors produced to initial conditions for both
ofﬂine and active estimation, an analysis of the beneﬁts of active
estimation on the robot’s actual reward, and results on actively
estimating user intentions as opposed to just driving style.
BDorsa Sadigh
dorsa@cs.stanford.edu
1Department of Computer Science, Stanford University,
Stanford, USA1 Introduction
Currently, autonomous cars tend to be overly defensive and
obliviously opaque . When needing to merge into another
lane, they will patiently wait for another driver to pass ﬁrst.
When stopped at an intersection and waiting for the driver on
the right to go, they will sit there unable to wave them by. They
are very capable when it comes to obstacle avoidance, lane
keeping, localization, active steering and braking (Urmson
et al. 2008 ; Levinson et al. 2011 ; Falcone et al. 2007 ,2008 ,
2007 ; Dissanayake et al. 2001 ; Leonard et al. 2008 ). But
when it comes to other human drivers, they tend to rely on
simplistic models: for example, assuming that other drivers
will be bounded disturbances (Gray et al. 2013 ; Raman et al.
2015 ), they will keep moving at the same velocity (Vitus and
Tomlin 2013 ; Luders et al. 2010 ; Sadigh and Kapoor 2015 ),
123
Autonomous Robots
(a) (b) (c) (d) (e)
Fig. 1 We equip autonomous cars with a model of how humans will
react to the car’s actions ( a). We test the planner in user studies, where
the car ﬁgures out that it can nudge into the human’s lane to check theirdriving style ( b,c): if it gets evidence that they are attentive it merges in
front, expecting that the human will slow down; else, it retracts back toits lane ( c). We also see coordination behavior at an intersection, with
the car planning to inch forward to ﬁnd out more about how the humanwill act, or even inch backwards to incentivize the human to go throughﬁrst ( d)
or they will approximately follow one of a set of known
trajectories (Vasudevan et al. 2012 ;H e r m e se ta l . 2009 ).
These models predict the trajectory of other drivers as
if those drivers act in isolation. They reduce human–robot
interaction to obstacle avoidance: the autonomous car’s task
is to do its best to stay out of the other drivers’ way. It willnot nudge into the lane to test if the other driver yields, norcreep into the intersection to assert its turn for crossing.
In reality, the actions of an autonomous car affect the
actions of other drivers. Our goal is to leverage these
effects in planning to improve efﬁciency and coordina-
tion with human drivers.
We are not the ﬁrst to propose that robot actions inﬂuence
human actions. Work in social navigation recognized this,
and addressed it by treating the human and the robot as beingpart of a team—a team that works together to make surethat each agent reaches their goal, and everyone avoids each
other (Trautman and Krause 2010 ; Trautman et al. 2013 ;
Trautman 2013 ; Kuderer et al. 2015 ). The robot computes
acoupled plan by optimizing the team’s objective jointly
over the human and robot plans, assuming the human will
follow their part of the plan, and re-planning at every step.Nikolaidis et al. ( 2015 ) recognized that the human might stray
away from the plan that the robot computed, and modeled the
human as switching to the robot’s plan at every time step withsome probability.
We propose that when people stray away from the cou-
pled plan, there is a fundamental reason for this: they have
a different objective altogether . This is particularly true in
driving, where coupled planning would assume that the per-
son is just as interested in the robot reaching its goal as they
are in themselves reaching theirs—selﬁsh drivers are likelyjust trying to get home and not get into an accident; they are
optimizing for something different.
In this work, we explicitly account for the robot’s inﬂuence
on the person by modeling the person as optimizing their ownobjective or utility function in response to the robot’s actions.
We develop an optimization-based method for planning an
autonomous vehicle’s behavior in a manner that is cognizantof the effects it will have on human driver actions via this
model. This optimization leads to plans like the ones in Fig. 1.
Assuming a universal human model that the robot learns
ofﬂine, we see that the the orange car in (c) decides to cut in
front of the human driver in order to more efﬁciently reach
its goal. It arrives at this plan by anticipating that taking this
action will cause the human to brake and make space for it.This comes in contrast to treating the person as an obstacle
that moves (b) and being more defensive. Since not all drivers
are the same, we also arm the robot with the ability to collectinformation about the human driver online. It then decides to
nudge into the person’s lane (d) and only merges if it gets evi-
dence that the person is paying attention; otherwise it retreatsback to its lane. We ﬁnd fascinating behaviors at intersections(e): the car inches forward to test human attention, or even
inches backwards to get the person to cross ﬁrst through the
intersection. These can be interpreted as signaling behaviors,but they emerge out of optimizing to affect human actions,
without ever explicitly modeling human inferences.
We achieve this by planning in an underactuated dynami-
cal system : the robot’s actions change not only robot state, but
also inﬂuence human actions and thus human state. We model
other drivers as acting approximately optimally according tosome reward function that depends on state, human actions,as well as robot actions. We explore learning this human
reward function ofﬂine, as well as online during the inter-
action: here, the robot has the opportunity to use its ownactions to actively gather more information about the under-
lying reward. Our contributions are as follows
1:
1A preliminary version of our results was reported in Sadigh et al.
(2016a ,b). This paper extends that work by providing more detailed
discussion and experiments...
123
Autonomous Robots
1. Form alism of Inter action with Drivers We begin by for-
malizing the interaction between a human and a robot as apartially observable stochastic game (POSG) in Sect. 2.T h e
human and the robot can both act to change the state of the
world, they have partial information because they don’t knoweach other’s reward functions, and they arrive at some tupleof policies that are at an equilibrium.
This formulation has two issues: intractability, especially
in continuous state and action spaces, and failing to capturehuman behavior, because humans tend to not follow Nash
equilibria in day to day tasks (Hedden and Zhang 2002 ).
We introduce a simpliﬁcation of this formulation to an
underactuated system. We assume that the robot decides on
a trajectory u
R, and the human computes a best response to
uR(as opposed to trying to inﬂuence uRas would happen in
a game). This reduction enforces turn-taking, and providesus with a dynamics model in which the human observes (or
predicts) the robot’s actions prior to selecting her actions.
It maintains our ability to model the effects of the robot’sactions on the human, because the robot can inﬂuence which
actions the human takes by selecting actions that force a
particular response.
2. Approxim ate Optimiz ation Solution for Known Hum an
ModelAssuming a known reward function for the human
[which we learn ofﬂine through Inverse ReinforcementLearning (Ng et al. 2000 ; Abbeel and Ng 2005 ; Ziebart et al.
2008 ; Levine and Koltun 2012 )], we derive an approximate
solution for our system based on Model Predictive Controland a quasi-newton optimization in Sect. 3.A te v e r ys t e p ,
the robot replans a trajectory u
Rby reasoning about the opti-
mization that the human would do based on a candidate uR.
We use implicit differentiation to obtain a gradient of thehuman’s trajectory with respect to the robot’s. This enables
the robot to compute a plan in close to real-time.
3. Extension to Online Inference of Hum anR e w ardThe
solution based on estimating human reward ofﬂine from
training data is useful in some cases, but ultimately every
driver is different, and even the same driver is sometimesmore or less aggressive, more or less attentive, and so on. In
Sect. 6, we thus also explore estimating the human reward
function online.
This turns the problem into a partially observable Markov
decision process (POMDP), with the human reward param-
eters as the hidden state. Prior work that incorporates somenotion of human state into planning has thus far separatedestimation and planning, always using the current estimate
of the human state to determine what the robot should do
(Javdani et al. 2015 ; Fern et al. 2007 ; Bandyopadhyay et al.
2013 ). Although efﬁcient, these approximations sacriﬁce an
important aspect of POMDPs: the ability to actively gather
information .In recent years, many efforts have developed more efﬁ-
cient algorithms for solving continuous POMDPs (Chaudhariet al. 2013 ; Seiler et al. 2015 ; Agha-Mohammadi et al. 2014 );
however, computing the transition function for our POMDP
is costly as it involves solving for human’s best response.This renders the aforementioned approaches ineffective asthey usually require either a succinct transition function or
an easily computable one.
Our work takes advantage of the underactuated system
to gather information about the human reward parameters.
Rather than relying on passive observations, the robot actu-
ally accounts for the fact that humans will react to theiractions: it uses this knowledge to select actions that will trig-
ger human reactions which in turn will clarify the internal
state. This is in line with work in active information gather-ing in manipulation, safe exploration, or patrolling (Javdaniet al. 2013 ; Atanasov et al. 2014 ; Atanasov 2015 ), now used
over human state as opposed to physical state.
4. An alysis of Pl anning in Hum an–Robot Driving Sce-
narios We present the consequences of planning in this
dynamical system in Sect. 4, showcasing behaviors that
emerge when rewarding the robot for certain effects onhuman state, like making the human slow down, change
lanes, or go ﬁrst through an intersection. We also show
that such behaviors can emerge from simply rewarding therobot for reaching its goal state fast—the robot becomes
more aggressive by leveraging its possible effects on human
actions. This does not happen always: in Sect. 7, we show the
robot maintains a belief over the human driver model, andstarts nudging into their lane to ﬁgure out if they are going
to let the robot merge, or inching forward at a 4-way stop.
We test the planner in two user studies in a driving sim-
ulator: one with a human reward learned ofﬂine in Sect. 5,
and one with online estimation in Sect. 8. Our results sug-
gest that despite the approximation our algorithm makes, itsigniﬁcantly outperforms the “people as moving obstacles”
interaction paradigm. We ﬁnd that the robot achieves signif-
icantly higher reward when planning in the underactuatedsystem, and that active information gathering does outper-form passive estimation with real users.
Overall, this paper takes a step towards robots that account
for their effects on human actions in situations that are notentirely cooperative, and leverage these effects to coordinate
with people. Natural coordination and interaction strategies
are not hand-coded, but emerge out of planning in our model.
2 Generalformalismforhuman–robot
interactionasagame
To enable a robot to autonomously generate behavior for
interaction and coordination with a human, we need to set up
123
Autonomous Robots
a model that goes beyond a single agent acting in the physi-
cal world among moving obstacles, and consider two-agentmodels. One aspect that sets interaction with people apart
from typical multi-agent models like those used for multi-
robot planning is that we do not know or have direct controlover what the human is trying to do and how. Furthermore,the human also does not know what the robot is trying to do
nor how the robot is going to do it.
Even the simplest formulation of the problem becomes
a two player game, and in what follows we introduce such
a formulation and expose some of its issues, including the
computational complexity of planning in such a model aswell as the model’s inaccuracy in capturing human behavior.
We then propose an approximation that simpliﬁes planning in
the game to planning in an underactuated system: the robot
had direct control over its actions, but also has a model forhow its actions will inﬂuence those of the human.
Much of robotics research focuses on how to enable a robot
to achieve physical tasks, often times in the face of percep-tion and movement error—of partially observable worlds and
nondeterministic dynamics (Prentice and Roy 2009 ; Javdani
et al. 2013 ; Patil et al. 2015 ). Part of what makes human–robot
interaction difﬁcult is that even if we assume the physical
world to be fully observable and deterministic, we are still
left with a problem as complex as an incomplete information
repeated two player game . It is a game because it involves
multiple rational (or rather, approximately rational) agents
who can take actions to maximize their own (expected) util-
ities. It is repeated because unlike single shot games, theagents act over a time horizon. It is incomplete informa-
tion because the agents do not necessarily know each others’
reward functions (Aumann et al. 1995 ).
Partially Observ able Stoch astic G ameExtrapolating from
the (PO)MDP formulation of a robot acting in isolation to
human–robot interaction, we can formulate interaction asa special case of a Partially-Observable Stochastic Game
(POSG) (Hansen et al. 2004 ): there are two “players”, the
robot Rand the human H;a te v e r ys t e p t, they can apply
control inputs u
t
R∈URand ut
H∈UH; they each have a
reward function, rRandrH; and there is a state space Swith
states sconsisting of both the physical state x,a sw e l la s
reward parameters θRandθH.
Including the reward parameters in the state is an unusual
trick, necessary here in order to ensure that the human and the
robot do not have direct access to each others’ reward func-tions, while maintaining a well deﬁned POSG: Rdoes not
observe θ
H, and Hdoes not observe θR, but both agents
can technically evaluate each reward at any state-controltuple(s
t,ut
R,ut
H)just because scontains the needed reward
parameter information: s=(x,θR,θH)—if an agent knew
the state, it could evaluate the other agent’s reward.To simplify the physical world component and focus on
the interaction component, we assume fully observable phys-ical states with deterministic dynamics. The observations
in the game are thus the physical state xand the controls
that each agent applies at each time step. The dynamicsmodel is deterministic, affecting xthrough the control inputs
and leaving the reward parameters unchanged (a reasonable
assumption for relatively short interactions).
Aside 1 The POSG above is not necessarily the most direct
extension of single-agent planning in deterministic fully
observable state spaces to interactions. Collaborative interac-
tions have been modeled simply as a centralized multi-agentsystem with a shared reward function (Kuderer et al. 2015 )
(i.e. r
R=rH), but this reduces to treating the human
as another robot that the system has full control over—essentially the system has more degrees of freedom to
control, and there is no difference between the human DoFs
and the robot DoFs. Unlike the POSG, it assumes that thehuman and the robot know each other’s reward, and evenmore, that their rewards are identical. This can be true of
speciﬁc collaborative tasks, but not of interactions in gen-
eral: robots do not know exactly what humans want, humansdo not know exactly what robots have been programmed to
optimize for, and their rewards might have common terms but
will not be identical. This happens when an autonomous carinteracts with other drivers or with pedestrians, and it even
happens in seemingly collaborative scenarios like rehabili-
tation, in which very long horizon rewards might be alignedbut not short-term interaction ones. The POSG formulationcaptures this intricacy of general interaction.
Aside 2 The POSG above is the simplest extension of the
single-agent models to interactions between a human and
a robot that do not share a reward function or know each
others’ rewards. More complex models might include richerstate information (such as human’s beliefs about the robot,trust, estimation of capability, mood, etc.), and might allow
it to change over time.
Limit ations of the G ame Formul ation The POSG formu-
lation is a natural way to characterize interaction from the
perspective of MDP-like models, but is limited in two funda-
mental ways: (1) its computational complexity is prohibitiveeven in discrete state and action spaces (Bernstein et al. 2002 ;
Hansen et al. 2004 ) and no methods are known to handle con-
tinuous spaces, and (2) it is not a good model for how peopleactually work—people do not solve games in everyday tasks
when they are not playing chess (Hedden and Zhang 2002 ).
Furthermore, solutions here are tuples of policies that are ina Nash equilibrium, and it is not clear what equilibrium toselect.
123
Autonomous Robots
3 Approximatesolutionasanunderactuated
system
To alleviate the limitations from above, we introduce an
approximate close to real-time solution, with a model ofhuman behavior that does not assume that people computeequilibria of the game.
3.1 Assumptions that simplify the POSG
Our approximation makes several simplifying assumptionsthat turn the game into an ofﬂine learning phase in which
the robot learns the human’s reward function, followed by
an online planning phase in which the robot is solving anunderactuated control problem:
Separation of Estim ation & Control We separate the pro-
cess of computing actions for the robot into two stages. First,the robot estimates the human reward function parameters
θ
Hofﬂine. Second, the robot exploits this estimate as a ﬁxed
approximation to the human’s true reward parameters dur-ing planning. In the ofﬂine phase, we estimate θ
Hfrom user
data via Inverse Reinforcement Learning (Ng et al. 2000 ;
Abbeel and Ng 2005 ; Ziebart et al. 2008 ; Levine and Koltun
2012 ). This method relies heavily on the approximation of
all humans to a constant set of reward parameters, but we
will relax this separation of estimation and control in Sect. 6.
Model Pre dictive Control (MPC) Solving the POSG
requires planning to the end of the full-time horizon. We
reduce the computation required by planning for a shorter
horizon of Ntime steps. We execute the control only for the
ﬁrst time step, and then re-plan for the next Nat the next
time step (Camacho and Alba 2013 ).
Letx=(x1,..., xN)/latticetopdenote a sequence of states over
a ﬁnite horizon, N, and let uH=(u1
H,..., uN
H)/latticetopand
uR=(u1
R,..., uN
R)/latticetopdenote a ﬁnite sequence of contin-
uous control inputs for the human and robot, respectively.
We deﬁne RRas the robot’s reward over the ﬁnite MPC time
horizon:
RR(x0,uR,uH)=N/summationdisplay
t=1rR(xt,ut
R,ut
H), (1)
where x0denotes the present physical state at the current iter-
ation, and each state thereafter is obtained from the previousstate and the controls of the human and robot using a given
dynamics model, f.
At each iteration, we desire to ﬁnd the sequence u
Rwhich
maximizes the reward of the robot, but this reward ostensibly
depends on the actions of the human. The robot might attempt
to inﬂuence the human’s actions, and the human, optimizingits own reward functions, might likewise attempt to inﬂuencethe actions of the robot. Despite our reduction to a ﬁnite
time horizon, the game formulation still demands computingequilibria to the problem. Our core assumption, which we
discuss next, is that this is not required for most interactions:
that a simpler model of what the human does sufﬁces.
Simpliﬁc ation of the Hum anM o delTo avoid computing
these equilibria, we propose to model the human as respond-
ing rationally to some ﬁxed extrapolation of the robot’sactions. At every time step, t,Hcomputes a simple estimate
ofR’s plan for the remaining horizon, ˜u
t+1:N
R, based on the
robot’s previous actions u0:t
R. Then the human computes its
planuHas a best response (Fudenberg and Tirole 1991 )t o
this estimate. We remark that with this assumption the human
is not modeled as a passive agent since the reward functioncan model the behavior of any active agent, if allowed to have
arbitrary complexity. With this simpliﬁcation, we reduce the
general game to a Stackelberg competition: the human com-putes its best outcome while holding the robots plan ﬁxed.This simpliﬁcation is justiﬁed in practice because usually
the agents in driving scenarios are competing with each other
rather than collaborating. Solutions to Stackelberg games arein general more conservative in competitive regimes since
an information advantage is given to the second player. As a
result, the quality of a good solution found in a Stackelberggame model does not decrease in practice. We additionally
use a short time horizon, so not much is lost by this approxi-
mation. But fundamentally, we do assume that people wouldnot try to inﬂuence the robot’s actions, and this is a limitationwhen compared to solving the POSG.
LetR
Hbe the human reward over the time horizon:
RH(x0,uR,uH)=N/summationdisplay
t=1rH(xt,ut
R,ut
H), (2)
then we can compute the control inputs of the human from
the remainder of the horizon by:
ut
H(x0,u0:t
R,˜ut+1:N
R)=arg maxut+1:T
HRH(xt,˜ut+1:N
R,ut+1:N
H).
(3)
This human model would certainly not work well in adver-
sarial scenarios, but our hypothesis, supported by our results,
is that it is useful enough in day-to-day tasks to enable robotsto be more effective and more ﬂuent interaction partners.
In our work, we propose to make the human’s estimate ˜u
R
equal to the actual robot control sequence uR. Our assump-
tion that the time horizon is short enough that the human
can effectively extrapolate the robot’s course of action moti-
vates this decision. With this presumption, the human’s planbecomes a function of the initial state and robot’s true plan:
123
Autonomous Robots
u∗
H(x0,uR)=arg maxuHRH(xt,uR,uH). (4)
This is now an underactuated system the robot has direct con-
trol over (can actuate) uRand indirect control over (cannot
actuate but does affect) uH. However, the dynamics model
in our setup is more sophisticated than in typical underactu-
ated systems because it models the response of the humans to
the robot’s actions. Evaluating the dynamics model requiressolving for the optimal human response, u
∗
H.
The system is also a special case of an MDP, with the state
as in the POSG, the actions being the actions of the robot, andthe world dynamics being dictated by the human’s responseand the resulting change on the world from both human and
robot actions.
The robot can now plan in this system to determine which
u
Rwould lead to the best outcome for itself:
u∗
R=arg maxuRRR/parenleftBig
x0,uR,u∗
H(x0,uR)/parenrightBig
. (5)
3.2 Planning with Quasi-Newton optimization
Despite the reduction to a single agent complete information
underactuated system, the dynamics remain too complex tosolve in real-time. We lack an analytical form for u
∗
H(x0,uR)
which forces us to solve ( 4) each time we evaluate the dynam-
ics.
Assuming a known human reward function rH[which
we will obtain later through Inverse Reinforcement Learning
(IRL), see Ng et al. ( 2000 ), Abbeel and Ng ( 2005 ), Ziebart
et al. ( 2008 ) and Levine and Koltun ( 2012 )], we can solve ( 5)
locally, using gradient-based methods. Our main contribution
is agnostic to the particular optimization method, but we use
L-BFGS (Andrew and Gao 2007 ), a quasi-Newton method
that stores an approximate inverse Hessian implicitly result-
ing in fast convergence.
To perform the local optimization, we need the gradient
of (2) with respect to uR:
∂RR
∂uR=∂RR
∂uH∂u∗
H
∂uR+∂RR
∂uR(6)
We can compute both∂RR
∂uHand∂RR
∂uRsymbolically through
back-propagation because we have a representation of RR
in terms of uHanduR.
What remains,∂u∗
H
∂uR, is difﬁcult to compute because u∗
His
technically the outcome of a global optimization. To compute
∂u∗
H
∂uR, we use the method of implicit differentiation. Since
RHis a smooth function whose minimum can be attained,
we conclude that for the unconstrained optimization in ( 4),
the gradient of RHwith respect to uHevaluates to 0 at its
optimum u∗
H:∂RH
∂uH/parenleftBig
x0,uR,u∗
H(x0,uR)/parenrightBig
=0( 7 )
Now, we differentiate the expression in ( 7) with respect to
uR:
∂2RH
∂u2
H∂u∗
H
∂uR+∂2RH
∂uH∂uR∂uR
∂uR=0( 8 )
Finally, we solve for a symbolic expression of∂u∗
H
∂uR:
∂u∗
H
∂uR=/bracketleftbigg
−∂2RH
∂uH∂uR/bracketrightbigg/bracketleftBigg
∂2RH
∂u2
H/bracketrightBigg−1
(9)
and insert it into ( 6), providing an expression for the gradient
∂RR
∂uR.
3.3 Offline estimation of human reward parameters
Thus far, we have assumed access to rH(xt,ut
R,ut
H). In our
implementation, we learn this reward function from humandata. We collect demonstrations of a driver in a simulation
environment, and use Inverse Reinforcement Learning (Ng
et al. 2000 ; Abbeel and Ng 2005 ; Ziebart et al. 2008 ;L e v i n e
and Koltun 2012 ; Shimosaka et al. 2014 ; Kuderer et al. 2015 )
to recover a reward function that explains the demonstrations.
To handle continuous state and actions space, and cope
with noisy demonstrations that are perhaps only locally opti-mal, we use continuous inverse optimal control with locally
optimal examples (Levine and Koltun 2012 ).
We parametrize the human reward function as a linear
combination of features:
r
H(xt,ut
R,ut
H)=θ/latticetop
Hφ(xt,ut
R,ut
H), (10)
and apply the principle of maximum entropy (Ziebart et al.
2008 ; Ziebart 2010 ) to deﬁne a probability distribution over
human demonstrations uH, with trajectories that have higher
reward being more probable:
P(uH|x0,θH)=exp/parenleftbig
RH(x0,uR,uH)/parenrightbig
/integraltext
exp/parenleftbig
RH(x0,uR,˜uH)/parenrightbig
d˜uH. (11)
We then optimize the weights θHin the reward function
that make the human demonstrations the most likely:
max
θHP/parenleftBig
uH|x0,θH/parenrightBig
(12)
We approximate the partition function in ( 11) following
(Levine and Koltun 2012 ), by computing a second order Tay-
lor approximation around the demonstration:
123
Autonomous Robots
Fig. 2 Features used in IRL for the human driven vehicle; warmer col-
ors correspond to higher reward. We illustrate features correspondingto (a) respecting road boundaries, bholding lanes, and cavoiding col-
lisions with other cars
RH(x0,uR,˜uH)/similarequalRH(x0,uR,uH)+(˜uH−uH)/latticetop∂RH
∂uH
+(˜uH−uH)/latticetop∂2RH
∂u2
H(˜uH−uH),
(13)
which results in the the integral in ( 11) reducing to a Gaussian
integral, with a closed form solution. See Levine and Koltun
(2012 ) for more details.
We display a heat map of features we used in Fig. 2.T h e
warmer colors correspond to higher rewards. In addition tothe features shown in the ﬁgure, we include a quadratic func-
tion of the speed to capture efﬁciency in the objective. The
ﬁve features include:
–φ
1∝c1·exp(−c2·d2): distance to the boundaries of
the road, where dis the distance between the vehicle and
the road boundaries and c1andc2are appropriate scaling
factors as shown in Fig. 2a.
–φ2: distance to the middle of the lane, where the function
is speciﬁed similar to φ1as shown in Fig. 2b.
–φ3=(v−vmax)2: higher speed for moving forward
through, where vis the velocity of the vehicle, and vmax
is the speed limit.
–φ4=βH·n: heading; we would like the vehicle to have
a heading along with the road using a feature, where βH
is the heading of H, and nis a normal vector along the
road.
–φ5corresponds to collision avoidance, and is a non-
spherical Gaussian over the distance of Hand R, whose
major axis is along the robot’s heading as shown in
Fig. 2c.
We collected demonstrations of a single human driving
for approximately an hour in an environment with multi-
ple autonomous cars, which followed precomputed routes.
Despite the simplicity of our features and robot actions duringthe demonstrations, the learned human model proved sufﬁ-
cient for the planner to produce human-interpretable behavior
(case studies in Sect. 4), and actions which affected human
action in the desired way (user study in Sect. 5).3.4 Implementation details
In our implementation, we used the software package Theano
(Bergstra et al. 2010 ; Bastien et al. 2012 ) to symbolically
compute all Jacobians and Hessians. Theano optimizes thecomputation graph into efﬁcient C code, which is crucial forreal-time applications.
This implementation enables us to solve each step of the
optimization in equation ( 5) in approximately 0.3 s for hori-
zon length N=5 on a 2.3 GHz Intel Core i7 processor with
16 GB RAM. Future work will focus on achieving better
computation time and a longer planning horizon.
4 Casestudieswithoﬄineestimation
We noted earlier that the state of the art autonomous driv-
ing plans conservatively because of its simple assumptions
regarding the environment and vehicles on the road. In our
experiments, we demonstrate that an autonomous vehicle canpurposefully affect human drivers, and can use this ability to
gather information about the human’s driving style and goals.
In this section, we introduce 3 driving scenarios, and show
the result of our planner assuming a simulated human driver,highlighting the behavior that emerges from different robot
reward functions. In the next section, we test the planner with
real users and measure the effects of the robot’s plan. Figure 3
illustrates our three scenarios, and contains images from the
actual user study data.
4.1 Conditions for analysis across scenarios
In all three scenarios, we start from an initial position ofthe vehicles on the road, as shown in Fig. 3.I nt h e control
condition, we give the car the reward function to avoid col-lisions and have high velocity. We refer to this as R
control .I n
the experimental condition, we augment this reward func-
tion with a term corresponding to a desired human action(e.g. low speed, lateral position, etc.). We refer to this as
R
control+Raffect . Sections 4.3through 4.5contrast the two
plans for each of our three scenarios. Section 4.6shows what
happens when instead of explicitly giving the robot a rewardfunction designed to trigger certain effects on the human, we
simply task the robot with reaching a destination as quickly
as possible.
4.2 Driving simulator
We use a simple point-mass model of the car’s dynamics.We deﬁne the physical state of the system x=[xyψv]
/latticetop,
where x,yare the coordinates of the vehicle, ψis the heading,
andvis the speed. We let u=[u1u2]/latticetoprepresent the control
input, where u1is the steering input and u2is the acceleration.
123
Autonomous Robots
(a) (b) (c)
Fig. 3 Driving scenarios. In ( a), the car plans to merge in front of the
human in order to make them slow down. In ( b), the car plans to direct
the human to another lane, and uses its heading to choose which lanethe human will go to. In ( c), the car plans to back up slightly in order to
make the human proceed ﬁrst at the intersection. None of these plansuse any hand coded strategies. They emerge out of optimizing with alearned model of how humans react to robot actions. In the training
data for this model, the learned was never exposed to situations where
another car stopped at an orientation as in ( b), or backed up as in ( c).
However, by capturing human behavior in the form of a reward, themodel is able to generalize to these situations, enabling the planner toﬁnd creative ways of achieving the desired effects
We denote the friction coefﬁcient by μ. We can write the
dynamics model:
[˙x˙y˙ψ˙v]=[v·cos(ψ) v ·sin(ψ) v ·u1u2−μ·v](14)
4.3 Scenario 1: Make human slow down
In this highway driving setting, we demonstrate that an
autonomous vehicle can plan to cause a human driver to slowdown. The vehicles start at the initial conditions depicted on
left in Fig. 3a, in separate lanes. In the experimental condi-
tion, we augment the robot’s reward with the negative of thesquare of the human velocity, which encourages the robot toslow the human down.
Figure 3a contrasts our two conditions. In the control
condition, the human moves forward uninterrupted. In the
experimental condition, however, the robot plans to move in
front of the person, anticipating that this will cause the human
to brake.
4.4 Scenario 2: Make human go left/right
In this scenario, we demonstrate that an autonomous vehi-cle can plan to affect the human’s lateral location, making
the human switch lanes. The vehicles start at the initial con-
ditions depicted on left in Fig. 3b, in the same lane, with
the robot ahead of the human. In the experimental condition,
we augment the robot’s reward with the lateral position of
the human, in two ways, to encourage the robot to make thehuman go either left (orange border image) or right (blue bor-
Fig. 4 Heat map of the reward functions in scenarios 2 and 3. The
warmer colors show higher reward values. In ( a,b), the reward function
of the autonomous vehicle is plotted, which is a function of the humandriven vehicle’s position. In order to affect the driver to go left, thereward is higher on the left side of the road in ( a), and to affect the
human to go right in ( b), the rewards are higher on the right side of the
road. In ( c), the reward of the autonomous vehicle is plotted for scenario
3 with respect to the position of the human driven car. Higher rewardscorrespond to making the human cross the intersection
der image). The two reward additions are shown in Fig. 4a,
b.
Figure 3b contrasts our two conditions. In the control con-
dition, the human moves forward, and might decide to changelanes. In the experimental condition, however, the robot plans
to intentionally occupy two lanes (using either a positive
or negative heading), anticipating this will make the humanavoid collision by switching into the unoccupied lane.
4.5 Scenario 3: Make human go first
In this scenario, we demonstrate that an autonomous vehiclecan plan to cause the human to proceed ﬁrst at an intersection.
123
Autonomous Robots
The vehicles start at the initial conditions depicted on the left
in Fig. 3c, with both human and robot stopped at the 4-way
intersection. In the experimental condition, we augment the
robot’s reward with a feature based on the yposition of the
human car yHrelative to the middle of the intersection y0.I n
particular, we used the hyperbolic tangent of the difference,tanh(y
H−y0). The reward addition is shown in Fig. 4c.
Figure 3c contrasts our two conditions. In the control
condition, the robot proceeds in front of the human. In the
experimental condition, however, the robot plans to inten-
tionally reverse slightly, anticipating that this will induce
the human cross ﬁrst. We might interpret such a trajectory
as communicative behavior, but communication was never
explicitly encouraged in the reward function. Instead, the
goal of affecting human actions led to this behavior.
Reversing at an intersection is perhaps the most surpris-
ing result of the three scenarios, because it is not an action
human drivers take. In spite of this novelty, our user study
suggests that human drivers respond in the expected way:they proceed through the intersection. Further, pedestrians
sometimes exhibit behavior like the robot’s, stepping back
from an intersection in order to let a car pass ﬁrst.
4.6 Behaviors also emerge from efficiency
Thus far, we have explicitly encoded a desired effect onhuman actions, and optimized it as a component of the robot’sreward. We have also found, however, that behaviors like
those we have seen so far can emerge out of the need for
efﬁciency.
Figure 5(bottom) shows the generated plan for when the
robot is given the goal to reach a point in the left lane as
quickly as possible (reward shown in Fig. 6). By modeling
the effects its actions have on the human actions, the robotplans to merge in front of the person, expecting that they will
slow down.
Fig. 5 A time lapse where the autonomous vehicle’s goal is to reach a
ﬁnal point in the left lane. In the top scenario, the autonomous vehiclehas a simple model of the human driver that does not account for theinﬂuence of its actions on the human actions, so it acts more defensively,waiting for the human to pass ﬁrst. In the bottom, the autonomous vehi-cle uses the learned model of the human driver, so it acts less defensivelyand reaches its goal faster
Fig. 6 Heat map of reward function for reaching a ﬁnal goal at the
top left of the road. As shown in the ﬁgure, the goal position is darkershowing more reward for reaching that point
In contrast, the top of the ﬁgure shows the generated plan
for when the robot uses a simple (constant velocity) model
of the person. In this case, the robot assumes that mergingin front of the person can lead to a collision, and defensively
waits for the person to pass, merging behind them.
We hear about this behavior often in autonomous cars
today: they are defensive. Enabling them to plan in a manner
that is cognizant that they can affect other driver actions can
make them more efﬁcient at achieving their goals.
4.7 The robot behavior adapts to the situation
Throughout the case studies, we see examples of coordina-tion behavior that emerges out of planning in our system:going in front of someone knowing they will brake, slowing
and nudging into another lane to incentivize a lane change,
or backing up to incentivize that the human proceeds ﬁrstthrough an intersection. Such behaviors could possibly behand-engineered for those particular situations rather than
autonomously planned. However, we advocate that the need
to plan comes from versatility: from the fact that the plannercan adapt the exact strategy to each situation.
With this work, we are essentially shifting the design bur-
den from designing policies to designing reward functions.We still need to decide on what we want the robot to do: do
we want it to be selﬁsh, do we want it to be extra polite and
try to get every human to go ﬁrst, or do we want it to be some-where in between? Our paper does not answer this question,and designing good reward functions remains challenging.
However, this work does give us the tools to autonomously
generate (some of) the strategies needed to then optimizesuch reward functions when interacting with people. With
policies, we’d be crafting that the car should inch forward
or backwards from the intersection, specifying by how muchand at what velocity, and how this depends on where the other
cars are, and how it depends on the type of intersection. With
this work, barring local optima issues and the fact that humanmodels could always be improved, all that we need to specify
123
Autonomous Robots
Fig. 7 The robot adapts its merging behavior depending on the relative
position of the person: it does not always cut the person off: sometimesit merges behind the person, and if it starts too close (depending on howthe reward function is set up) it will not merge at all
is the regular reward that we would give an autonomous car,
or, if we want it to purposefully inﬂuence human behavior,the desired outcome. The car ﬁgures out how to achieve it,adapting its actions to the different settings.
Figure 7shows a spectrum of behaviors for the robot
depending on where it starts relative to the human: frommerging behind the person, to not merging, to merging in
front.
5 Userstudywithoﬄineestimation
The previous section showed the robot’s plans when interact-
ing with a simulated user that perfectly ﬁts the robot’s model
of the human. Next, we present the results of a user study
that evaluates whether the robot can successfully have thedesired effects on real users.
5.1 Experimental design
We use the same 3 scenarios as in the previous section.Manipul atedFactors We manipulate a single factor: the
reward that the robot is optimizing, as described in Sect. 4.1.
This leads to two conditions: the experimental condition
where the robot is encouraged to have a particular effect
on human state though the reward R
control +Raffect , and
the control condition where that aspect is left out of the
reward function and the robot is optimizing only Rcontrol
(three conditions for Scenario 2, where we have two experi-
mental conditions, one for the left case and one for the right
case).
Depen dent Me asures For each scenario, we measure the
value along the user trajectory of the feature added to thereward function for that scenario, Raffect . Speciﬁcally, we
measure the human’s negative squared velocity in Scenario1, the human’s xaxis location relative to center in Scenario 2,
and whether the human went ﬁrst or not through the intersec-
tion in Scenario 3 (i.e. a ﬁltering of the feature that normalizesfor difference in timing among users and measures the desiredobjective directly).
Hypothesis We hypothesize that our method enables the
robot to achieve the effects it desires not only in simulation,but also when interacting with real users:
The reward function that the robot is optimizing has a
signiﬁcant effect on the measured reward during inter-action. Speciﬁcally, R
affect is higher, as planned, when
the robot is optimizing for it.
Subject Alloc ation We recruited 10 participants (2 female, 8
male). All the participants owned drivers license with at least
2 years of driving experience. We ran our experiments using
a 2D driving simulator, we have developed with the driverinput provided through driving simulator steering wheel and
pedals.
5.2 Analysis
Scen ario 1 A repeated measures ANOV A showed the square
speed to be signiﬁcantly lower in the experimental conditionthan in the control condition ( F(1,160)=228.54, p<
0.0001). This supports our hypothesis: the human moved
slower when the robot planned to have this effect on thehuman.
We plot the speed and latitude proﬁle of the human driven
vehicle over time for all trajectories in Fig. 8. Figure 8a
shows the speed proﬁle of the control condition trajectories ingray, and of the experimental condition trajectories in orange.
Figure 8b shows the mean and standard error for each con-
dition. In the control condition, human squared speed keepsincreasing. In the experimental condition however, by merg-
ing in front of the human, the robot is triggering the human
to brake and reduce speed, as planned. The purple trajectoryrepresents a simulated user that perfectly matches the robot’s
model, showing the ideal case for the robot. The real inter-
action moves signiﬁcantly in the desired direction, but doesnot perfectly match the ideal model, since real users do notact exactly as the model would predict.
The ﬁgure also plots the yposition of the vehicles along
time, showing that the human has not travelled as far forwardin the experimental condition.
Scen ario 2 A repeated measures ANOV A showed a sig-
niﬁcant effect for the reward factor ( F(2,227)=55.58,
p<0.0001). A post-hoc analysis with Tukey HSD showed
that both experimental conditions were signiﬁcantly differ-
ent from the control condition, with the user car going moreto the left than in the control condition when R
affect rewards
123
Autonomous Robots
(a)
(b) (d)(c)
Fig. 8 Speed proﬁle and latitude of the human driven vehicle for Sce-
nario 1. The ﬁrst column shows the speed of all trajectories with its meanand standard errors in the bottom graph. The second column shows thelatitude of the vehicle over time; similarly, with the mean and standarderrors. The gray trajectories correspond to the control condition, and theorange trajectories correspond to the experimental condition: the robotdecides to merge in front of the users and succeeds at slowing themdown. The purple plot corresponds to a simulated user that perfectlymatches the model that the robot is using
left user positions ( p<0.0001), and more to the right in the
other case ( p<0.001). This supports our hypothesis.
We plot all the trajectories collected from the users
in Fig. 9. Figure 9a shows the control condition trajecto-
ries in gray, while the experimental conditions trajectoriesare shown in orange (for left) and blue (for right). By occu-
pying two lanes, the robot triggers an avoid behavior from
the users in the third lane. Here again, purple curves show asimulated user, i.e. the ideal case for the robot.
Scen ario 3 An ordinal logistic regression with user as a ran-
dom factor showed that signiﬁcantly more users went ﬁrstin the intersection in the experimental condition than in the
baseline ( χ
2(1,129)=106.41,p<0.0001). This supports
our hypothesis.
Figure 10plots the yposition of the human driven vehicle
with respect to the xposition of the autonomous vehicle. For
trajectories that have a higher yposition for the human vehi-
cle than the xposition for the robot, the human car has crossed
the intersection before the autonomous vehicle. The lines cor-
responding to these trajectories travel above the origin, which
is shown with a blue square in this ﬁgure. The mean of theorange lines travel above the origin, which means that the
autonomous vehicle has successfully affected the humans to
cross ﬁrst. The gray lines travel below the origin, i.e. thehuman crossed second.(a) (b)
Fig. 9 Trajectories of the human driven vehicle for Scenario 2. The ﬁrst
column ashows all the trajectories, and the second column bshows the
mean and standard error. Orange (blue) indicates conditions where thereward encouraged the robot to affect the user to go left (right)
(a) (b)
Fig. 10 Plot of yHwith respect to xR. The orange curves correspond to
when the autonomous vehicle affects the human to cross the intersectionﬁrst. The gray curves correspond to the nominal setting
Over all, our results suggest that the robot was able to affect
the human state in the desired way, even though it does nothave a perfect model of the human.
6 Extensiontoonlineestimationofthe
humanmodel
We have thus far in our approximate solution treated the
human’s reward function as estimated once, ofﬂine. This has
worked well in our user study on seeking speciﬁc coordina-
tion effects on the human, like slowing down or going ﬁrstthrough the intersection. But in general, this is bound to run
123
Autonomous Robots
into problems, because not all people behave according to
the same estimated θH.
Different drivers have different driving styles . Some are
very defensive, more so than our learned model. Others are
much more aggressive, and for instance would not actuallybrake when the car merges in front of them. Even for thesame driver, their style might change over time, for instance
when they get distracted on their phone.
In this section we relax our assumption of an ofﬂine esti-
mation of the human’s reward parameters θ
H. Instead, we
explore estimating this online. We introduce an algorithm
which maintains a belief over a space of candidate rewardfunctions, and enable the robot to perform inference over
this space throughout the interaction. We maintain tractabil-
ity by clustering possible θ
Hs into a few options that the robot
maintains a belief over.
6.1 A POMDP with human reward as the hidden
variable
The human’s actions are inﬂuenced by their internal reward
parameters θHthat the robot does not directly observe. So
far, we estimated θHofﬂine and solved an underactuated
system, a special case of an MDP. Now, we want to be able
to adapt our estimate of θHonline, during interaction. This
turns the problem into a partially observable markov decisionprocess (POMDP) with θ
Has the hidden state. By putting
θHin the state, we now have a know dynamics model like in
the underactuated system before for the robot and the humanstate, and we assume θ
Hto remain ﬁxed regardless of the
robot’s actions.
If we could solve the POMDP, the robot would estimate
θHfrom the human’s actions, optimally trading off between
exploiting it’s current belief over θHand actively taking
information gathering actions intended to cause human reac-
tions, which result in a better estimate of θH.
Because POMDPs cannot be solved tractably, several
approximations have been proposed for similar problem for-
mulations (Javdani et al. 2015 ; Lam et al. 2015 ; Fern et al.
2007 ). These approximations are passively estimating the
human internal state, and exploiting the belief to plan robot
actions.2
In this work, we take the opposite approach: we focus
explicitly on active information gathering Our formulation
enables the robot to choose to actively probe the human, and
thereby improve its estimate of θH. We leverage this method
in conjunction with exploitation methods, but the algorithm
we present may also be used alone if human internal state
(reward parameters) estimation is the robot’s primary objec-tive.
2One exception is Nikolaidis et al. ( 2016 ), who propose to solve the full
POMDP, albeit for discrete and not continuous state and action spaces.6.2 Simplification to information gathering
We denote a belief in the value of the hidden variable, θ,a s
a distribution b(θ), and update this distribution according to
the likelihood of observing a particular human action, giventhe state of the world and the human internal state:
b
t+1(θ)∝bt(θ)·P(ut
H|xt,uR,θ). (15)
In order to update the belief b, we require an observation
model. Similar to before, we assume that actions with lower
reward are exponentially less likely, building on the principle
of maximum entropy (Ziebart et al. 2008 ):
P(uH|x,uR,θ)∝exp/parenleftBig
Rθ
H(x0,uR,uH)/parenrightBig
. (16)
To make explicit our emphasis on taking actions which
effectively estimate θ, we redeﬁne the robot’s reward func-
tion to include an information gain term, i.e., the difference
between entropies of the current and updated beliefs: H(bt)−
H(bt+1). The entropy over the belief H(b)evaluates to:
H(b)=−/summationtext
θb(θ)log(b(θ))/summationtext
θb(θ). (17)
We now optimize our expected reward with respect to
the hidden state θ, and this optimization explicitly entails
reasoning about the effects that the robot actions will have
on the observations, i.e., the actions that the human will take
in response, and how useful these observations will be inshattering ambiguity about θ.
6.3 Explore-exploit trade-off
In practice, we use information gathering in conjunction with
exploitation. We do not solely optimize the information gainterm H(b
t)−H(bt+1), but optimize it in conjunction with the
robot’s actual reward function assuming the current estimate
ofθ:
raugmented
R(xt,uR,uH)=λ(H(bt)−H(bt+1))
+rR(xt,uR,uH,bt) (18)
At the very least, we do this as a measure of safety,
e.g., we want an autonomous car to keep avoiding collisionseven when it is actively probing a human driver to test their
reactions. We choose λexperimentally, though existing tech-
niques that can better adapt λover time (Vanchinathan et al.
2014 ).
123
Autonomous Robots
(a) (b) (c)
Fig. 11 Our three scenarios, along with a comparison of robot plans for
passive estimation (gray) versus active information gathering (orange).In the active condition, the robot is purposefully nudging in or brakingto test human driver attentiveness. The color of the autonomous car in
the initial state is yellow, but changes to either gray or orange in casesof passive and active information gathering respectively
6.4 Solution via model predictive control
To ﬁnd the control inputs for the robot we locally solve:
u∗
R=arg maxuREθ/bracketleftBig
RR/parenleftBig
x0,uR,u∗,θ
H(x0,uR)/parenrightBig/bracketrightBig
(19)
over a ﬁnite horizon N, where u∗,θ
H(x0,uR)corresponds to
the actions the human would take from state x0if the robot
executed actions uR. This objective generalizes ( 5) with an
expectation over the current belief over θ,b0.
We still assume that the human maximizes their own
reward function, rθ
H(xt,ut
R,ut
H); we add the superscript θ
to indicate the dependence on the hidden state. We can write
the sum of human rewards over horizon Nas:
Rθ
H(x0,uR,uH)=N−1/summationdisplay
t=0rθ
H(xt,ut
R,ut
H) (20)
Computing this over the continuous space of possible
reward parameters θis intractable even with discretization.
Instead, we learn clusters of θs ofﬂine via IRL, and online
use estimation to ﬁgure out which cluster best matches the
human.
Despite optimizing the trade-off in ( 18), we do not claim
that our method as-is can better solve the general POMDP
formulation: only that it can be used to get better estimates ofhuman internal state. Different tradeoffs λwill result in dif-
ferent performance. Our results below emphasize the utility
of gathering information, but also touch on the implicationsfor active information gathering on R
R.7 Casestudieswithonlineestimation
In this section, we show simulation results that use the
method from the previous section to estimate human driver
type in the interaction between an autonomous vehicleand a human-driven vehicle. We consider three different
autonomous driving scenarios. In these scenarios, the human
is either distracted or attentive during different driving exper-iments. The scenarios are shown in Fig. 11, where the yellow
car is the autonomous vehicle, and the white car is the human
driven vehicle. Our goal is to plan to actively estimate the
human’s driving style in each one of these scenarios, by usingthe robot’s actions.
7.1 Attentive versus distracted human driver models
Our technique requires reward functions rθ
Hthat model the
human behavior for a particular internal state θ. We obtain a
generic driver model via Continuous Inverse Optimal Controlwith Locally Optimal Examples (Levine and Koltun 2012 )
from demonstrated trajectories in a driving simulator in an
environment with multiple autonomous cars, which followedprecomputed routes.
We then adjust the learned weights to model attentive ver-
sus distractive drivers. Speciﬁcally, we modify the weights
of the collision avoidance features, so the distracted humanmodel has less weight for these features. Therefore, the dis-
tracted driver is more likely to collide with the other cars
while the attentive driver has high weights for the collisionavoidance feature. In future work, we plan to investigate ways
of automatically clustering learned θ
Hs from data from dif-
ferent users, but we show promising results even with thesesimple options.
123
Autonomous Robots
7.2 Manipulated factors
We manipulate the reward function that the robot is optimiz-
ing. In the passive condition, the robot optimizes a simple
reward function for collision avoidance based on the cur-rent belief estimate. It then updates this belief passively, byobserving the outcomes of its actions at every time step. In
the active condition, the robot trades off between this reward
function and information gain in order to explore the human’sdriving style.
We also manipulate the human internal reward parame-
tersto be attentive or distracted . The human is simulated to
follow the ideal model of reward maximization for our two
rewards.
7.3 Scenarios and qualitative results
Scen ario 1: Nu dging In to Explore on aHighw ayIn this
scenario, we show an autonomous vehicle actively explor-
ing the human’s driving style in a highway driving setting.We contrast the two conditions in Fig. 11a. In the passive
condition, the autonomous car drives on its own lane with-
out interfering with the human throughout the experiment,
and updates its belief based on passive observations gath-ered from the human car. However, in the active condition,
the autonomous car actively probes the human by nudging
into her lane in order to infer her driving style. An attentivehuman signiﬁcantly slows down (timid driver) or speeds up
(aggressive driver) to avoid the vehicle, while a distracted
driver might not realize the autonomous actions and main-tain their velocity, getting closer to the autonomous vehicle .
It is this difference in reactions that enables the robot to better
estimate θ.
Scen ario 2: Br aking to Explore on aHighw ayIn the sec-
ond scenario, we show the driving style can be explored bythe autonomous car probing the human driver behind it. The
two vehicles start in the same lane as shown in Fig. 11b,
where the autonomous car is in the front. In the passive con-dition, the autonomous car drives straight without exploring
or enforcing any interactions with the human driven vehi-
cle.In the active condition, the robot slows down to actively
probe the human and ﬁnd out her driving style. An attentive
human would slow down and avoid collisions while a dis-
tracted human will have a harder time to keep safe distancebetween the two cars.
Scen ario 3: Nu dging In to Explore atan Intersection In
this scenario, we consider the two vehicles at an intersection,
where the autonomous car actively tries to explore human’sdriving style by nudging into the intersection. The initial con-
ditions of the vehicles are shown in Fig. 11c. In the passive
condition, the autonomous car stays at its position withoutprobing the human, and only optimizes for collision avoid-Fig. 12 Legends indicating active/passive robots, attentive/distracted
humans, and real user/ideal model used for all following ﬁgures
(a) (b)
(c) (d)
(e) (f)
Fig. 13 The probability that the robot assigns to attentive as a function
of time, for the attentive (left) and distracted (right). Each plot comparesthe active algorithm to passive estimation, showing that active informa-tion gathering leads to more accurate state estimation, in simulation andwith real users
ance. This provides limited observations from the human car
resulting in a low conﬁdence belief distribution. In the active
condition, the autonomous car nudges into the intersection
to probe the driving style of the human. An attentive human
would slow down to stay safe at the intersection while a dis-tracted human will not slow down.
123
Autonomous Robots
7.4 Quantitative results
Throughout the remainder of the paper, we use a common
color scheme to plot results for our experimental conditions.
We show this common scheme in Fig. 12: darker colors (black
and red) correspond to attentive humans, and lighter colors(gray and orange) correspond to distracted humans. Further,
the shades of orange correspond to active information gath-
ering, while the shades of gray indicate passive informationgathering. We also use solid lines for real users, and dotted
lines for scenarios with an ideal user model learned through
inverse reinforcement learning.
Figure 13plots, using dotted lines, the beliefs over time for
the attentive (left) and distracted (right) conditions, compar-
ing in each the passive (dotted black and gray respectively)with the active method (dotted dark orange and light orangerespectively). In every situation, the active method achieves
a more accurate belief (higher values for attentive on the
left, when the true θis attentive, and lower values on the
right, when the true θis distracted). In fact, passive esti-
mation sometimes incorrectly classiﬁes drivers as attentive
when they are distracted and vice-versa.
The same ﬁgure also shows (in solid lines) results from
our user study of what happens when the robot no longer
interacts with an ideal model. We discuss these in the nextsection.
Figures 14and15plot the corresponding robot and human
trajectories for each scenario. The important takeaway from
these ﬁgures is that there tends to be a larger gap betweenattentive and distracted human trajectories in the active con-
dition (orange shades) than in the passive condition (gray
shades), especially in scenarios 2 and 3. It is this differencethat helps the robot better estimate θ:the robot in the active
condition is purposefully choosing actions that will lead to
large differences in human reactions , in order to more easily
determine the human driving style.
7.5 Robot behavior adapts to the situation
As Fig. 14suggests, active info gathering results in interest-
ing coordination behavior. In Scenario 1, the robot decides
to nudge into the person’s lane. But what follows next nicelyreacts to the person’s driving style. The robot proceeds withthe merge if the person is attentive, but actually goes back
to its lane if the person is distracted. Even more interesting
is what happens in Scenario 3 at the 4way stop. The robotinches forward into the intersection, and proceeds if the per-
son is attentive, but actually goes back to allow the person
through if they are distracted! These all emerge as the optimain our system.
The behavior also naturally changes as the initial state
of the system changes. Figure 16shows different behaviors
arising from an attentive driver model but different initial(a) (b) (c)
Fig. 14 Robot trajectories for each scenario in the active information
gathering condition. The robot acts differently when the human is atten-tive (dark orange) versus when the human is distracted (light orange)due to the trade-off with safety
position of the human driver. This shows that even for the
same driver model, the robot intelligently adapts its coordi-
nation behavior to the situation, sometimes deciding to mergebut sometimes not.
This is particularly important, because it might be easy to
handcode these coordination strategies for a particular situa-tion. Much like in the onfﬂine estimation case, the robot not
only comes up with these strategies, but actually adapts them
depending on the situation—the driver style, the initial state,and so on.
7.6 Active information gathering helps the robot’s
actual reward
So far, we have looked at how active information gathering
improves estimation of the driver model. This is useful in
itself in situations where human internal state estimation is
the end-goal. But it is also useful for enabling the robot tobetter achieve its goal.
Intuitively, knowing the human driving style more accu-
rately should improve the robot’s ability to collect reward.
For instance, if the robot starts unsure of whether the humanis paying attention or not, but collects enough evidence that
she is, then the robot can safely merge in front of the per-
son and be more efﬁcient. Of course, this is not always thecase. If the person is distracted, then the information gather-
ing actions could be a waste because the robot ends up not
merging anyway.
Figure 17shows what happens in the merging scenario: the
robot gains more reward compared to passive estimation by
doing information gathering with attentive drivers, because it
ﬁgures out it is safe to merge in front of them; the robot loosessome reward compared to passive estimation with distracted
drivers, because it makes the effort to nudge in but has to
retreat back to its lane anyway because it cannot merge.
Of course, all this depends on choosing λ, the trade-
off between exploitation and exploration (information gain).
Figure 18shows the effect of λhas on the robot’s goal reward
(not its information gain reward), which shows that not all λs
123
Autonomous Robots
(a) (b) (c)
(d) (e) (f)
Fig. 15 The user trajectories for each scenario. The gap between attentive and distracted drivers’ actions is clear in the active information gathering
case (ﬁrst row)
are useful. In other situations, we would also expect to see a
large decrease in reward from too much weight on informa-
tion gain.
7.7 Beyond driving style: active intent inference
Driving style is not the only type of human internal state that
our method enables robots to estimate. If the human has a
goal, e.g. of merging into the next lane or not, or of exitingthe highway or not, the robot could estimate this as well usingthe same technique.
Each possible goal corresponds to a feature. When esti-
mating which goal the human has, the robot is decidingamong θs which place weight on only one of the possi-
ble goal features, and 0 on the others. Figure 19shows the
behavior that emerges from estimating whether the humanwants to merge into the robot’s lane. In the passive case, the
human is side by side with the robot. Depending on the driv-
ing style, they might slow down slightly, accelerate slightly,or start nudging into the robot’s lane, but since the obser-
Fig. 16 Effect of varying the initial condition (relative yposition) in
the active merge scenario. The robot adapts to merge when feasible andavoid otherwise. The human is attentive in all cases
vation model is noisy the robot does not get quite enough
conﬁdence in the human’s intent early on. Depending on the
123
Autonomous Robots
Fig. 17 Active info gathering improves the robot’s ability to efﬁciently
achieve its goal in the case when the human is attentive: where a passiveestimator never gets enough information to know that the person ispaying attention, an active estimator nudges in, updates its belief, andproceeds with the merge. At the same time, active info gathering doesnot hurt too much when the person is distracted: the robot nudges inslightly (this does decrease its reward relative to the passive case, butnot by much), updates its belief, and retreats to its lane
Fig. 18 Active information gathering behavior when the robot’s goal
is to merge into the left lane for different values of λ, together with
the reward the robot obtains. λ=0 results in low reward because the
robot does not ﬁgure out that the person is attentive and does not merge.A small λhurts the reward because the information gathering costs
but does not buy anything. For higher values, the robot gets enoughinformation that it forces a merge in front of the human
robot’s reward, it might take a long time before the person
can merge. In the active case, the robot decides to probe the
person by slowing down and shifting away from the personin order to make room. It then becomes optimal for the per-
(a)
(b)
Fig. 19 Actively estimating the human’s intent (whether they want to
merge in the right lane or not). The robot slows down and shifts slightlyaway from the person, which would make someone who wants to mergeproceed. This could be useful for robots trying to optimize for the goodof all drivers (rather than for their selﬁsh reward function)
son wanting to merge to start shifting towards the robot’s
lane, giving the robot enough information now to update
its belief. In our experiment, we see that this is enough forthe person to be able to complete the merge faster, despite
the robot not having any incentive to help the person in its
reward.
8 Userstudywithonlineestimation
In the previous section, we explored planning for an
autonomous vehicle that actively probes a human’s driv-
ing style, by braking or nudging in and expecting to causereactions from the human driver that would be different
depending on their style. We showed that active exploration
does signiﬁcantly better at distinguishing between atten-tive and distracted drivers using simulated (ideal) models
of drivers. Here, we show the results of a user study that
evaluates this active exploration for attentive and distractedhuman drivers.
123
Autonomous Robots
8.1 Experimental design
We use the same three scenarios discussed in the previous
section.
Manipul atedFactors We manipulated the same two factors
as in our simulation experiments: the reward function that
the robot is optimizing (whether it is optimizing its reward
through passive state estimation, or whether it is trading off
with active information gathering), and the human internal
state (whether the user is attentive or distracted). We asked
our users to pay attention to the road and avoid collisions for
the attentive case, and asked our users to play a game on amobile phone during the distracted driving experiments.
Depen dent Me asure We measured the probability that the
robot assigned along the way to the human internal state.Hypothesis The active condition will lead to more accurate
human internal state estimation, regardless of the true human
internal state.
Subject Alloc ation We recruited 8 participants (2 female, 6
male) in the age range of 21–26 years old. All participants
owned a valid driver license and had at least 2 years of driving
experience. We ran the experiments using a 2D driving simu-lator with the steering input and acceleration input provided
through a steering wheel and a pedals as shown in Fig. 1.
We used a within-subject experiment design with counter-balanced ordering of the four conditions.
8.2 Analysis
We ran a factorial repeated-measures ANOV A on the prob-
ability assigned to “attentive”, using reward (active vs.
passive) and human internal state (attentive vs. distracted)as factors, and time and scenario as covariates. As a manip-ulation check, attentive drivers had signiﬁcantly higher esti-
mated probability of “attentive” associated than distracted
drivers (0.66 vs 0.34, F=3080.3,p<0.0001). More
importantly, there was a signiﬁant interaction effect between
the factors ( F=1444.8,p<0.000). We ran a post-hoc
analysis with Tukey HSD corrections for multiple compar-isons, which showed all four conditions to be signiﬁcantly
different from each other, all contrasts with p<0.0001. In
particular, the active information gathering did end up withhigher probability mass on “attentive” than the passive esti-mation for the attentive users, and lower probability mass
for the distracted user. This supports our hypothesis that our
method works, and active information gathering is better atidentifying the correct state.
Figure 13compares passive (grays and blacks) and active
(light and dark oranges) across scenarios and for attentive(left) and distracted (right) users. It plots the probability of
attentive over time, and the shaded regions correspond to
standard error. From the ﬁrst column, we can see that ouralgorithm in all cases detects human’s attentiveness withmuch higher probably than the passive information gathering
technique shown in black. From the second column, we seethat our algorithm places signiﬁcantly lower probability on
attentiveness, which is correct because those users were dis-
tracted users. These are in line with the statistical analysis,with active information gathering doing a better job estimat-ing the true human internal state.
Figure 14plots the robot trajectories for the active infor-
mation gathering setting. Similar to Fig. 13, the solid lines are
the mean of robot trajectories and the shaded regions show
the standard error. We plot a representative dimension of the
robot trajectory (like position or speed) for attentive (darkorange) or distracted (light orange) cases. The active robot
probed the user, but ended up taking different actions when
the user was attentive versus distracted in order to maintainsafety. For example, in Scenario 1, the trajectories show therobot is nudging into the human’s lane, but the robot decides
to move back to its own lane when the human drivers are dis-
tracted (light orange) in order to stay safe. In Scenario 2, therobot brakes in front of the human, but it brakes less when the
human is distracted. Finally, in Scenario 3, the robot inches
forward, but again it stops when if the human is distracted,and even backs up to make space for her.
Figure 15plots the user trajectories for both active
information gathering (ﬁrst row) and passive informationgathering (second row) conditions. We compare the reactionsof distracted (light shades) and attentive (dark shades) users.
There are large differences directly observable, with user
reactions tending to indeed cluster according to their internalstate. These differences are much smaller in the passive case
(second row, where distracted is light gray and attentive is
black). For example, in Scenario 1 and 2, the attentive users(dark orange) keep a larger distance to the car that nudges
in front of them or brakes in front of them, while the dis-
tracted drivers (light orange) tend to keep a smaller distance.In Scenario 3, the attentive drivers tend to slow down anddo not cross the intersection, when the robot actively inches
forward. None of these behaviors can be detected clearly in
the passive information gathering case (second row). Thisis the core advantage of active information gathering: the
actions are purposefully selected by the robot such that users
would behave drastically differently depending on their inter-nal state, clarifying to the robot what this state actually is.
Over all, these results support our simulation ﬁndings, that
our algorithm performs better at estimating the true humaninternal state by leveraging purposeful information gatheringactions.
9 Discussion
Summ aryIn this paper, we took a step towards autonomously
producing behavior for interaction and coordination between
123
Autonomous Robots
autonomous cars and human-driven vehicles. We formulated
a dynamical system in which the robot accounts for howits actions are going to inﬂuence those of the human as a
simpliﬁcation to a partially observable stochastic game. We
introduced approximations for optimizing in the dynami-cal system that bring the robot’s computation close to realtime (.3 s/time step). We showed in an empirical analysis
that when the robot estimates the human model ofﬂine, it
produces behavior that can purposefully modify the humanbehavior: merging in front of them to get them to slow down,
or pulling back at an intersection to incentivize them to pro-
ceed ﬁrst through. We also showed that these behaviors canemerge out of directly optimizing for the robot’s efﬁciency.
We further introduced an online estimation algorithm in
which the robot actively uses its actions to gather informa-tion about the human model so that it can better plan itsown actions. Our analysis again shows coordination strate-
gies arising out of planning in our formulation: the robot
nudges into someones’s lane to check if the human is payingattention, and only completes the merge if they are; the robot
inches forward at an intersection, again to check if the human
is paying attention, and proceeds if they are, but backs up tolet them through if they are not; the robot slows down slightly
and shifts in its lane away from the human driver to check if
they want to merge into its lane or not.
Importantly, these behaviors change with the human driver
style and with the initial conditions—the robot takes different
actions in different situations, emphasizing the need to start
generating such coordination behavior autonomously ratherthan relying on hand coded strategies. Even more impor-
tantly, the behaviors seem to work when the robot is planning
and interacting with real users.
Limit ations All this work happened in a simple driving sim-
ulator. To put this on the road, we will need more emphasis
on safety, as well as a longer planning horizon.
While performing these experiments, we found the robot’s
nominal reward function (trading off between safety and
reaching a goal) to be insufﬁcient—in some cases it led to
getting dangerously close to the human vehicle and even col-lisions, going off the road, oscillating in the lane due to minorasymmetries in the environment, etc.
Figure 20shows an example of such behavior that comes
from the 4way stop domain. For the most part, the car plansto back up to incentivize the human to go through ﬁrst. But
for some values of the human’s initial velocity, we observed
bad behavior, likely due to convergence to local maxima: thecar did not ﬁgure out to slow down or back up, and instead
if proceeded forward—then it tried to avoid collisions with
the person and went off the road, and in the wrong direction(i.e. in the person’s way).
It seems like while a reward function might be a good
enough model for the human, it might be difﬁcult to devise
Fig. 20 Example of bad local optima occurring for certain initial veloc-
ities of the human in the 4way intersection scenario
such a universal function for the robot, and the use of hard
constraints to ensure safe control would be welcome.
Another limitation is that we currently focus on a sin-
gle human driver. Looking to the interaction among multiplevehicles is not just a computational challenge, but also a mod-
eling one—it is not immediately clear how to formulate the
problem when multiple human-driven vehicles are interact-ing and reacting to each other.Conclusion Despite these limitations, we are encouraged to
see autonomous cars generate human-interpretable behaviors
through optimization, without relying on hand-coded heuris-tics. Even though in this work we have focused on modeling
the interaction between an autonomous car and a human-
driven car, the same framework of underactuated systems
can be applied to modeling the interaction between humans
and robots in more general settings. We look forward to appli-
cations of these ideas beyond autonomous driving, to mobilerobots, UA Vs, and in general to human–robot interactive sce-narios where robot actions can inﬂuence human actions.
Acknowledgements This work was partially supported by Berkeley
DeepDrive, NSF VeHICaL 1545126, NSF Grants CCF-1139138 andCCF-1116993, ONR N00014-09-1-0230, NSF CAREER 1652083, andan NDSEG Fellowship.
References
Abbeel, P., & Ng, A. Y . (2005). Exploration and apprenticeship learning
in reinforcement learning. In Proceedings of the 22nd international
conference on Machine learning (pp. 1–8). ACM.
Agha-Mohammadi, A.-A., Chakravorty, S., & Amato, N. M. (2014).
FIRM: Sampling-based feedback motion-planning under motionuncertainty and imperfect measurements. The International Jour-
nal of Robotics Research ,33(2), 268–304.
Andrew, G.,& Gao, J. (2007). Scalable training of L1-regularized log-
linear models. In Proceedings of the 24th international conference
on Machine learning (pp. 33–40). ACM.
123
Autonomous Robots
Atanasov, N. A. (2015). Active information acquisition with mobile
robots.
Atanasov, N., Ny Le J., Daniilidis, K. & Pappas, G. J. (2014). Informa-
tion acquisition with sensing robots: Algorithms and error bounds.In2014 IEEE international conference on robotics and automation
(ICRA) (pp. 6447–6454). IEEE.
Aumann, R. J., Maschler, M., & Stearns, R. E. (1995). Repeated games
with incomplete information . Cambridge: MIT Press.
Bandyopadhyay, T., Won, K. S., Frazzoli, E., Hsu, D., Lee, W. S., &
Rus, D.(2013). Intention-aware motion planning. In Algorithmic
foundations of robotics X (pp. 475–491). Springer.
Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I. J.,
Bergeron, A., Bouchard, N., & Bengio, Y . (2012). Theano: newfeatures and speed improvements. In Deep learning and unsuper-
vised feature learning NIPS 2012 workshop .
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Des-
jardins, G., Turian, J., Warde-Farley, D., & Bengio, Y . (2010).Theano: a CPU and GPU math expression compiler. In Proceed-
ings of the python for scientiﬁc computing conference (SciPy) , Oral
Presentation.
Bernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002). The
complexity of decentralized control of Markov decision processes.
Mathematics of Operations Research ,27(4), 819–840.
Camacho, E. F., & Alba, C. B. (2013). Model predictive control .B e r l i n :
Springer.
Chaudhari, P., Karaman, S., Hsu, D., & Frazzoli, E. (2013). Sampling-
based algorithms for continuous-time POMDPs. In American
control conference (ACC), 2013 (pp. 4604–4610). IEEE.
Dissanayake, M., Newman, P., Clark, S., Durrant-Whyte, H. F., &
Csorba, M. (2001). A solution to the simultaneous localization andmap building (SLAM) problem. IEEE Transactions on Robotics
and Automation ,17(3), 229–241.
Falcone, P., Borrelli, F., Asgari, J., Tseng, H. E., & Hrovat, D. (2007).
Predictive active steering control for autonomous vehicle systems.IEEE Transactions on Control Systems Technology ,15(3), 566–
580.
Falcone, P., Borrelli, F., Tseng, H. E., Asgari, J., & Hrovat, D. (2007).
Integrated braking and steering model predictive control approachin autonomous vehicles. Advances in Automotive Control ,5, 273–
278.
Falcone, P., Tseng, H. E., Borrelli, F., Asgari, J., & Hrovat, D. (2008).
MPC-based yaw and lateral stabilisation via active front steeringand braking. Vehicle System Dynamics ,46(sup1), 611–628.
Fern, A., Natarajan, S., Judah, K., & Tadepalli, P. (2007). A decision-
theoretic model of assistance .I nIJCAI .
Fudenberg, D., & Tirole, J. (1991). Game theory (V ol. 393). Cambridge,
Massachusetts.
Gray, A., Gao, Y ., Hedrick, J. K. & Borrelli, F.(2013). Robust pre-
dictive control for semi-autonomous vehicles with an uncertaindriver model. In Intelligent vehicles symposium (IV), 2013 IEEE
(pp. 208–213). IEEE.
Hansen, E. A., Bernstein, D. S., & Zilberstein, S. (2004). Dynamic
programming for partially observable stochastic games. AAAI ,4,
709–715.
Hedden, T., & Zhang, J. (2002). What do you think i think you think?:
Strategic reasoning in matrix games. Cognition ,85(1), 1–36.
Hermes, C., Wohler, C., Schenk, K., & Kummert, F. (2009). Long-
term vehicle motion prediction. In 2009 IEEE intelligent vehicles
symposium (pp. 652–657).
Javdani, S., Bagnell, J. A., & Srinivasa, S. (2015). Shared autonomy via
hindsight optimization. arXiv preprint arXiv:1503.07619 .
Javdani, S., Klingensmith, M., Bagnell, J. A., Pollard, N. S., & Srinivasa,
S. S.(2013). Efﬁcient touch based localization through submod-ularity. In 2013 IEEE international conference on robotics and
automation (ICRA) (pp. 1828–1835). IEEE.Kuderer, M., Gulati, S., & Burgard, W. (2015). Learning driving styles
for autonomous vehicles from demonstration. In Proceedings of
the IEEE international conference on robotics & automation(ICRA), Seattle, USA , V ol. 134.
Lam, C.-P., Yang, A. Y ., & Sastry, S. S.(2015). An efﬁcient algorithm for
discrete-time hidden mode stochastic hybrid systems. In Control
conference (ECC), 2015 European . IEEE.
Leonard, J., How, J., Teller, S., Berger, M., Campbell, S., Fiore, G., et al.
(2008). A perception-driven autonomous urban vehicle. Journal of
Field Robotics ,25(10), 727–774.
Levine, S, & Koltun, V .(2012). Continuous inverse optimal control with
locally optimal examples. arXiv preprint arXiv:1206.4617 .
Levinson, J., Askeland, J., Becker, J., Dolson, J., Held, D., Kammel, S.,
Kolter, J. Z., Langer, D., Pink, O., Pratt, V ., et al. (2011). Towardsfully autonomous driving: Systems and algorithms. In 2011 IEEE
Intelligent Vehicles Symposium (IV) , pages 163–168.
Luders, B., Kothari, M., & How, J. P.(2010). Chance constrained RRT
for probabilistic robustness to environmental uncertainty. In AIAA
guidance, navigation, and control conference (GNC) , Toronto,
Canada.
Ng, A. Y ., Russell, S. J., et al.(2000). Algorithms for inverse reinforce-
ment learning. In Proceedings of the 17th international conference
on Machine learning , pages 663–670.
Nikolaidis, S., Kuznetsov, A., Hsu, D., & Srinivasa, S. (2016). For-
malizing human-robot mutual adaptation via a bounded memorybased model. In Human-robot interaction .
Nikolaidis, S., Ramakrishnan, R., Gu, K., & Shah, J. (2015). Efﬁcient
model learning from joint-action demonstrations for human-robotcollaborative tasks. In Proceedings of the tenth annual ACM/IEEE
international conference on human-robot interaction (pp. 189–
196). ACM.
Patil, S., Kahn, G., Laskey, M., Schulman, J., Goldberg, K., & Abbeel,
P. (2015). Scaling up Gaussian belief space planning throughcovariance-free trajectory optimization and automatic differenti-ation. In Algorithmic foundations of robotics XI (pp. 515–533).
Springer.
Prentice, S., & Roy, N. (2009). The belief roadmap: Efﬁcient planning in
belief space by factoring the covariance. The International Journal
of Robotics Research ,28, 1448–1465.
Raman, V ., Donzé, A., Sadigh, D., Murray, R. M., & Seshia, S. A.
(2015). Reactive synthesis from signal temporal logic speciﬁ-cations. In Proceedings of the 18th international conference on
hybrid systems: Computation and control (pp. 239–248). ACM.
Sadigh, D., & Kapoor, A. (2015). Safe control under uncertainty. arXiv
preprint arXiv:1510.07313 .
Sadigh, D., Sastry, S. A., Seshia, S., & Dragan, A. D. (2016a). Plan-
ning for autonomous cars that leverages effects on human actions.
InProceedings of the robotics: Science and systems conference
(RSS) .
Sadigh, D., Sastry, S. S., Seshia, S. A., & Dragan, A. (2016b). Infor-
mation gathering actions over human internal state. In IEEE/RSJ
international conference on intelligent robots and systems (IROS)(pp. 66–73). IEEE.
Sadigh, D., Sastry, S. S., Seshia, S. A., & Dragan, A. (2016c). Plan-
ning for autonomous cars that leverages effects on human actions.InProceedings of the robotics: Science and systems conference
(RSS) .
Seiler, K. M., Kurniawati, H., & Singh, S. P. (2015). An online and
approximate solver for POMDPs with continuous action space. In2015 IEEE international conference on robotics and automation(ICRA) (pp. 2290–2297). IEEE.
Shimosaka, M., Kaneko, T., & Nishi, K. (2014). Modeling risk antic-
ipation and defensive driving on residential roads with inversereinforcement learning. In 2014 IEEE 17th international confer-
ence on intelligent transportation systems (ITSC) (pp. 1694–1700).
IEEE.
123
Autonomous Robots
Trautman, P. (2013). Robot navigation in dense crowds: Statistical
models and experimental studies of human robot cooperation .
Pasadena: California Institute of Technology.
Trautman, P., & Krause, A. (2010). Unfreezing the robot: Navigation
in dense, interacting crowds. In 2010 IEEE/RSJ international con-
ference on intelligent robots and systems (IROS) (pp. 797–803).
Trautman, P., Ma, J., Murray, R. M., & Krause, A. (2013). Robot nav-
igation in dense human crowds: the case for cooperation. In 2013
IEEE international conference on robotics and automation (ICRA)(pp. 2153–2160). IEEE.
Urmson, C., Anhalt, J., Bagnell, D., Baker, C., Bittner, R., Clark, M.,
et al. (2008). Autonomous driving in urban environments: Boss andthe urban challenge. Journal of Field Robotics ,25(8), 425–466.
Vanchinathan, H. P., Nikolic, I., Bona, F. De., & Krause, A. (2014).
Explore-exploit in top-n recommender systems via Gaussian pro-cesses. In Proceedings of the 8th ACM conference on recommender
systems (pp. 225–232). ACM.
Vasudevan, R., Shia, V ., Gao, Y ., Cervera-Navarro, R., Bajcsy, R., &
Borrelli, F. (2012). Safe semi-autonomous control with enhanceddriver modeling. In American control conference (ACC) (pp. 2896–
2903). IEEE.
Vitus, M. P. & Tomlin, C. J. (2013). A probabilistic approach to plan-
ning and control in autonomous urban driving. In 2013 IEEE 52nd
annual conference on decision and control (CDC) (pp. 2459–
2464).
Ziebart, B. D. (2010). Modeling purposeful adaptive behavior with the
principle of maximum causal entropy.
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2008)
Maximum entropy inverse reinforcement learning. In AAAI (pp.
1433–1438).
Dorsa Sadigh is an assistant pro-
fessor in Computer Science andElectrical Engineering at StanfordUniversity. Her research interestslie in the intersection of robotics,control theory, formal methods,and human-robot interaction.Speciﬁcally, she works on devel-oping efﬁcient algorithms for safeand interactive human-robot sys-tems such as semiautonomousdriving. She has received her doc-toral degree in Electrical Engi-neering and Computer Sciences(EECS) at UC Berkeley in 2017,
and has received her bachelor’s degree in EECS at UC Berkeley in2012. She is awarded the NSF and NDSEG graduate research fellow-ships as well as Leon O. Chua departmental award, Arthur M. Hopkindepartmental award, and the Google Anita Borg Scholarship.
Nick Landolﬁ is an undergraduate
studying Electrical Engineering &Computer Science and Statisticsat UC Berkeley. He works withProf. Anca Dragan in the Inter-ACT Lab, where his researchfocuses on interactive autonomyin robotics. He is interested inmodeling and planning in multi-agent scenarios, incorporatingaspects of machine learning,stochastic control and optimiza-tion. He was awarded the Regents’and Chancellor’s Scholarship andCal Leadership Award in 2014.
S h a n k a rS .S a s t r y received his
Ph.D. degree in 1981 from theUniversity of California, Berke-ley. He was on the faculty of MITas Assistant Professor from 1980to 82 and Harvard University asa chaired Gordon Mc Kay profes-sor in 1994. He is currently theDean of Engineering at Universityof California, Berkeley. His areasof personal research are embed-ded control especially for wirelesssystems, cybersecurity for embed-ded systems, critical infrastruc-ture protection, autonomous soft-
ware for unmanned systems (especially aerial vehicles), computervision, nonlinear and adaptive control, control of hybrid and embed-ded systems, and network embedded systems and software. He hassupervised over 60 doctoral students and over 50 M.S. students tocompletion. His students now occupy leadership roles in several places
and on the faculties of many major universities. He has coauthored
over 450 technical papers and 9 books. Dr. Sastry served on the edito-rial board of numerous journals, and is currently an Associate Editorof the IEEE Proceedings.
Sanjit A. Seshia received the
B.Tech. degree in Computer Sci-ence and Engineering from theIndian Institute of Technology,Bombay in 1998, and the M.S.and Ph.D. degrees in ComputerScience from Carnegie MellonUniversity in 2000 and 2005 respe-ctively. He is currently a Professorin the Department of ElectricalEngineering and Computer Sci-ences at the University of Califor-nia, Berkeley. His research inter-ests are in dependable computingand computational logic, with a
current focus on applying automated formal methods to embedded
and cyber-physical systems, electronic design automation, computersecurity, and synthetic biology. His awards and honors include a Pres-idential Early Career Award for Scientists and Engineers (PECASE)from the White House, an Alfred P. Sloan Research Fellowship, theProf. R. Narasimhan Lecture Award, and the School of Computer Sci-ence Distinguished Dissertation Award at Carnegie Mellon University.He is a Fellow of the IEEE.
123
Autonomous Robots
Anca D. Dragan is an Assistant
Professor in UC Berkeley’s EECSDepartment. She completed herPh.D. in Robotics at Carnegie Mel-lon. She was born in Romaniaand received her B.Sc. in Com-puter Science from Jacobs Uni-versity in Germany in 2009. Herresearch lies at the intersectionof robotics, machine learning, andhuman–computer interaction: sheworks on algorithms that enablerobots to seamlessly work with,around, and in support of people.Her’s research and her outreach
activities with children have been recognized by the Intel Fellowshipand by scholarships from Siebel, the Dan David Prize, and GoogleAnita Borg. She has been honored by the Sloan Fellowship, MITTR35, the Okawa award, and an NSF CAREER award.
123

Dorsa Sadigh
Nick Landolfi
Shankar S. Sastry
Sanjit A. Seshia
Anca D. Dragan