AvE: Assistance via Empowerment.

AvE: Assistance via Empowerment
Yuqing Du
UC Berkeley
yuqing_du@berkeley.eduStas Tiomkin
UC Berkeley
stas@berkeley.eduEmre Kıcıman
Microsoft Research
emrek@microsoft.com
Daniel Polani
University of Hertfordshire
d.polani@herts.ac.ukPieter Abbeel
UC Berkeley
pabbeel@berkeley.eduAnca Dragan
UC Berkeley
anca@berkeley.edu
Abstract
One difﬁculty in using artiﬁcial agents for human-assistive applications lies in
the challenge of accurately assisting with a person’s goal(s). Existing methods
tend to rely on inferring the human’s goal, which is challenging when there are
many potential goals or when the set of candidate goals is difﬁcult to identify.
We propose a new paradigm for assistance by instead increasing the human’s
ability to control their environment, and formalize this approach by augmenting
reinforcement learning with human empowerment . This task-agnostic objective
preserves the person’s autonomy and ability to achieve any eventual state. We test
our approach against assistance based on goal inference, highlighting scenarios
where our method overcomes failure modes stemming from goal ambiguity or
misspeciﬁcation. As existing methods for estimating empowerment in continuous
domains are computationally hard, precluding its use in real time learned assistance,
we also propose an efﬁcient empowerment-inspired proxy metric. Using this, we
are able to successfully demonstrate our method in a shared autonomy user study
for a challenging simulated teleoperation task with human-in-the-loop training.
1 Introduction
We aim to enable artiﬁcial agents, whether physical or virtual, to assist humans in a broad array of
tasks. However, training an agent to provide assistance is challenging when the human’s goal is
unknown because that makes it unclear what the agent should do. Assistance games [ 18] formally
capture this as the problem of working together with a human to maximize a common reward function
whose parameters are only known to the human and not to the agent. Naturally, approaches to
assistance in both shared workspace [ 32,14,13,33,31,25] and shared autonomy [ 21,20,11,15,32]
settings have focused on inferring the human’s goal (or, more broadly, the hidden reward parameters)
from their ongoing actions, building on tools from Bayesian inference [ 5] and Inverse Reinforcement
Learning [ 30,37,2,4,45,19,34,12,26]. However, goal inference can fail when the human model
is misspeciﬁed, e.g. because people are not acting noisy-rationally [ 27,36], or because the set of
candidate goals the agent is considering is incorrect [ 7]. In such cases, the agent can infer an incorrect
goal, causing its assistance (along with the human’s success) to suffer, as it does in Figure 1.
Even in scenarios where the agent correctly infers the human’s goal, we encounter further questions
about the nature of collaborations between humans and assistive agents: what roles do each of them
play in achieving the shared goal? One can imagine a scenario where a human is attempting to
traverse down a hallway blocked by heavy objects. Here, there are a range of assistive behaviours: a
robot could move the objects and create a path so the human is still the main actor, or a robot could
physically carry the person down the hallway, making the human passive. Depending on the context,
either solution may be more or less appropriate. Speciﬁcally, the boundary between assisting humans
34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.
Figure 1: Toy scenario where a robot operates multiple doors. On the left, the robot attempts to
infer the human’s intended goal, C, but mistakenly infers B. On the right, the robot assesses that the
human’s empowerment would increase with doors BandCopen. Naively opening all doors will not
increase empowerment as Ais too small for the person and Dleads to the same location as C.
in tackling challenging tasks and solving these tasks in the place of humans is not clearly deﬁned.
As AI improves at ‘human’ jobs, it is crucial to consider how the technology can complement and
amplify human abilities, rather than replace them [43, 42].
Our key insight is that agents can assist humans without inferring their goals or limiting their autonomy
by instead increasing the human’s controllability of their environment – in other words, their ability
to affect the environment through actions . We capture this via empowerment , an information-theoretic
quantity that is a measure of the controllability of a state through calculating the logarithm of the
number of possible distinguishable future states that are reachable from the initial state [ 39]. In
our method, Assistance via Empowerment (AvE), we formalize the learning of assistive agents as
an augmentation of reinforcement learning with a measure of human empowerment. The intuition
behind our method is that by prioritizing agent actions that increase the human’s empowerment, we
are enabling the human to more easily reach whichever goal they want. Thus, we are assisting the
human without information about their goal – the agent does not carry the human to the goal, but
instead clears a path so they can get there on their own. Without any information or prior assumptions
about the human’s goals or intentions, our agents can still learn to assist humans.
We test our insight across different environments by investigating whether having the agent’s be-
haviour take into account human empowerment during learning will lead to agents that are able to
assist humans in reaching their goal, despite having no information about what the goal truly is. Our
proposed method is the ﬁrst one, to our knowledge, that successfully uses the concept behind empow-
erment with real human users in a human-agent assistance scenario. Our experiments suggest that
while goal inference is preferable when the goal set is correctly speciﬁed and small, empowerment can
signiﬁcantly increase the human’s success rate when the goal set is large or misspeciﬁed. This does
come at some penalty in terms of how quickly the human reaches their goal when successful, pointing
to an interesting future work direction in hybrid methods that try to get the best of both worlds. As
existing methods for computing empowerment are computationally intensive, we also propose an
efﬁcient empowerment-inspired proxy metric that avoids the challenges of computing empowerment
while preserving the intuition behind its usefulness in assistance. We demonstrate the success of this
algorithm and our method in a user study on shared autonomy for controlling a simulated dynamical
system. We ﬁnd that the strategy of stabilizing the system naturally emerges out of our method, which
in turn leads to higher user success rate. While our method cannot outperform an oracle method that
has knowledge of the human’s goal, we ﬁnd that increasing human empowerment provides a novel
step towards generalized assistance, including in situations where the human’s goals cannot be easily
inferred.
Main contributions:
We formalize learning for human-agent assistance via the empowerment method.
We directly compare our method against a goal inference approach and conﬁrm where our
method is able to overcome potential pitfalls of inaccurate goal inference in assistance.
We propose a computationally efﬁcient proxy for empowerment in continuous domains,
enabling human-in-the-loop experiments of learned assistance in a challenging simulated
teleoperation task.
2
2 Empowerment Preliminary
Background. To estimate the effectiveness of actions on the environment, [ 23] proposed computing
thechannel capacity between an action sequence, ~AT:= (A1A2;:::;A T), and the ﬁnal states, ST,
where the channel is represented by the environment. Formally, empowerment of a state sis:
E(s) =maximum
p(~ATjs)I[~AT;STjs]; (1)
whereI[~AT;STjs]:=H(STjs)H(STj~AT;s)is the mutual information between ~ATand the
ST,H()is entropy, and Tis the time horizon, which is a hyperparameter. The probing probability
distribution p(~ATjs)is only used for the computation of the empowerment/channel capacity E(s),
and never generates the behavior directly. Note that the actions are applied in an open-loop fashion,
without feedback.
In the context of learning, empowerment as an information-theoretic quantity has mainly been seen
as a method for producing intrinsic motivation [ 39,29,24,10]. [17] showed that a composition of
human empowerment, agent empowerment, and agent-to-human empowerment are useful for games
where a main player is supported by an artiﬁcial agent. An alternative view of this compositional
approach was proposed in [ 40], where the agent-to-human empowerment was replaced by human-to-
agent empowerment. The latter is a human-centric approach, where only the humans’ actions affect
the agents’ states in a way which is beneﬁcial for the human. These approaches have conceptually
discussed the applicability of empowerment to human-agent collaborative settings and building on
this, our paper concretely proposes that assistance in shared workspaces and shared autonomy tasks
can be cast as an empowerment problem, and evaluates that idea with real users.
Given the challenge of computing empowerment, some existing estimation methods are:
Tabular case approximations. In tabular cases with a given channel probability, p(STj~AT;s), the
problem in (1)is solved by the Blahut-Arimoto algorithm [ 6]. This method does not scale to: a) high
dimensional state and/or action space, and b) long time horizons. An approximation for a) and b)
can be done by Monte Carlo simulation [ 22], however, the computational complexity precludes user
studies of empowerment-based methods for assistance in an arbitrary state/action space.
Variational approximations. Previous work has proposed a method for using variational approx-
imation to provide a lower bound on empowerment [ 29]. This was extended later to closed-loop
empowerment [ 16]. Both of these methods estimate empowerment in a model-free setting. Recent
work has also proposed estimating empowerment by the latent space water-ﬁlling algorithm [ 44], or
by applying bi-orthogonal decomposition [ 41], which assumes known system dynamics. However,
these estimation methods are computationally hard, which precludes their use in reinforcement
learning, especially, when a system involves learning with humans, as in our work.
3 Assistance via Empowerment
3.1 Problem Setting
We formulate the human-assistance problem as a reinforcement learning problem, where we model
assistance as a Markov Decision Process (MDP) deﬁned by the tuple (S;Aa;P;R;
 ).Sconsists
of the human and agent states, (Sh;Sa),Aais the set of agent actions, Pis an unknown transition
function,Ris a reward function, and
is a discount factor. We assume the human has an internal
policyhto try to achieve a goal g2Shthat is unknown to the agent. Note that the human may
not behave completely rationally and hmay not enable them to achieve g, requiring assistance.
As the agent does not know the human policy, we capture state changes from human actions in the
environment transition function (st+1
h;st+1
a) =P((st
h;st
a);aa).
Our method uses reinforcement learning to learn an assistive policy athat maximizes the expected
sum of discounted rewards E[P
t
tR(st
a;st
h)]. For assistance, we propose the augmented reward
function
R(sa;sh) =Roriginal (sa;sh) +cempEhuman (sh)
whereRoriginal captures any general parts of the reward that are non-speciﬁc to assisting with
the human’s goal, such as a robot’s need to avoid collisions, and where Ehuman (sh)is the agent’s
3
θ∈ [− π,π]in rad/uni0307θ∈ [−6, ̇ 6] in rad /sec
θ∈ [− π,π]in rad/uni0307θ∈ [−6, ̇ 6] in rad /secFigure 2: Comparison between
our proxy for empowerment (left)
and the known landscape (right)
of the non-linear pendulum over
a long time horizon. The proxy
landscape captures the essential
properties of the empowerment
landscape: maximum at the up-
right position and comparatively
low values for states with energy
below the separatrix energy.
estimation of the human’s empowerment in their current state, sh. In simple tabular environments
we can directly compute empowerment, however, this does not extend to more realistic assistance
scenarios. Thus for our user study, we propose an empowerment-inspired proxy as detailed in Section
3.2. We access the human’s action and state space either through observation in the shared workspace
case or directly in shared autonomy. The coefﬁcient cemp0is included to balance the weighting
between the original reward function and the empowerment term.
3.2 Approach for Human-in-the-Loop Training
As our work is intended for real-time human assistance applications, we strongly prioritize computa-
tional efﬁciency over the numerical accuracy of the empowerment estimation in our user study. To
compute true empowerment, one considers potential forward-simulations of agent actions of duration
Tstarting insinitand their resulting effect. However, since existing methods for approximating
empowerment in continuous domains from empirical data are computationally quite hard and do not
scale well, here we instead draw on the intuition of empowerment to propose a proxy metric using a
measure of diversity of ﬁnal states as a surrogate for the channel capacity.
Namely, we use a fast and simple analogy of the sparse sampling approximation from [ 38] for the
empowerment method. In that work, the number of discrete states visited by the forward-simulations
was counted. A large number of different ﬁnal states corresponded to an initial state with high
empowerment. In the continuum, counting distinct states becomes meaningless. Therefore, here,
we measure diversity of the ﬁnal state by computing the variance of the ﬂattened sample vectors of
Sf, as summarized in Algorithm 1, and use this pragmatic approach directly in place of Ein Eq. 1.
While this proxy relies on an assumption of homogeneous noise and high SNR, it can be computed
much more efﬁciently than empowerment, lending it to be directly applicable to human-in-the-loop
training of assistive agents. Even as a coarse proxy for empowerment, we ﬁnd that it is sufﬁcient to
signiﬁcantly increase human success in our user study in Section 4.2. To empirically motivate our
proxy, we compare the known empowerment landscape of a non-linear pendulum with our proxy
result in Figure 2. A rigourous study of the properties of the proxy is deterred to the future work.
Algorithm 1: Empowerment-inspired Diversity Bonus
initialize environment at state sinit;
initialize empty list of ﬁnal states Sf;
// rollout Ntrajectories of horizon T
forn = 1 ... N dos sinit
fort = 1 ... T do
// randomly generate actions and update state
a= sample action;
s P(s;a)
end
Sf Sf+ [s]
end
// To compute scalar reward bonus
return Var( ﬂatten (Sf))
4
4 Experiments
In this section we evaluate our method in two distinct assistive tasks where a human attempts to
achieve a goal: ﬁrstly, a shared workspace task where the human and assistant are independent
agents, and secondly, a shared autonomy control task. The ﬁrst task is used to directly compare the
empowerment method in a tabular case against a goal inference baseline to motivate our method,
and the second task is used to test both our proxy metric in Algorithm 1and evaluate our overall
method with real users in a challenging simulated teleoperation task. For accompanying code, see
https://github.com/yuqingd/ave .
4.1 Shared Workspace – Gridworld
Experiment Setup. To motivate our method as an alternative when goal inference may fail, we
constructed a gridworld simulation as follows: a simulated proxy human attempts to move greedily
towards a goal space. The grid contains blocks that are too heavy for them to move, however, an
agent can assist by moving these blocks to any adjacent open space. The agent observes the location
of the blocks and the humans but not the location of the goal. As a metric for successful assistance,
we measure the success rate of the human reaching the goal and the average number of steps taken.
Figure 3: Sample rollouts with proxy human , immovable blocks , and goal at the star. Each frame
consists of an action taken by the agent. Left column shows two cases of goal inference: ideal case
(top) and failure mode with misspeciﬁed case (bottom) where the goal is blocked.
We compare against a goal inference method where the agent maintains a probability distribution
over a candidate goal set and updates the likelihood of each goal based on the human’s action at each
step. We test variations of goal inference that highlight potential causes of failure: 1) when the goal
set is too large, and 2) when the true goal is not in the goal set. Here, we deﬁne failure as the human’s
inability to achieve the goal before the experiment times out at 1000 steps. In this discrete scenario,
we can compute empowerment directly by sampling 1000 trajectories and computing the logarithm
of the number of distinct states the blocks and the human end up in. For comparison, we also provide
results where we approximate empowerment with our proxy. The agent assumes that the human can
move into any adjacent free space but does not know the human’s policy or goal. We simulate a
variety of initializations of the human’s position, the number of blocks and their positions, and the
goal position. Here we highlight the results of two main scenarios in Table 1: where the blocks trap
the human in a corner, and where the blocks trap the human in the center of the grid. This is because
the center of the grid has highest empowerment and the corner of the grid has lowest empowerment,
and situations where the human is trapped are where assistance is most crucial. We ran 100 trials of
randomized goal initializations and computed the number of steps the human takes to get to their
goal (if successful) under each of the reward formulations.
Analysis. Our results suggest that in the case where we have a small and correctly speciﬁed goal
set, goal inference is the best strategy to use as it consistently succeeds and takes fewer steps to
reach the goal on average. However, this represents an ideal case. When we have a misspeciﬁed goal
set and/or a larger goal set, the success rate drops signiﬁcantly. The failure modes occur when the
agent is mistaken or uncertain about the goal location, and chooses to move a block on top of the
true goal. As the human can no longer access the goal, they wait in an adjacent block and cannot
act further to inform the agent about the true goal location, leading to an inﬁnite loop. Since our
empowerment method does not maintain a goal set, it does not encounter this issue and is able to
succeed consistently, albeit with a tradeoff in higher mean steps performance. Interestingly, we note
that the failure cases that the goal inference method encounters could easily be resolved by switching
to the empowerment method – that is, since the human is next to a block that is on top of the goal,
5
Table 1: Success rates and mean steps to goal (after removing failed trials) for human in corner (top)
and human in center (bottom) scenarios. For human in corner, the human is randomly initialized in
one of the four corners of the grid with two blocks trapping them. For human in center, the human is
initialized at the center of the grid with four blocks surrounding them, one on each side. GI – goal
inference with either the goal in the goal set (known) goal or the goal missing from the goal set
(unknown). Large Goal (LG) Set considers every possible space as a goal, Small Goal (SG) Set only
considered two possible goals, and No Goal (NG) is our method.
Human in Corner | Success %, (Mean Steps) LG Set SG Set NG
GI (known) 90%, (4.35) 100% , (4.35)
GI (unknown) 90%, (4.35) 92%, (4.51)
Empowerment 100% , (5.24)
Empowerment Proxy 99% , (8.79)
Oracle 100%, (4.1)
Human in Center | Success %, (Mean Steps) LG Set SG Set NG
GI (known) 50%, (2.27) 100% , (2.57)
GI (unknown) 50%, (2.27) 55%, (3.3)
Empowerment 100% , (4.8)
Empowerment Proxy 100% , (6.71)
Oracle 100%, (1.91)
increasing human empowerment would move the block away and allow the human to access it. This
highlights a potential for future work in hybrid assistance methods. In assistance scenarios where
accurate goal inference may be too challenging to complete or the risks of assisting with an incorrect
goal are too high, our goal-agnostic method provides assistance while circumventing potential issues
with explicitly inferring human goals or rewards. As motivation for our proxy, we note that the proxy
increases the mean steps to goal and can lead to a 1% decrease in success rate. This is because the
inaccurate measure of empowerment can occasionally lead to blocking the goal, as in the non-ideal
goal inferences cases. However, the proxy success rate is still much higher than that of goal inference.
4.2 Shared Autonomy – Lunar Lander
Our previous experiments motivate the beneﬁts of a goal-agnostic human assistance method using
human empowerment. To evaluate our method with real users, we ran a user study in the shared
autonomy domain. The purpose of this experiment is two-fold: we demonstrate that a simpliﬁed
empowerment-inspired proxy metric is sufﬁcient for assisting the human while avoiding the compu-
tational demands of computing empowerment, and we demonstrate the efﬁcacy of our method by
assisting real humans in a challenging goal-oriented task without inference. For clarity, in this section
we use ‘empowerment’ to denote the proxy deﬁned in Section 3.2.
We build on recent work in this area by [ 35] which proposed a human-in-the-loop model-free deep
reinforcement learning (RL) method for tackling shared autonomy without assumptions of known
dynamics, observation model, or set of user goals. Their method trains a policy (via Q-learning)
that maps state and the user control input to an action. To maintain the user’s ability to control the
system, the policy only changes the user input when it falls below the optimal Q-value action by
some margin . In this case, RL will implicitly attempt to infer the goal from the human input in
order to perform better at the task, but this is incredibly challenging. With our method, empowerment
is a way to explicitly encourage the shared autonomy system to be more controllable, allowing the
user to more easily control the shared system. As a result, we hypothesize that optimizing for human
empowerment will lead to a more useful assistant faster than vanilla RL.
Experiment Setup. The main simulation used is Lunar Lander from OpenAI Gym [ 8], as it emulates
challenges of teleoperating a control system. Generally human players ﬁnd the game incredibly
difﬁcult due to the dynamics, fast reaction time required, and limited control interface. The goal
of the game is to control a lander using a main thruster and two lateral thrusters in order to land at
a randomly generated site indicated by two ﬂags, giving the player a positive reward. The game
ends when the lander crashes on the surface, exceeds the time limit, or ﬂies off the edges of the
screen, giving the player a negative reward. The action space consists of six discrete actions that are
combinations of the main thruster settings (on, off) and the lateral thruster settings (left, right, off).
To assist players, we use DQN to train a copilot agent whose reward depends on landing successfully
6
at the goal, but cannot observe where that goal is and instead keeps a memory of the past 20 actions
input by the user. The user input and copilot action space are identical and the state-space contains
the lander’s position, velocity, tilt, angular velocity, and ground contact sensors. We compute the
empowerment-based reward bonus by randomly rolling out 10 trajectories at each state and computing
the variance in the ﬁnal positions of the lander, as described in Section 3.2. Since both the human
and agent control the same lander, the empowerment of the shared autonomy system is coupled. We
hypothesize that including the empowerment term would be particularly useful for a challenging
control task such as Lunar Lander, since states where the lander is more stable are easier for the
player to control.
Simulation Experiments. We ﬁrst test our method using simulated ‘human’ pilots as designed in
the original study [ 35]. First, an optimal pilot is trained with the goal as a part of its state space
using DQN. Each of the ‘human’ pilots are imperfect augmentations of this optimal pilot as follows:
the Noop pilot takes no actions, the Laggy pilot repeats its previously taken action with probability
p= 0:85, the Noisy pilot takes a wrong action with probability p= 0:3(e.g. go down instead of up),
and the Sensor pilot only moves left or right depending on its position relative to the goal, but does
not use the main thruster. We use these pilots as simulated proxies for humans and train two copilots
with each of them: one with empowerment added to the reward function and one without. For the
empowered copilots, we do a hyperparameter sweep over cempfrom 0.00001 to 10.0 and found the
best performance with cemp= 0:001. The copilots were trained on 500 episodes of max length 1000
steps on AWS EC2. We then conduct a cross evaluation for each copilot and pilot pair by counting
successful landings across 100 evaluation episodes, averaged across 10 seeds.
Table 2: Best successful landing percentages for each simulated pilot in 100 episodes, averaged across
10 seeds. Our method improves upon the baseline for all pilots except Laggy, where the performance
is on par with the baseline.
Full Noop Laggy Noisy Sensor
No Copilot 58.2 0 11.9 11.6 0
No Empowerment (Baseline) 5.9 38.4 8.5 2.8
Empowerment 9.7 37.3 30.7 10.8
We ﬁnd that for almost all simulated pilots, our method leads to increased successful landing rate
(shown in Table 2) when paired with the best copilot. In the only case where we do not outperform
the baseline, with the Laggy pilot, the success rates between the baseline and our method only differ
by 1.1%. In all cases, empowerment perform better than the simulated single pilot with no copilot.
From our simulation results we see that our method can increase controllability for these simulated
pilots, leading to higher successful landing rates in Lunar Lander.
Copilot taking over
Landing too quickly
(a) Without empowerment.
Slowing
Down
Correcting
Behaviour
 (b) With empowerment.
Figure 4: Sample trajectories from the user study. The leftmost image shows the copilot take over and
swing the lander to the left even though the user is trying to move right. On the second image from
the left, the copilot does not slow the motion down sufﬁciently so the lander has too much momentum
and fails to land on the legs, leading to a crash. In the two right images, we see the frames overlap
more than in the no empowerment case due to the slower, more stable motion. The rightmost image
shows corrective stabilization behaviour when approaching the goal from an inconvenient angle. See
https://youtu.be/oQ1TvWG-Jns for a video summary of user study results.
User Study. To test our method with real pilots, we conducted an IRB-approved user study to
compare human player performance on Lunar Lander with human-in-the-loop training. As with the
simulated experiments, we manipulate the objective the copilot is trained with: our method with the
empowerment bonus in the reward function ( cemp= 0:001) and the baseline with no empowerment.
We found that the quality of assistance depends on this coefﬁcient as follows: increasing cemp
7
generally makes the copilot more inclined to focus on stabilization, but if cempis too high, the copilot
tends to override the pilot and focus only on hovering in the air. The objective measure of this study
is the successful landing rate of the human-copilot team, and the subjective measure is based on a
7-point Likert scale survey about each participant’s experience with either copilot. We designed the
survey to capture whether the copilots improved task performance, increased/decreased the user’s
autonomy and control, and directly compare the user’s personal preference between the two copilots
(refer to Table 4). We recruited 20 (11 male, 9 female) participants aged 21-49 (mean 25). Each
participant was given the rules of the game and an initial practice period of 25 episodes to familiarize
themselves with the controls and scoring, practice the game, and alleviate future learning effects.
To speed up copilot learning, each copilot was ﬁrst pretrained for 500 episodes with the simulated
Laggy pilot, then ﬁne-tuned as each human participant played for 50 episodes with each of the two
copilot conditions without being informed how the copilots differ. To alleviate the confounding factor
of improving at the game over time, we counterbalanced the order of the two conditions.
The objective success rates from the user study are summarized in Table 3. We ran a paired two-
sample t-Test on the success rate when the copilot was trained with and without empowerment and
found that the copilot with empowerment helped the human succeed at a signiﬁcantly higher rate
(p<0:0001 ), supporting our hypothesis.
Table 3: Objective User Study Results: Average success percentages with standard deviation.
No Empowerment Empowerment
Success Rate 179 337
The results of our survey are summarized in Table 4 where we report the mean response to each
question for each copilot and the p-value of a paired two-sample t-Test. To account for multiple
comparisons, we also report the Bonferroni-corrected p-values. We ﬁnd that the participants perceived
the empowerment copilot’s actions made the game signiﬁcantly easier (p= 0:007) as compared to
the no-empowerment copilot. Furthermore, the two comparison questions have a signiﬁcant level
of internal consistency, with Chronbach’s = 0:96, and the most frequent response is a strong
preference (7) for the empowerment copilot. Although on average the perception of control and
assistance is higher with the empowerment copilot, this difference was not found to be signiﬁcant.
Comments from the participants suggest that the empowerment copilot generally provided more
stabilization at the expense of decreased human left/right thruster control, which the most users
were able to leverage and collaborate with – the copilot increased stability and reduced the lander
speed, allowing the user to better navigate to the goal (see Figure 4b). On the other hand, the no-
empowerment copilot did not consistently provide stability or would take over user control, moving
away from the goal (see Figure 4a). These results are exciting as prior work in shared autonomy has
proposed heuristics for assistance, such as virtual ﬁxtures [ 28,1] or potential ﬁelds [ 3,9], but we
ﬁnd that one beneﬁt of empowerment is the natural emergence of stabilization without relying on
heuristics. We also note that our empowerment-inspired reward bonus allowed us to leverage the
intuitive beneﬁts of controllability without the large computational costs of the empowerment method,
allowing the empowerment-based copilot to learn alongside each human player in real time. While
our method empirically demonstrated the merits of empowerment-based assistance, the user study
also highlighted limitations of assistance through pure empowerment. Predominantly optimizing
for empowerment can lead to failure modes where the assistant prevents landing altogether due to
prioritizing stability excessively.
5 Discussion and Conclusion
Summary. In this work, we introduce a novel formalization of assistance: rather than attempting
to infer a human’s goal(s) and explicitly helping them to achieve what we think they want, we
instead propose empowering the human to increase their controllability over the environment. This
serves as proof-of-concept for a new direction in human-agent collaboration. In particular, our work
circumvents typical challenges in goal inference and shows the advantage of this method in different
simulations where the agents are generally able to assist the human without assumptions about the
human’s intentions. We also propose an efﬁcient algorithm inspired by empowerment for real time
8
Table 4: Subjective User Study Survey Results. 1 = strongly disagree, 7 = strongly agree. p-value
are Bonferonni-corrected p-values.
Question Emp No-Emp p-value p-valueAutonomyI had sufﬁcient control over the lander’s movement. 4.7 3.6 0.025 0.175
The copilot often overrode useful controls from me. 4.5 5.35 0.053 0.371
The copilot did not provide enough assistance. 3 3.45 0.14 0.98
The copilot provided too much assistance. 3.15 3.95 0.12 0.84
The quality of the copilot improved over time. 6.25 5.75 0.096 0.672PerformThe copilot’s actions made parts of the task easier. 6.4 5.55 0.001 0.007
The copilot’s assistance increased my performance
at the task.6.4 5.6 0.032 0.21
Comparison Questions Chronbach’s  Mean Mode
I preferred the assistance from the empowerment copilot to the
no empowerment copilot.0.96 4:92 7I was more successful at completing the task with the
empowerment copilot than the no empowerment copilot.
assistance-based use cases, allowing us to conduct human-in-the-loop training with our method and
demonstrate success in assisting humans in a user study.
Limitations and Future Work. Our experiments ﬁnd that in cases where the human’s goals can be
inferred accurately, general empowerment is not the best method to use. As there exist situations
where optimizing for human empowerment will not be the best way to provide assistance, in future
work we seek to formalize how goal-agnostic and goal-oriented assistance can be combined. For
example, a natural continuation of this work we will explore a hybrid approach, combining local
and global planning. Another area of future work is to explore the use of human empowerment for
assisting humans with general reward functions. In this paper, we primarily focus on assisting with
goal states as a way to compare with existing goal inference assistance methods, but the human
empowerment objective can potentially apply to more general reward formulations. Furthermore, we
proposed a proxy to empowerment in order to make a user study feasible, which came at a cost of
not being an accurate measure of true empowerment. Although we found that our simpliﬁed method
led to signiﬁcant improvement in our user study, the current proxy assumes homogeneous noise and
is sensitive to scenarios with noise varying between different states, and the sample-based method
naturally requires more computational power as action space grows – that being said, there are many
meaningful assistance applications with small action spaces (e.g. navigation with mobility devices,
utensil stabilization). Future work can analyze the tradeoffs between computational tractability and
numerical empowerment accuracy in the human assistance domain.
Broader Impact
As our work is focused on enabling artiﬁcial agents to learn to be more useful assistants, we believe
it has the potential for signiﬁcant broader impact in both the research community and for the future
usefulness of assistive agents in a variety of real world applications (e.g. assistive robotics in
elder care, prosthetics). The most immediate impact of our work lies in the direct application of
empowerment for assisting humans. In the research community, the novel use of empowerment
for human assistance can motivate further work in goal-agnostic human-agent collaboration and
assistance. As emphasized in our work, in cases where goal inference is challenging or when aiding
under the assumption of an incorrectly inferred goal may have risks, our method acts as an alternative
to potential pitfalls of goal inference. This, we believe, can be crucial when applied to assisting
people in the real world. In particular, our successful results with the Lunar Lander user study
suggests that our method can assist humans engaging in a challenging shared autonomy control task.
Our method invites extensions to real-world shared autonomy tasks (e.g. ﬂying a quadrotor drone,
teleoperation of a high DOF robotic arm with a joystick controller, etc.). Outside of research labs,
the broader impact of a goal-agnostic human assistive method lies in the potential of applying this
general method to a wide variety of assistive tasks – ranging from software assistants to assitive
robotics. We also emphasize that while assistive technologies are developed to aid people, safety
procedures and strict certiﬁcation are necessary before a real-world application of our method. With
direct human interaction, failures of the system can critically impact people (whether that be through
physical robotic failures, or privacy concerns with software assistants).
9
At a societal scale, we also hope that proposing a method that optimizes for human controllability
will encourage future ethical discussions about how to realize learning assistive agents that balance
providing effective help while also ultimately guaranteeing as much human autonomy and control as
possible. Importantly, as empowerment aims to enhance the element of autonomy in the human, this
offers a systematic route to avoid the possible drawback of an overly helpful, but constricting artiﬁcial
assistant. Depending on the situation in which AI agents are employed, there can be uncertainty
around the extent to which people (at a personal level, cultural level, etc.) require a balance between
autonomy and assistance. The importance of different types of autonomy (e.g. personal, moral) for
different groups of individuals (e.g. age groups, cultural groups) and how they can be positively
or negatively affected by applications of human empowerment can be examined in other areas of
research (e.g. sociology, philosophy, psychology).
Acknowledgements
This research is supported by: an Azure sponsorship from Microsoft, a Berkeley AI Lab (BAIR)
Fellowship, ONR through PECASE N000141612723, and NSF under grant NRI-#1734633.
References
[1]D. Aarno, S. Ekvall, and D. Kragic. Adaptive virtual ﬁxtures for machine-assisted teleoperation
tasks. volume 2005, pages 897– 903, 05 2005.
[2]P. Abbeel and A. Y . Ng. Apprenticeship learning via inverse reinforcement learning. In
Proceedings of the twenty-ﬁrst international conference on Machine learning , page 1, 2004.
[3]P. Aigner and B. McCarragher. Human integration into robot control utilising potential ﬁelds.
InProceedings of International Conference on Robotics and Automation , volume 1, pages
291–296 vol.1, 1997.
[4]B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from
demonstration. Robotics and autonomous systems , 57(5):469–483, 2009.
[5]C. Baker, R. Saxe, and J. B. Tenenbaum. Bayesian models of human action understanding. In
Advances in neural information processing systems , pages 99–106, 2006.
[6]R. Blahut. Computation of channel capacity and rate-distortion functions. IEEE transactions
on Information Theory , 18(4):460–473, 1972.
[7]A. Bobu, A. Bajcsy, J. F. Fisac, and A. D. Dragan. Learning under misspeciﬁed objective spaces.
CoRR , abs/1810.05157, 2018.
[8]G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba.
Openai gym. CoRR , abs/1606.01540, 2016.
[9]J. W. Crandall and M. A. Goodrich. Characterizing efﬁciency of human robot interaction: a
case study of shared-control teleoperation. In IEEE/RSJ International Conference on Intelligent
Robots and Systems , volume 2, pages 1290–1295 vol.2, 2002.
[10] I. M. de Abril and R. Kanai. A uniﬁed strategy for implementing curiosity and empowerment
driven reinforcement learning. CoRR , abs/1806.06505, 2018.
[11] A. D. Dragan and S. S. Srinivasa. A policy-blending formalism for shared control. Int. J. Rob.
Res., 32(7):790–805, June 2013.
[12] Y . Duan, M. Andrychowicz, B. Stadie, O. J. Ho, J. Schneider, I. Sutskever, P. Abbeel, and
W. Zaremba. One-shot imitation learning. In Advances in neural information processing
systems , pages 1087–1098, 2017.
[13] J. F. Fisac, M. A. Gates, J. B. Hamrick, C. Liu, D. Hadﬁeld-Menell, M. Palaniappan, D. Malik,
S. S. Sastry, T. L. Grifﬁths, and A. D. Dragan. Pragmatic-pedagogic value alignment. In
Robotics Research , pages 49–57. Springer, 2020.
10
[14] J. F. Fisac, C. Liu, J. B. Hamrick, S. S. Sastry, J. K. Hedrick, T. L. Grifﬁths, and A. D. Dragan.
Generating plans that predict themselves, 2018.
[15] D. Gopinath, S. Jain, and B. D. Argall. Human-in-the-loop optimization of shared autonomy in
assistive robotics. IEEE Robotics and Automation Letters , 2(1):247–254, 2016.
[16] K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. arXiv preprint
arXiv:1611.07507 , 2016.
[17] C. Guckelsberger, C. Salge, and S. Colton. Intrinsically motivated general companion npcs via
coupled empowerment maximisation. In 2016 IEEE Conference on Computational Intelligence
and Games (CIG) , pages 1–8. IEEE, 2016.
[18] D. Hadﬁeld-Menell, S. J. Russell, P. Abbeel, and A. Dragan. Cooperative inverse reinforcement
learning. In Advances in neural information processing systems , pages 3909–3917, 2016.
[19] J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in neural information
processing systems , pages 4565–4573, 2016.
[20] S. Javdani, H. Admoni, S. Pellegrinelli, S. S. Srinivasa, and J. A. Bagnell. Shared autonomy via
hindsight optimization for teleoperation and teaming. The International Journal of Robotics
Research , 37(7):717–742, 2018.
[21] S. Javdani, S. S. Srinivasa, and J. A. Bagnell. Shared autonomy via hindsight optimization.
Robotics science and systems: online proceedings , 2015, 2015.
[22] T. Jung, D. Polani, and P. Stone. Empowerment for continuous agent—environment systems.
Adaptive Behavior , 19(1):16–39, 2011.
[23] A. S. Klyubin, D. Polani, and C. L. Nehaniv. All else being equal be empowered. In European
Conference on Artiﬁcial Life , pages 744–753. Springer, 2005.
[24] A. S. Klyubin, D. Polani, and C. L. Nehaniv. Empowerment: A universal agent-centric measure
of control. In 2005 IEEE Congress on Evolutionary Computation , volume 1, pages 128–135.
IEEE, 2005.
[25] O. Macindoe, L. P. Kaelbling, and T. Lozano-Pérez. Pomcop: Belief space planning for
sidekicks in cooperative games, 2012.
[26] J. Mainprice and D. Berenson. Human-robot collaborative manipulation planning using early
prediction of human motion. In 2013 IEEE/RSJ International Conference on Intelligent Robots
and Systems , pages 299–306. IEEE, 2013.
[27] A. Majumdar, S. Singh, A. Mandlekar, and M. Pavone. Risk-sensitive inverse reinforcement
learning via coherent risk models. In Robotics: Science and Systems , 2017.
[28] P. Marayong, A. Bettini, and A. Okamura. Effect of virtual ﬁxture compliance on human-
machine cooperative manipulation. In IEEE/RSJ International Conference on Intelligent Robots
and Systems , volume 2, pages 1089–1095 vol.2, 2002.
[29] S. Mohamed and D. J. Rezende. Variational information maximisation for intrinsically motivated
reinforcement learning. In Advances in neural information processing systems , pages 2125–
2133, 2015.
[30] A. Y . Ng, S. J. Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1,
pages 663–670, 2000.
[31] S. Nikolaidis and J. Shah. Human-robot cross-training: Computational formulation, modeling
and evaluation of a human team training strategy. In 2013 8th ACM/IEEE International
Conference on Human-Robot Interaction (HRI) , pages 33–40, 2013.
[32] S. Pellegrinelli, H. Admoni, S. Javdani, and S. Srinivasa. Human-robot shared workspace
collaboration via hindsight optimization. In 2016 IEEE/RSJ International Conference on
Intelligent Robots and Systems (IROS) , pages 831–838. IEEE, 2016.
11
[33] C. Pérez-D’Arpino and J. A. Shah. Fast target prediction of human reaching motion for
cooperative human-robot manipulation tasks using time series classiﬁcation. In 2015 IEEE
International Conference on Robotics and Automation (ICRA) , pages 6175–6182, 2015.
[34] D. Ramachandran and E. Amir. Bayesian inverse reinforcement learning. In IJCAI , volume 7,
pages 2586–2591, 2007.
[35] S. Reddy, A. D. Dragan, and S. Levine. Shared autonomy via deep reinforcement learning.
arXiv preprint arXiv:1802.01744 , 2018.
[36] S. Reddy, A. D. Dragan, and S. Levine. Where do you think you’re going?: Inferring beliefs
about dynamics from behavior. CoRR , abs/1805.08010, 2018.
[37] S. Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual
conference on Computational learning theory , pages 101–103, 1998.
[38] C. Salge, C. Glackin, and D. Polani. Changing the environment based on empowerment as
intrinsic motivation. CoRR , 2014.
[39] C. Salge, C. Glackin, and D. Polani. Empowerment–an introduction. In Guided Self-
Organization: Inception , pages 67–114. Springer, 2014.
[40] C. Salge and D. Polani. Empowerment as replacement for the three laws of robotics. Frontiers
in Robotics and AI , 4:25, 2017.
[41] S. Tiomkin, D. Polani, and N. Tishby. Control capacity of partially observable dynamic systems
in continuous time. arXiv preprint arXiv:1701.04984 , 2017.
[42] H. J. Wilson and P. R. Daugherty. Collaborative intelligence: humans and ai are joining forces.
Harvard Business Review , Jul 2018.
[43] H. J. Wilson and P. R. Daugherty. Human Machine: Reimagining Work in the Age of AI . Harvard
Business Review, 2018.
[44] R. Zhao, S. Tiomkin, and P. Abbeel. Learning efﬁcient representation for intrinsic motivations.
2019.
[45] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement
learning. In Aaai , volume 8, pages 1433–1438. Chicago, IL, USA, 2008.
12

Yuqing Du
Stas Tiomkin
Emre Kiciman
Daniel Polani
Pieter Abbeel
Anca D
Dragan


Human-robot interaction for truck platooning using hierarchical dynamic games.

Human-robot interaction for truck platooning using
hierarchical dynamic games
Elis Stefansson1, Jaime F. Fisac2, Dorsa Sadigh3, S. Shankar Sastry2and Karl H. Johansson1
Abstract — This paper proposes a controller design frame-
work for autonomous truck platoons to ensure safe interaction
with a human-driven car. The interaction is modelled as a
hierarchical dynamic game, played between the human driver
and the nearest truck in the platoon. The hierarchical decompo-
sition is temporal with a high-ﬁdelity tactical horizon predicting
immediate interactions and a low-ﬁdelity strategic horizon
estimating long-horizon behaviour. The hierarchical approach
enables feasible computations where human uncertainties are
represented by the quantal response model, and the truck is
supposed to maximise its payoff. The closed-loop control is
validated via case studies using a driving simulator, where
we compare our approach with a short-horizon alternative
using only the tactical horizon. The results indicate that our
controller is more situation-aware resulting in natural and safe
interactions.
I. I NTRODUCTION
A. Motivation
The deployment of autonomous vehicles on public roads
has received substantial attention with the DARPA urban
challenge [7] and the Waymo car [12] as notable exam-
ples. In contrast to earlier proposals often considering the
autonomous system in isolation [21], these new approaches
aim to integrate the autonomous vehicles into the exist-
ing infrastructure. However, new challenges arise since the
autonomous vehicles need to interact with human drivers.
Accurate interaction models are hence crucial [13], [18],
because simple models may result in opaque and potentially
unsafe behaviour.
An important vehicle application domain is road freight,
where recent work has shown a signiﬁcant beneﬁt for truck
platooning [6] due to reduced aerodynamic drag [1]. Typical
platoon controllers have been designed in isolation from
other drivers or used simple interaction models [19], which
can generate safety and performance degradation once de-
ployed on public roads. In particular, a human-driven car
could try to cut in between two trucks in the platoon, as in
Fig. 1. Such interference may happen if the human is subject
to danger in her own lane (e.g., blocked lane by road work),
but should in non-critical situations be discouraged due
1School of Electrical Engineering and Computer Science, KTH Royal
Institute of Technology, Sweden.
2Department of Electrical Engineering and Computer Sciences. Univer-
sity of California, Berkeley, United States.
3Computer Science Department, Stanford University, United States.
Email: {elisst, kallej }@kth.se, {jﬁsac, shankar sastry }@berkeley.edu,
dorsa@cs.stanford.edu
This work was partially funded by the Swedish Foundation for Strategic
Research, the Swedish Research Council, and the Knut och Alice Wallen-
berg foundation. The authors would like to thank Ehsan Nekouei for valuable
feedback on this paper.
Fig. 1. A human-driven car has cut in between two trucks in the platoon.
Such interference may happen if the human is subject to danger in its own
lane (e.g., blocked lane by road work), but should in non-critical situations
be discouraged due to performance degradation of the platoon.
to performance degradation of the platoon. Consequently,
situation-aware interacting platoons are needed.
B. Contribution
In this paper, we propose a platoon controller modelling
the human-platoon interaction as a hierarchical dynamic
game. The result is a situation-aware interacting platoon
which deliberately opens up wider gaps for safety-critical
lane changes, but otherwise discourage such cut-ins. More
precisely, our contributions are three-fold:
We ﬁrst propose a hierarchical dynamic game framework
for modelling the interaction between an autonomous vehicle
and a human driver, building on the formulation proposed in
[10]. The game is dynamic to capture the sequential decision
structure between the vehicle and the human, critical for
long-horizon interactions. To obtain tractable solutions, a hi-
erarchical approach is used. The hierarchical decomposition
is temporal with a high-ﬁdelity tactical horizon predicting
immediate interactions and a low-ﬁdelity strategic horizon
estimating longer-term interactions. The human driver is rep-
resented by the quantal response model [22] for the strategic
horizon to capture long-term uncertainties. The solution of
the game is constructive as the autonomous vehicle obtains
optimal controls by maximising its payoff.
Secondly, we present a situation-aware platoon controller
interacting with a human-driven car as in Fig. 1. The platoon
controller is decentralised and restricts the human interaction
to the nearest truck (the interacting truck). The interacting
truck uses a hierarchical dynamic game to predict future
interactions and obtain optimal controls. The non-interacting
trucks use instead simple single-agent optimisation schemes
to get optimal controls.
Thirdly, we validate the platoon controller using case
studies with a human driver. We compare our controller with
a short look-ahead alternative using only the tactical horizon.
We show that the situation-aware behaviour described above
naturally emerges from our controller, whereas the latter
alternative is largely situation-agnostic.
C. Related work
Autonomous vehicles typically treat human drivers as
immutable and plan thereafter, which may lead to opaque
and overly-defence behaviour [18]. A work [13] not re-
quiring this assumption shows how an autonomous car can
instead leverage effects on a human-driven car. Here, a high-
ﬁdelity short-horizon prediction (0.5 s) is used, assuming
that the human can infer the autonomous car’s planned
trajectory and react accordingly. Unfortunately, for long-
horizon predictions, this assumption may generate overconﬁ-
dent autonomous vehicle behaviour. To address long-horizon
interaction, dynamic game models have been applied [15],
[11], [3]. However, these models use simpliﬁed high-level
dynamics with pre-deﬁned ﬁxed manoeuvres (e.g., ﬁxed lane
changes) not necessarily capturing immediate high-ﬁdelity
interactions; the distinction between autonomous vehicles
and human drivers is not addressed either. The hierarchical
dynamic game framework presented here can be seen as
taking the best from these approaches.
Previous work on the human-platoon interaction problem
include always maintaining tight truck gaps to discourage car
cut-ins [9]; or opening up a wider truck gap when a car enters
the platoon identiﬁed by sensors [8], [20]. These models
are situation-agnostic, as opposed to the platoon controller
proposed here.
Finally, the work [10] considers hierarchical dynamic
games for an autonomous car interacting with a simulated
human-driven car. We here extend their game framework
to a more complex problem having an autonomous platoon
interacting with a human-driven car.
D. Outline
The remaining paper is structured as follows. Section II
presents the hierarchical dynamic game framework. The
game considers two agents, an autonomous vehicle and a
human driver, interacting on the road. Section III presents
the platoon controller, where the game is used to model
the interplay between the human driver and the interacting
truck. Section IV considers case studies. Finally, Section V
concludes the paper with discussion and future research
directions.
A more detailed description of the material covered in this
paper can be found in the Master’s thesis [17].
II. H IERARCHICAL DYNAMIC GAME
Consider an autonomous vehicle Rinteracting with a
human driver Hon the road. The interaction is formalisedby a discrete-time control system with state xt2Rnat time
t2Z. The agents can apply control inputs ut
R2URand
ut
H2UHwithin permissible compact sets UR✓RnRand
UH✓RnHrespectively. The control inputs result in a new
state xt+1according to a transition map f
xt+1=f(xt,ut
R,ut
H), (1)
updated over time intervals of length t2R+. The objective
of the autonomous vehicle is captured by a look-ahead
reward. More precisely, setting a ﬁnite horizon of N2Z+
steps, it seeks to maximise
RR(x0,u0:N
R,u0:N
H)=NX
t=0rR(xt,ut
R,ut
H), (2)
where x0is the current state and rR(xt,ut
R,ut
H)is the
cumulative reward at time t. The reward rRencodes prefer-
ences such as obstacle avoidance, lane-keeping and smooth
driving. We assume that the dynamics (1) and x0are known
to the autonomous vehicle. What remains is thus to predict
u0:N
H. Since the human is affected by the actions of the
robot with objectives that may conﬂict, a game-theoretic
formulation is proposed. One such formulation is to assume
that the human can observe xtat each time, has a similar
reward RH(x0,u0:N
R,u0:N
H)she seeks to maximise, and that
we have a predictive model ut
H=ut
H(xt,ut
R). This gener-
ates a closed-loop feedback structure where optimal actions
can be computed using dynamic programming [4], [16].
However, for long-horizon interactions (large N), computing
such schemes can be cumbersome due to the curse of
dimensionality; an approximation is needed. Towards this,
we note that while immediate interactions (e.g., 0.0–0.5 s)
may need a high-ﬁdelity representation of the dynamics,
future long-horizon interactions (e.g., 0.5–5.5 s) may instead
be accurately estimated using only approximative represen-
tations.
With this insight, similar to [10], we introduce a bilevel
hierarchical dynamic game decomposed in time by a high-
ﬁdelity tactical horizon modelling immediate interactions and
a low-ﬁdelity strategic horizon approximating future long-
horizon interactions. A strategic planner solves the strategic
horizon as a dynamic game, with value functions as outputs.
A tactical planner then solves the tactical plus the strategic
horizon by augmenting high-ﬁdelity dynamics with long-
horizon estimates using the strategic value functions.
A. Strategic Planner
The strategic planner considers a discrete-time hybrid
system approximation with simpliﬁed dynamics1
˜sk+1=˜f(˜sk,˜uk
R,˜uk
H), (3)
representing continuous time steps of ˜t2R+. Here,
˜sk=( ˜qk,˜xk)is the state with discrete state ˜qk2˜Q,
continuous state ˜xk2˜X(˜qk)✓R˜n, and continuous control
1For brevity, we incorporate both discrete and continuous dynamics in ˜f.
inputs ˜uk
i2˜Ui(˜qk)✓R˜ni.2The idea with the hybrid system
approach is to partition the original state space into smaller
domains {˜X(˜q)}˜q2˜Qwhere local approximations can be
applied to reduce problem size (in Section III, two domains
are considered conﬁning the human-driven car to its lane or
entering the truck gap). As a result we typically get ˜n<n ,
˜nR<n R,˜nH<n H, and usually set ˜t>tto further
reduce computations.
We also specify a simpliﬁed reward for the autonomous
vehicle of the form
˜RR(˜s0,˜u0:K1
R,˜u0:K1
H)=K1X
k=0˜rR(˜sk,˜uk
R,˜uk
H)+R(˜sK)
(4)
which the autonomous vehicle seeks to maximise. The ter-
minal reward Rcan be used to represent a critical event
(e.g., a collision) without the need of adding extra states. We
assume a human reward ˜RHon an analogous form, which
can be based on inverse reinforcement learning [13].
The solution of the game is a modiﬁcation of the classical
closed-loop feedback Stackelberg solution [5] having an
uncertain follower acting according to a probability distri-
bution instead of being a (purely) rational maximiser. More
precisely, we let the human be the (uncertain) follower with
a noisy decision rule p(˜uk
H|˜sk,˜uk
R)at each time step kgiven
by the quantal response model [22]
p(˜uk
H|˜sk,˜uk
R)/eQk
H(˜sk,˜uk
R,˜uk
H). (5)
Here, Qk
His the state-action value of the human at time step
kand>0is the rationality parameter.3The autonomous
vehicle, being the leader, maximises instead its state-action
value Qk
R(˜s,˜uR)subject to the distribution (5).
The values Qk
R,Qk
Hare obtained in backward time via
dynamic programming according to
˜uk+1?
R(˜s) := arg max
˜uQk+1
R(˜s,˜u),8˜s2[˜q2˜Q{˜q}⇥˜X(˜q)
˜ui
H⇠eQk
H(˜sk,˜uk
R),i2{k,k+1}
Qk
H(˜sk,˜uk
R,˜uk
H)=˜rH(˜sk,˜uk
R,˜uk
H)+
E˜uk+1
HQk+1
H(˜sk+1,˜uk+1?
R(˜sk+1),˜uk+1
H)
Qk
R(˜sk,˜uk
R)=E˜uk
H˜rR(˜sk,˜uk
R,˜uk
H)+
Qk+1
R(˜sk+1,˜uk+1?
R(˜sk+1)) (6)
initialised by QK
R(˜s,·)= R(˜s),QK
H(˜s,·,·)= H(˜s). The
(state) value functions Vk
Rand Vk
H, determining optimal
remaining rewards from step k, are then given by Vk
R(˜sk)=
Qk
R(˜sk,uk?
R)andVk
H(˜sk)=Euk
HQk
H(˜sk,uk?
R,uk
H).
Equation (6) enables a straight-forward numerical imple-
mentation by discretising the state-action space, referring to
[17] for details.
We stress that the actions in the strategic planner will
never be executed. Instead the value functions are used by
the tactical planner to estimate long-horizon interactions.
2Discrete control inputs could easily be incorporated, but are not used in
our application.
3In words, (5) models the human as nosily-rational agent (exponentially)
more likely to take actions yielding higher payoffs.B. Tactical Planner
In the tactical planner, the autonomous vehicle consid-
ers its immediate reward over a short horizon M⌧N
augmented with an estimate ERof the future long-horizon
reward based on the value functions from the strategic
planner. This yields the total reward
¯RR(x0,u0:M1
R,u0:M1
H)=M1X
t=0rR(xt,ut
R,ut
H)+ER(xM)
(7)
subject to (1). We assume a completely analogous human
reward ¯RH, where rHcan be obtained via inverse rein-
forcement learning [13]. The estimates ERand EHare
constructed by ﬁrst obtaining value function proxies ˆVRand
ˆVHwhich patch together the strategic value functions over
the hybrid domains and the time domain. These proxies
represent the strategic value functions but in continuous
time tand high-ﬁdelity state x. More precisely, ˆVi(t, x)
represent the strategic value function for agent istarting at
time t2[0,K˜t]on the strategic horizon and in state x(in
particular, t=0 uses the full strategic horizon), see [17] for
details. The estimates are then set to Ei(xM): = ˆVi(t, xM).
Typically, t=0 yielding a total look-ahead of Mt+K˜t
seconds (tactical plus strategic horizon), but Section III also
considers a varying tadapting the strategic horizon to a
critical event.
We ﬁnally specify the game’s solution. Due to the short
horizon ( M), the dynamic coupling between the agents is
less critical and we therefore use the open-loop Stackelberg
solution [5] assuming that the human maximises her reward
¯RH. This leads to a nested optimisation
u0:M1?
R= arg max
u0:M1
R¯RR(x0,u0:M1
R,u0:M1?
H)
s.t. u0:M1?
H= arg max
u0:M1
H¯RH(x0,u0:M1
R,u0:M1
H)(8)
for the autonomous vehicle. We have assumed that the human
accurately infers the state x0, the planned control trajectory
u0:M1
R(justiﬁed by small M) and its long-horizon reward
estimate EH. Numerically, this nested optimisation can be
solved using the Quasi-Newton gradient method L-BFGS [2],
see [17] for details.
C. Summary
The presented hierarchical dynamic game predicts future
interactions between an autonomous vehicle Rand a human
driver H. The solution is constructive and can be used by the
autonomous vehicle to obtain optimal controls: At current
state x0, it solves (8) and executes u0?
R, while the human
driver executes u0
H(which may not equal u0?
H); The system
then moves to a new current state according to (1) and the
process is repeated in a receding horizon fashion.
The next section considers the platoon setup with multiple
semi-autonomous4trucks. The hierarchical dynamic game is
4The trucks are autonomous longitudinally but ﬁxed to their platoon lane
by manual control.
then used to model the interaction between a human driver
andoneof the trucks.
III. P LATOON CONTROLLER
Consider a platoon of four semi-autonomous trucks driv-
ing on a two-lane highway shared with a human-driven car
as in Fig. 2 (cf. Fig. 1).(b)
Fig. 2. The four semi-autonomous trucks and the human-driven car on the
highway. The interacting truck is currently truck 3 since the car is between
truck 2 and 3 (dashed red lines).
H
T3=R T2 T1 T4
Fig. 3. Control architecture represented as a block diagram. The current
interacting truck (truck 3) models the interaction with the human Has a
hierarchical dynamic game.
Towards a platoon controller, we adapt a decentralised
control architecture for the platoon controller where the
planned trajectory of the truck ahead is passed down to the
truck behind, initialised by the leading truck. The truck that
is closest behind the human-driven car, the interacting truck
(see Fig. 2), models the human interaction as a hierarchical
dynamic game from Section II; The non-interactive trucks
use instead simple single-agent optimisation schemes. The
control architecture is summarised in Fig. 3 as a block
diagram.
We next specify the interacting truck in Sections III-A
to III-D, assuming it to be one of the following trucks for
simplicity. We then describe the single-agent optimisation
used by the non-interactive trucks in Section III-E and
conclude with a summary in Section III-F.
A. Interactive Truck
The interactive truck uses a game as in Section II to predict
the interaction with the human-driven car, solving (8) in a
receding horizon with time steps t=0.1s. The rewards
¯Riin (8) here depend on the current execution via Ei. More
precisely, at each time step, the truck monitors any upcoming
critical event and acts according to two cases: If a critical
event is predicted to happen Mt<t c<M t+K˜t
seconds ahead (i.e., the critical event happens on the strategic
horizon), then Ei(xM)= ˆVi(t, xM), where tis picked so
that the total look-ahead Mt+(K˜tt)coincides with
the predicted event time tc; the critical event can then be
incorporated into the strategic terminal rewards i. Other-
wise, the interactive controller uses the full strategic horizon,
Ei(xM)= ˆVi(0,xM). The procedure is depicted in Fig. 4.
Henceforth, the critical event is restricted to a potential head-
on collision between the human-driven car and another car
(thehead-on car ) driving in the opposite (wrong) direction,Current timeTimeCritical event
Time = Current time(a) Full strategic horizon
(c) Full strategic horizon(b) Adapted strategic horizon
Tactical horizonStrategichorizonFig. 4. Execution procedure for the interactive truck: (a) a critical event
(e.g., a collision) is too far away on the prediction horizon and hence full
strategic horizon is used; (b) the event happens on the strategic horizon
and the horizon is adapted to the event and incorporated into the strategic
terminal rewards (red bars); (c) the event is not on the strategic horizon and
the full strategic horizon is again used.
see Fig. 7. This critical event was picked for implementation
simplicity and could in principle be exchanged by any other
critical obstacle, e.g., road work.
The next two sections consider the game speciﬁcation in
detail followed by execution details in Section III-D.
B. Interactive Truck: Tactical Horizon
We start by specifying the tactical horizon, i.e., dynamics
(1) and rewards rias in (7). We set M=5.
1) Dynamics: The human-driven car is assumed to follow
unicycle dynamics
⇥
˙x˙y˙✓˙v⇤
=⇥vcos ( ✓)vsin (✓)vu1u2↵cv|v|⇤
,
(9)
where (x, y),vand✓is the car’s position, speed and heading
angle, and ↵c=2.79·104m1is the air friction coefﬁcient
for a typical car. The control input u=[u1,u2]Tspeciﬁes
steering input u12[3.7,3.7]·103and acceleration
u22[2.5,2.5], bounded by typical car values. The truck
is instead ﬁxed in its lane with one-dimensional dynamics
⇥˙x˙v⇤
=⇥vu ↵t·v|v|⇤
(10)
where ↵t=5.64·105m1is the air friction coefﬁcient
for a typical truck. We conﬁne acceleration input to typical
truck values u2[2.9,0.59]. Both (9) and (10) are executed
discretely with t=0.1s.5
We also predict future states on the tactical horizon for the
truck ahead (the truck in front of the interacting truck) and
the head-on car. These predictions are considered ﬁxed in
the game: For the head-on car, we use a constant-velocity
prediction assuming access to the head-on car’s current
position x0
hand velocity v0
h; The predicted states of the
5Below, labels RandHare sometimes used to differentiate between the
vehicles’ parameters when needed (e.g., uH,2).
truck ahead is instead assumed given by the truck ahead
in accordance with the control architecture (cf. Fig. 3).
These predictions together with the truck and car dynamics
gives (1).
2) Rewards: The reward rRconsists of two parts:
rR(xt,ut
R,ut
H)=rcore
R(xt,ut
R,ut
H)+rgoal
R(xt,ut
R,ut
H).
The ﬁrst term rcore
R=✓TR(xt,ut
R,ut
H)is a linear com-
bination of core features Rwith inverse reinforcement
learning weights ✓from [13]; Included features are a speed
reference term (vvref)2(we set vref= 30 m/s), a
control penalty u2and a penalty for being near the truck
ahead and the human-driven car (using logistic sigmoids
with boundaries corresponding to collision). The second
term rgoal
Rencodes more speciﬁc objectives: A truck tailing
feature (x(xpxref))2where xpis the position of the
truck ahead and xrefis desired truck distance (we set xref
to an optimal truck gap of 9 meters); a penalty for having the
human-driven car near the truck ahead (logistic sigmoids)6;
and an extra high penalty for having a collision between the
human-driven car and the head-on car.7
The reward rHsimilarly equals rH=rcore
H+rgoal
H. The
features Hofrcore
H=✓THinclude staying on lanes,
staying inside road boundaries (one-dimensional Gaussians),
a control penalty u2, a speed reference term (vvref)2
(again vref= 30 m/s) and a penalty for being near the truck
and the truck ahead (logistic sigmoids), with weights ✓again
from [13]. The reward rgoal
Hhas an extra high penalty for
colliding with the head-on car and an extra right lane reward
modelling common preference.
Having speciﬁed the tactical horizon, we continue with
the strategic horizon and how it is solved with the strategic
planner as in II-A.
C. Interactive Truck: Strategic Horizon
To reduce problem size, we assume that the truck ahead
travels at constant velocity vref= 30 m/s and introduce a
hybrid system approximation. The hybrid system consists of
two discrete states called the lane mode and the merge mode .
Brieﬂy, the human is ﬁxed to the left lane in the lane mode
while it is near the truck gap with small velocity changes
in the merge mode, depicted in Fig. 5. Also, the head-
on car is not modelled dynamically; instead, its potential
collision with the human-driven car is solely incorporated in
the terminal rewards. We describe the setup in detail below
specifying the dynamics (3) ﬁrst, then the rewards (4), and
conclude with implementation details.
1) Dynamics: Truck dynamics coincide with (10) except
that relative coordinates xR,rel:=xRxp,vR,rel:=vR
vrefwith respect to the truck ahead are used
˙xR,rel
˙vR,rel
=vR,rel
uR↵t·(vR,rel+vref)|(vR,rel+vref)|
.
(11)
6This altruistic penalty is needed in practice since the truck ahead is not
actively interacting with the human.
7Omitting this penalty may lead to safety issues where the truck tries to
block the car by closing the truck gap.
Fig. 5. Domain of lane mode (upper) and merge mode (lower). Note that
the domain of the lane mode agrees with the interacting borders in Fig. 2.
The dynamics of the human-driven car depend instead on
the discrete state, as detailed below.
Lane mode. In the lane mode, the car is assumed to stay in
the left lane with domain as in Fig. 5 and dynamics analogous
to (11) but with relative coordinates xH,rel:=xHxp,
vH,rel:=vHvrefand↵cinstead of ↵t. Additionally, the
car may drive outside the domain (by passing the middle
of the truck or the truck ahead). To incorporate this, we
set a switch from the lane mode to itself approximately
corresponding to periodic conditions: If xH,rel>0(the car
has passed the truck ahead), the state is reset to8
xH,rel:=xH,relxref,x R,rel:=xref
vH,rel:=vH,rel,v R,rel:=vref.
and if xH,rel<x R,rel(the car has fallen behind the truck)9
xH,rel:=xH,relxR,rel,x R,rel:=xref
vH,rel:=vH,rel,v R,rel:=vR,rel.
Merge mode. In the merge mode, the car is near the
truck gap having low relative velocity (with respect to the
platoon). We then enable the car to change its position both
longitudinally and laterally with domain depicted in Fig. 5.
This is done at the expense of kinematic approximations
⇥˙xH,rel ˙yH⇤
=⇥wH,xwH,y⇤
,
where yHis the car’s lateral position, and we bound the
longitudinal velocity input wH,x2[2,1]and the lateral
velocity input wH,y2[l/3,l /3]; the latter corresponds to
a 3 s lane change having lane width l=3.7m.
Discrete transitions. We enable the car to switch between
the modes by placing transitions in both directions. If the
car slows down and is next to the truck gap, it is natural to
switch from lane mode to merge mode. Hence, we place a
switching condition
xR,rel+L/2xH,relL/2,2vH,rel1
and reset function xH,rel:=xH,rel,yH:=l.10Here,
L= 19 .8m is the truck length. For the other direction, we
8In words, the truck ahead becomes the new truck initially holding the
desired truck gap and reference speed. The new truck ahead is also assumed
to hold the reference speed vref.
9In words, the truck becomes the new truck ahead and the truck behind
it becomes the new truck, initially holding the desired truck gap and the
previous following velocity. It is also assumed that the new truck ahead
changes its velocity momentarily to vref.
10Here, yH=lcorresponds to the middle of the left lane. The ﬁrst
condition says that the position of the car is next to the truck gap, while the
second condition is consistent with the velocity bounds in the merge mode.
place a switching condition
xH,rel>L/2orxH,rel<x R,rel+L/2
l˜t·l/3yHl+˜t·l/3
and reset function xH,rel:=xH,rel,vH,rel:=wH,x.11
The discrete transitions are summarised in Fig. 6. The
dynamics (3) is formed by executing above discretely with
˜t=0.5s.
Lane
modeMerge
mode
near gap and low
relative speedaway from gap
passes a truck
Fig. 6. Informal overview of the discrete transitions with respect to the
human-driven car.
2) Rewards: The strategic rewards are based on the tac-
tical rewards. The truck reward equals ˜rR=˜rcore
R+˜rgoal
R
where rcore
Ris re-used to form ˜rcore
Rand all terms in rgoal
R
except the car collision penalty are re-used to form ˜rgoal
R.
The collision is instead represented by a high penalty 'in
the terminal reward for having the human-driven car in the
left lane (logistic sigmoids) with magnitude consistent with
the collision penalty on the tactical horizon. The result is
R(˜sK)=˜ rR(˜sK,0,0) + '(˜sK). This terminal reward is
used if a car collision is predicted on the strategic horizon
(cf. Fig. 4); otherwise, R(˜sK)=˜rR(˜sK,0,0).
The human reward ˜rHin both modes re-uses rHexcept
omitting the collision penalty, and in the merge mode: control
and speed features are appropriately modiﬁed for having
velocities as control inputs. The collision penalty is instead
incorporated in the terminal reward if needed, analogous to
the truck case.
3) Implementation details: We set K= 10 and run the
strategic planner with discretised domains as in Table I. We
then obtain value function proxies ˆVi(t, x)speciﬁed as neural
networks, see [17] for details.
D. Interactive Truck - Execution Details
The interacting truck solves (8) with time steps t=0.1s
and game speciﬁed above. More precisely, the predicted
collision time is given by tc=(x0
hx0
H)/(|v0
h|+|v0
H|). If
Mt<t c<Mt+K˜t, the collision is incorporated into
the terminal rewards ias in III-C.2 and estimates are set to
Ei(xM)= ˆVi(t, xM)where tupdates according to Mt+
(K˜tt)= tc. Otherwise we set Ei(xM)= ˆVi(0,xM).
This proceeds in a receding horizon fashion.
E. Non-Interactive Trucks
The non-interactive trucks need no game-theoretic
setup since the human driver is not close. Hence,
11The conditions merely says that the car is outside the truck gap lines
and sufﬁciently close to the middle of the left lane. More precisely, the
condition on yHis set so that the middle of the left lane yH=lcan be
reached in one time step from the merge mode.TABLE I
Variable Discretised interval Number of points
Lane mode
xR,rel [37.8,19.8] 41
vR,rel [2,1] 13
xH,rel [27.9,0] 21
vH,rel [6,6] 13
uR [2.9,0.59] 7
uH [5,2.5] 7
Merge mode
xR,rel [37.8,19.8] 41
vR,rel [2,1] 13
xH,rel [27.9,9.9] 32
yH [3.7/2,3·3.7/2] 13
uR [2.9,0.59] 7
wH,x [-2,1] 4
wH,y [-3.7/3, 3.7/3] 5
we restrict them to simple optimisation schemes
arg maxuR2UM
RPM1
t=0rR(xt,ut
R)+ rR(xM,0) with
M=5, dynamics as on the tactical horizon and rR(xt,ut
R)
including the features speed reference, control penalty and
truck tailing similar to the interactive truck.12This proceeds
in a receding horizon fashion with time steps t=0.1s.
F . Summary
We conclude with a summary. The platoon controller is
executed with time steps t=0.1s. At each time step, the
leading truck initiates the controller by computing its planned
state-control trajectory {xt
R,ut
R}M1
t=0. The state trajectory
is then passed down to the next truck, which computes its
planned trajectory, and the process is repeated until the last
truck is done. All non-interactive trucks use the single-agent
optimisation speciﬁed in Section III-E, while the interacting
truck uses the game as in Sections III-A to III-D.
IV . C ASESTUDIES
This section presents case studies involving a truck platoon
and a car on a two-lane highway, conducted in a driving
simulator. The platoon acts according to a speciﬁed controller
while the car is human-driven. We consider two scenarios,
thehead-on car and the off-ramp scenario for short. In
both scenarios, the car starts behind the platoon with aim
to keep a higher velocity. The human driver therefore starts
to overtake the platoon. In the head-on scenario, the driver
faces an approaching head-on car when driving next to the
middle trucks; the driver is notiﬁed by a countdown arrow in
the simulator, specifying how many seconds are left till the
head-on car appears on the screen (see Fig. 7), starting at 5
seconds. The human driver wants to avoid a head-on collision
and typically tries to cut in between the trucks. In the off-
ramp scenario, an off-ramp is instead approaching, alerted
by a similar 5-second countdown arrow (see Fig. 11). The
human driver may or may not want to take the off-ramp; we
consider both options.
The dynamics of the car and the trucks are given by (9)
and (10). The human driver steers the car via keyboard
inputs.
12The truck tailing term is omitted for the leading truck.
Hierarchical ControllerTactical Controller
Fig. 7. The controllers’ manoeuvres in the head-on scenario run. The
orange car is the head-on car.
Fig. 8. Truck acceleration and distance to its truck ahead (as a function
of time) for the controllers corresponding to Fig. 7.
We compare the proposed platoon controller in Section III
with a controller using only the tactical horizon for the in-
teractive truck (otherwise identical, see [17] for details). We
call them the hierarchical and tactical controller respectively.
For consistency, the trajectory of the human driver is pre-
recorded with the hierarchical controller and then re-used
for both controllers.13
Finally, in all plots, ‘time’ encodes the countdown time
until either a head-on car or an off-ramp appears in the
driving simulator.
1) Head-on car: The behaviour of the two controllers
from a head-on car scenario run can be seen in in Fig. 7
in the form of snapshots, while Fig. 8 plots the acceleration
of the (interacting) truck and the distance to its truck ahead.
An earlier brake can be noted for the hierarchical controller
predicting a merging car, leading to an earlier and wider
truck gap; the tactical controller is instead agnostic and starts
to brake only when the human actually merges (around 2 s),
due to its shorter horizon. This agnostic behaviour can lead to
safety-critical issues for the latter controller shown in Fig. 9.
Here, the countdown starts when the human overtakes the
truck ahead and hence she needs to brake heavily to reach the
gap in time. The manoeuvre is not predicted by the tactical
controller and the truck therefore almost hits the car as seen
13This could give the hierarchical controller an advantage. However, pre-
recording with the tactical controller yields similar behaviours indicating
that this advantage is minor, see [17] for details.
Hierarchical ControllerTactical Controller
Fig. 9. The controllers’ manoeuvres in the head-on car scenario run. Here,
the countdown starts when the human driver overtakes the truck ahead.
Fig. 10. Truck acceleration and truck-car distance for the controllers
corresponding to Fig. 9.
in Fig. 10. The hierarchical controller avoids this safety issue
with an earlier and heavier brake.
2) Off-ramp: Results from an off-ramp scenario run with
a human who does not merge are shown via snapshots in
Fig. 11 with truck gap and acceleration plotted in Fig. 12.
Since the event is not critical, the hierarchical controller
automatically discourages the human from entering its lane
by tightening the gap slightly, favouring performance from
reduced aerodynamic drag. The tactical controller is again
agnostic. If a human merges anyway, both controllers open
up a wider gap as seen in Fig. 13.
3) Conclusion: The scenarios indicate that the hierar-
chical controller generates situation-aware platoon-human
interactions. This situation-aware feature emerges from long-
horizon predictions and was not seen for the tactical con-
troller using only short-horizon predictions.Hierarchical ControllerTactical Controller
Fig. 11. The controllers’ manoeuvres in the off-ramp scenario run with a
human who does not merge.
Fig. 12. Truck acceleration and distance to its truck ahead for the
controllers corresponding to Fig. 11.Hierarchical ControllerTactical Controller
Fig. 13. The controllers’ manoeuvres in the off-ramp scenario run with a
merging human.
V. D ISCUSSION
We have proposed a truck platoon controller interacting
with a human-driven car. The platoon controller is decen-
tralised and restricts the human interaction to the nearest
truck, where the interacting truck uses a hierarchical dynamic
game [10] to predict future interactions. The hierarchical
decomposition is temporal with a high-ﬁdelity tactical hori-
zon predicting immediate interactions and a low-ﬁdelity
strategic horizon estimating long-horizon interactions. The
human is modelled with the quantal response model [22] on
the strategic horizon to capture long-term uncertainties. The
solution of the game is constructive where the truck obtains
optimal controls by maximising its payoff.
We demonstrated the performance of the platoon using
case studies with a human-driven car, comparing our con-
troller with a short-horizon alternative using only the tactical
horizon. We observed that gap openings emerged naturally
to facilitate safety-critical lane changes for the human driver,
but were avoided under normal cruising to discourage cut-
ins, favouring performance. This situation-aware behaviour
was not seen for the short-horizon alternative.
Future work should focus on games with incomplete
information structures. In particular, the human may be
unsure if the platoon will open up a wider gap, which can
be modelled via uncertainties concerning players’ intents.
Incomplete information structures can also predict differentdriving styles [14]. Finally, another challenge is to extend the
model to a platoon interacting with multiple human-driven
vehicles.
REFERENCES
[1]A. A. Alam, A. Gattami, and K. H. Johansson. An experimental
study on the fuel reduction potential of heavy duty vehicle platooning.
In13th International IEEE Conference on Intelligent Transportation
Systems , pages 306–311. IEEE, 2010.
[2]G. Andrew and J. Gao. Scalable training of L1-regularized log-
linear models. In Proceedings of the 24th international conference
on Machine learning , pages 33–40. ACM, 2007.
[3]M. Bahram, A. Lawitzky, J. Friedrichs, M. Aeberhard, and D. Woll-
herr. A Game-Theoretic Approach to Replanning-Aware Interactive
Scene Prediction and Planning. IEEE Transactions on Vehicular
Technology , 65(6):3981–3992, 2016.
[4]T. Bas ¸ar and A. Haurie. Feedback equilibria in differential games with
structural and modal uncertainties. Advances in Large Scale Systems ,
pages 163–301, 1984.
[5]T. Bas ¸ar and G. J. Olsder. Dynamic noncooperative game theory .
SIAM, 1998.
[6]B. Besselink, V . Turri, S. H. van de Hoef, K. Liang, A. Alam,
J. Martensson, and K. H. Johansson. Cyber-physical control of road
freight transport. Proceedings of the IEEE , 104(5):1128–1141, 2016.
[7]M. Buehler, K. Iagnemma, and S. Singh. The DARPA urban challenge:
autonomous vehicles in city trafﬁc , volume 56. Springer, 2009.
[8]E. Chan, P. Gilhead, P. Jelinek, P. Krejci, and T. Robinson. Cooperative
control of SARTRE automated platoon vehicles. In 19th ITS World
Congress , 2012.
[9]S. Deutschle, G. C. Kessler, C. Lank, G. Hoffmann, M. Hakenberg,
and M. Brummer. Use of electronically linked konvoi truck platoons
on motorways. ATZautotechnology , 10(4):20–25, 2010.
[10] J. F. Fisac, E. Bronstein, E. Stefansson, D. Sadigh, S. S. Sastry, and
A. D. Dragan. Hierarchical game-theoretic planning for autonomous
vehicles. In 2019 International Conference on Robotics and Automa-
tion (ICRA) . IEEE, 2019.
[11] D. Lenz, T. Kessler, and A. Knoll. Tactical cooperative planning
for autonomous highway driving using Monte-Carlo Tree Search. In
2016 IEEE Intelligent Vehicles Symposium (IV) , pages 447–453. IEEE,
2016.
[12] J. Markoff. Google cars drive themselves, in trafﬁc. The New York
Times , 10(A1):9, 2010.
[13] D. Sadigh, S. Sastry, S. A. Seshia, and A. D. Dragan. Planning for
autonomous cars that leverage effects on human actions. In Robotics:
Science and Systems , 2016.
[14] D. Sadigh, S. S. Sastry, S. A. Seshia, and A. Dragan. Information
gathering actions over human internal state. In 2016 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS) ,
pages 66–73. IEEE, 2016.
[15] W. Schwarting and P. Pascheka. Recursive conﬂict resolution for
cooperative motion planning in dynamic highway trafﬁc. In 17th
International IEEE Conference on Intelligent Transportation Systems
(ITSC) , pages 1039–1044. IEEE, 2014.
[16] M. Simaan and J. B. Cruz Jr. Additional aspects of the Stackelberg
strategy in non-zero sum games. Journal of Optimization Theory and
Applications , 11(6):613–626, 1973.
[17] E. Stefansson. Hierarchical dynamic games for human-robot interac-
tion with applications to autonomous vehicles. Master’s thesis, KTH
Royal Institute of Technology, 2018.
[18] P. Trautman, J. Ma, R. M. Murray, and A. Krause. Robot navigation
in dense human crowds: Statistical models and experimental studies
of human-robot cooperation. The International Journal of Robotics
Research , 34(3):335–356, 2015.
[19] S. Tsugawa, S. Jeschke, and S. E. Shladover. A review of truck pla-
tooning projects for energy savings. IEEE Transactions on Intelligent
Vehicles , 1(1):68–77, 2016.
[20] S. Tsugawa, S. Kato, and K. Aoki. An automated truck platoon
for energy saving. In 2011 IEEE/RSJ International Conference on
Intelligent Robots and Systems , pages 4109–4114. IEEE, 2011.
[21] P. Varaiya. Smart cars on smart roads: problems of control. IEEE
Transactions on automatic control , 38(2):195–207, 1993.
[22] J. R. Wright and K. Leyton-Brown. Beyond equilibrium: Predicting
human behavior in normal-form games. In AAAI , pages 901–907,
2010.

Elis Stefansson
Jaime F
Fisac
Dorsa Sadigh
S
Shankar Sastry
Karl H
Johansson