Implicitly Assisting Humans to Choose Good Grasps in Robot to Human Handovers.

Implicitly Assisting Humans to Choose
Good Grasps in Robot to Human Handovers
Aaron Bestick, Ruzena Bajcsy, and Anca D. Dragan
Department of Electrical Engineering and Computer Sciences
University of California, Berkeley
Abstract. We focus on selecting handover conﬁgurations that result in
low human ergonomic cost not only at the time of handover, but also
when the human is achieving a goal with the object after that handover.
People take objects using whatever grasping conﬁguration is most com-
fortable to them. When the human has a goal pose they’d like to place
the object at, however, the most comfortable grasping conﬁguration at the
handover might be cumbersome overall, requiring regrasping or the use
of an uncomfortable conﬁguration to reach the goal. We enable robots
to purposefully inﬂuence the choices available to the person when tak-
ing the object, implicitly helping the person avoid suboptimal solutions and
account for the goal. We introduce a probabilistic model of how humans se-
lect grasping conﬁgurations, and use this model to optimize expected cost.
We present results in simulation, as well as from a user study, showing
that the robot successfully inﬂuences people’s grasping conﬁgurations
for the better.
1 Introduction
Handovers happen frequently in collaborative manipulation tasks. Be it when
cooking a meal or assembling a device in a factory workcell, we need to pass
objects to each other in order to work more effectively. As a result, making
robot-to-human handovers seamless has been an area of growing importance
in robotics research [1–10].
Imagine unloading the dishwasher with a robot. The robot comes to give
you a mug so that you can place it in the cupboard. The way the robot presents
you the mug (its position, orientation, and the grasp the robot is already occu-
pying on the object) leads to you having a number of options for how to grasp
it, some demanding more effort than others. In the end, the robot’s choice of
grasp and the object’s pose in SE(3)affects how comfortable the handover is for
you, as well as what you can easily do with the object after the handover: how
easily you can just lay it down in the desired spot in the cupboard.
Naturally, the robot can take this into account when planning its handover.
Prior work has focused on selecting robot grasping conﬁgurations [2,4,6–8,10]
or object handover locations [1, 3, 5, 9] that maximize the number or range of
feasible human grasps [2,7,8] or minimize human ergonomic cost [1,3–6,9,10].
In contrast, our work enables the robot to minimize expected cost: our in-
sight is that, although we can’t control the human’s grasp directly, we can
model the probability that the human will select a particular grasping conﬁgu-
ration. This probability distribution can then be used to evaluate the ergonomic
2 Bestick, Bajcsy, Dragan
𝑇"#$%&'𝑇"#(&)*𝑔,
Fig. 1. Setup & Summary. We focus on ﬁnding robot handover grasps and object trans-
forms that encourage the human to select good grasps, especially when the human has
a next goal for the object. We model how humans select grasping conﬁgurations, and
leverage that model to minimize expected total ergonomic cost to the human.
cost to the person in expectation, accounting for what they are more or less
likely to do.
We propose to model the human as approximately-rational, selecting a
grasping conﬁguration with higher probability if its ergonomic cost is lower.
Having such a model of how the human will select a grasp enables the
robot to inﬂuence the human to select better grasps. In particular, we investi-
gate two implications:
Avoiding suboptimal choices for the human, but only when these choices are
actually likely: The natural alternatives to having a model of how the user
takes the object and minimizing expected cost are either 1) to maximize the
total number of grasping conﬁgurations available to the user and give them
the most ﬂexibility [2], or 2) to minimize average cost to the person [4, 6, 10],
without weighting the choices by the probability that the human will actually
select them.
Compared to the ﬁrst, minimizing expected cost enables the robot to pro-
duce good conﬁgurations as opposed to many conﬁgurations. The second,
minimizing average cost, also achieves that. However, it also tries to avoid al-
lowing high-cost conﬁgurations, because these increase the cost mean. In con-
trast, in our approach, high-cost conﬁgurations do not actually matter, so long
as low-cost conﬁgurations are available, because the human is very unlikely to
select them. Instead, it is suboptimal yet low-cost conﬁgurations that are trou-
blesome — these are the conﬁgurations that the human might select with high
probability, due to the fact that they are not perfectly rational. Our formalism
naturally eliminates such choices for the human to the extent possible, helping
them select the better options.
Encouraging the human to plan ahead: Usually when we receive an object, it
is because we need to do something with it. There is some goal (or set of) goal
pose(s) for the object. However, humans are not always very good at planning
ahead: they might select a grasping conﬁguration comfortable for taking the
object, without thinking of how they will need to manipulate it afterwards.
By modeling the human as approximately rational for the handover stage, but
myopic to the next stage, we enable robots to minimize expected total cost to
the human at both the handover and goal, accounting for this myopia. As a
result, the robot avoids handing over an object in ways that allow for low-cost
grasps which would have high cost at the goal: if a grasp looks tempting to the
Implicitly Assisting Humans to Choose Good Grasps 3
human locally, but would make it difﬁcult for the human to satisfy the goal
afterwards, then the robot will try to hand over the object in a way that makes
such a grasp infeasible.
We tested our approach in simulation and in a user study, suggesting that
the robot can successfully inﬂuence people to take objects in a way that makes
it easier for them to achieve their goal.
2 Related Work
Our main contribution is to explicitly model the probability of the human
choosing different available grasps during handover planning, enabling the
robot to optimize for expected ergonomic cost. A secondary contribution of
our work is accounting for the human’s goal in the context of minimizing
ergonomic cost, enabling the robot to inﬂuence the person to select a better
grasp. Table 2 categorizes related work along three axes: whether the method
accounts for feasibility only or also for ergonomic cost, whether the method
accounts for the human’s goal, and whether the method accounts for positions
of the object only or also grasps.
Table 1. Prior Handover Planning Approaches
Feasibility Only Ergonomic Cost
H Only H + G H Only H + G
Position Only [1], [3], [5], [9]
Grasp Conﬁg. [2] [7], [8] [6], [10], [4] (this paper)
3 Technical Approach
Notation. To choose a handoff conﬁguration, we must select the robot’s grasp
on the object gRand the object pose with respect to the world frame Thand
OWat
which the human will take the object. The object to be handed off allows the
human to grasp it at some set of poses GHSE(3), which we represent as
a Task Space Region [11], and discretize to give a ﬁnite set of feasible human
grasps, so GH,fgH1, ...,gHkg.
Given a handoff grasp and object pose, (gR,Thand
OW), each possible human
grasp gHiwill be reachable with zero or more inverse kinematics (IK) solutions,
which we collect into a set Qhand
gHi.
The union of these sets Qhand
(gR,Thand
OW),SQhand
gHigives all the available “taking”
conﬁgurations available to the human given the robot’s choice of (gR,Thand
OW).
A human grasp gHialso induces IK solutions at the object’s goal pose, Tgoal
OW,
which we collect in a set Qgoal
gHi.
Human Grasp Selection Model. Among possible options, we chose to model
the human ergonomic cost as the distance from some ideal nominal resting
conﬁguration qw.r.t. some metric w:
C(q),q
w|(qq) (1)
4 Bestick, Bajcsy, Dragan
While we chose this cost function for its simplicity, it would be easy to
substitute any other function which maps human limb conﬁgurations to er-
gonomic costs. We would expect superior performance when using cost func-
tions which more accurately capture the human’s preferences.
We model the human as approximately-rational, selecting a grasping con-
ﬁguration qat handover time with higher probability when it has lower cost:
P(q)µelC(q)(2)
P(q)at the time of handover is normalized over all possible grasping conﬁgu-
rations Qhand
(gR,Thand
OW). We can also compute the probability of a grasping conﬁg-
uration given a particular grasp, P(qhandjgH), by normalizing over Qhand
gH, and
P(qgoaljgH)at the goal by normalizing over Qgoal
gH. Finally, we can compute the
probability of a human grasp by summing over all the IK solutions at that
grasp: P(gH) =åq2QhandgHP(q).
Optimization. When the human does not have a (known) goal, we optimize
for expected cost at the handover time:
min
gR,Thand
OWå
q2Qhand
(gR,Thand
OW)P(q)C(q)
(3)
When the human does have a goal, we optimize for expected total cost. The
expected cost at the goal is based on the probability of each grasp based on
what happened at the handover, P(gH), and the probability of each conﬁgura-
tion given that grasp:
min
gR,Thand
OWå
gH2
64å
q2QhandgHP(qjgH)P(gH)C(q) +å
q2Qgoal
gHP(qjgH)P(gH)C(q)3
75 (4)
4 Case Studies
We start with two case studies, highlighting the beneﬁts of our approach: elim-
inating suboptimal yet tempting grasping conﬁgurations.
Expected Cost at Handover Time. Fig. 2 compares optimizing for feasibility,
average cost, and expected cost, in a scenario where the PR2 robot is handing
over a mug to a human. For each case, we take the robot grasp and object trans-
form that arises from the optimization, and compute: 1) the human grasping
conﬁguration of minimum cost; 2) the most “risky” human grasping conﬁgu-
ration, that is not high-cost cost enough to be easily discarded by the human;
3) all human grasping conﬁgurations available; and 4) the histogram of costs
for these conﬁgurations.
We ﬁnd that maximizing the number of feasible options can be dangerous,
because it might mean the expected cost is rather high, and the best conﬁg-
uration is not as good. Compared to minimizing average cost, we ﬁnd that
minimizing expected cost will allow more high-cost conﬁgurations because
there is a very low probability for the human to pick them (marked “unimpor-
tant” on the histogram), but will allow fewer conﬁgurations that have good
cost but not great (marked “problematic” on the histogram). These are conﬁg-
Implicitly Assisting Humans to Choose Good Grasps 5
max|𝑄|minσ𝑞𝐻∈𝑄𝐶(𝑞𝐻)
|𝑄| min𝐸𝑞𝐻∈𝑄[𝐶𝑞𝐻]
min𝐶(𝑞) max𝑃𝑞𝐶𝑞 Q Cost Histogram
greatproblematic
unimportantE[Cost ] = 3.09
E[Cost ] = 1.59
E[Cost ] = 0.33
Fig. 2. Case Study w/o Goal - Increasing Great Choices, Reducing OK Choices, Dis-
regarding Bad Choices. A comparison between maximizing the number of feasible
human grasping conﬁgurations Q(top), minimizing the average ergonomic cost (mid-
dle), and minimizing expected ergonomic cost (bottom), for the case of a single handover
without a known object goal pose. The columns show the most probable human conﬁg-
uration (left), the conﬁguration with the largest contribution to the total cost (middle),
and the full space of conﬁgurations (right). Our method increases the number of great
choices and decreases the number of OK choices which the human might actually pick.
It also keeps bad choices if needed, because they have a low probability of being se-
lected anyway.
urations for which the probability is high enough that the human might pick
them, but they are not as good as the best conﬁgurations.
Experimental Insight 1: A robot that models human handover choices can
make it more likely that the person will actually select a comfortable han-
dover grasp.
Expected Total Cost (Handover + Goal). Fig. 3 compares the three approaches
from above when accounting for the human goal. Feasibility here accounts
for the number of feasible conﬁgurations at both the handover and the goal,
average cost accounts for cost at the start and goal, and so does expected cost.
For each case, we take the resulting robot grasping conﬁguration and compute
1) the human grasping conﬁguration of minimum handover cost, which is
what the human will most likely choose if they are being myopic; 2) given this
grasp, the conﬁguration of minimum cost at the goal (assuming no regrasp);
and 3) the expected cost at the handover and at the goal for each human grasp.
We ﬁnd that maximizing feasible options can lead to very poor options
at the goal. Compared to minimizing average cost, we ﬁnd that minimizing
expected cost is better at eliminating grasps that have low cost at handover
time but only allow for high cost at the goal.
6 Bestick, Bajcsy, Dragan
max|𝑄|minσ𝑞𝐻∈𝑄𝐶(𝑞𝐻)
|𝑄| min𝐸𝑞𝐻∈𝑄[𝐶𝑞𝐻]Handoff Goal
Cost
Human GraspGoal HandoverCost
Human GraspGoal HandoverCost
Human GraspGoal HandoverHuman Grasp Costs
E[Total Cost]
= 9.08
E[Total Cost]
= 6.83
E[Total Cost]
= 5.13
Fig. 3. Case Study w. Goal - Reducing Total Cost. A comparison between maximiz-
ing the number of feasible human grasping conﬁgurations Qat the handover and goal
(top), minimizing the average ergonomic total cost (middle), and minimizing expected
total cost (bottom). The columns show the most probable human conﬁguration at han-
dover time (left), and at the goal (center), along with a plot of cost for each available
grasp to the human. Our method makes it such that the tempting conﬁgurations (low
cost at handover) also have low cost at the goal.
Experimental Insight 2: A robot that models human handover choices can
make it more likely that the person will select a handover grasp that also
allows for comfortably achieving the goal after the handover.
5 Simulation Study
Our case studies used a single object and a single goal conﬁguration. Here we
expand to an experiment that manipulates both as factors.
5.1 Experimental Design
Manipulated Factors. We manipulate three factors. The ﬁrst is the metric we
optimize, as in the case study: maximizing number of feasible options, min-
imizing average cost, or our metric, minimizing expected cost. The second is
theobject being handed over by the robot: a mug as before, a glass, a pitcher,
and a plate, for a total of 4 objects. These objects have vastly different TSR
choices. The third is the goal pose , for which we use 5 different poses. This
leads to a total of 3(metrics) x 4(objects) x 5(goals) = 60 conditions.
Dependent Measures. As in the case studies, we measure expected total cost.
Hypothesis. Our metric is designed to optimize expected total cost (the de-
pendent measure), so we already know it will perform the best. The question
remains whether our metric will be better by a signiﬁcant margin. Our hypoth-
esis is that it will: Our metric will result in a signiﬁcant improvement in expected
cost compared to the baselines.
Implicitly Assisting Humans to Choose Good Grasps 7
12.612
7.906
3.092
 max|Q|  minAvg(c)  min E(C)Average E[Cost]
Fig. 4. Optimal Handover for Different Goal Poses. The different goal poses in our ex-
periment lead to different optimal handover conﬁgurations for the robot, each selected
to minimize expected total cost at the handover time andat that particular goal. The
chart averages the expected total (handoff + goal) ergonomic costs for each of the three
metrics.
5.2 Analysis
We ran an ANOVA with metric as a factor to test differences among the three
metrics across objects and goal poses. We found a signiﬁcant main effect,
F(2, 58) = 1031.07, p<.0001. A post-hoc analysis with Tukey HSD showed
that all three metrics were signiﬁcantly different from each other, with the
average cost outperforming maximum feasibility ( p<.0001) and our metric
outperforming average cost ( p<.001), in line with our hypothesis.
Fig. 4 shows how the robot’s grasping conﬁguration changes as the goal
pose for the human changes. The robot will present the mug so that the per-
son grabs it by the top when it needs to be placed right side up, by the side
when it needs to be placed upside down, etc. In line with our hypothesis, the
expected cost was three times lower with our approach compared to the maxi-
mum feasibility baseline, and two times lower compared to the minimum cost
baseline.
Fig. 5 shows how the robot’s grasping conﬁguration changes, for a given
goal pose, as the object changes. The robot holds the objects in different ways
to ensure that the person can easily grasp them by the side and set them down
vertically with ease.
6 User Study
The previous sections tested our method in simulation, assuming users who
act according to our model. Real people do not. We conducted a user study to
test whether the simulation results generalize, and to explore whether users
perceive the improvement brought about by our method.
6.1 Experimental Design
Manipulated Factors. We manipulated three factors. We manipulated the met-
ricthe robot used to compute its handover conﬁguration, using our metric
based on the user model we proposed, min E[C], and the maximum feasibility
baseline minjQj.
8 Bestick, Bajcsy, Dragan
16.6610.496.39Average E[Cost]max|𝑄|min	𝜇(𝐶)		minΕ[𝐶]
Fig. 5. Optimal Handover for Different Objects. The different objects in our experi-
ment lead to different optimal handover conﬁgurations for the robot for a given goal.
The chart averages the expected total (handoff + goal) ergonomic costs for each of the
three metrics across objects.
We used the mug as the handover object for this experiment, and manipu-
lated the goal pose using 10 different poses. In these poses, the mug was placed
upside down, upright, and to the side to ensure variance.
Finally, we manipulated whether the user knows the goal (Fig. 6). We did this
because we wanted to separate the two assumptions our method is making:
that users select grasping conﬁgurations based on ergonomic cost, and that
users are myopic or greedy in this selection, only accounting for ergonomic
cost at handover time but not at the goal. Therefore, manipulating the user’s
knowledge of the goal enables us to test not only how our method performs
overall (in realistic situations in which users have a goal and are aware of it),
but also whether our method is inﬂuencing the users’ grasp choice in the way
we expected, assuming users are actually myopic (which in reality might or
might not be the case). Altogether, this led to 2(metrics) x 10(goals) x 2(knowl-
edge) = 40 conditions.
Subject Allocation. We recruited 9 users (6 male, 3 female, ages 22-29). All of
the factors were within-subjects , meaning each user experienced all conditions.
We counterbalanced the order of the metrics to avoid order effects, and ran-
domized the order of the goals. We split the experiment in two parts, the ﬁrst
in which the user did not know the goal, and the second in which they did:
In Part 1, the robot handed the object to the person at each of the 20 optimal
handover conﬁgurations (one for each metric and goal pose), but the user was
not told the goal used by the planner. We instructed the user to take the object
from the robot and immediately drop it in a box. This ensured that no notion
of a goal pose would impact the subject’s choice of object grasp. This portion of
the experiment evaluated the two algorithms’ ability to inﬂuence the subjects
to select a particular grasp when the subject was not aware of a goal, i.e. when
the myopic/greediness assumption holds.
In Part 2, a pictoral marker was placed on a table next to the subject indicat-
ing the object’s goal pose during each handoff. The subject was told that two
different algorithms, “Program 1” and “Program 2,” would be used during
this part of the experiment. We conducted handovers at the same 20 conﬁgu-
rations as before, but this time the subject was instructed to place the object
at the indicated goal pose. We told the users before each handover which of
Programs 1 and 2 was in use. This portion of the experiment evaluated the
algorithm’s ability to inﬂuence people to select ergonomically optimal grasps
even when they know the goal, i.e. they are not necessarily myopic. Further-
more, it enabled us to ask users to compare the two methods, seeing if their
Implicitly Assisting Humans to Choose Good Grasps 9
Handover
Part 1: Drop
(No Goal)Part 2: Place
(Specific Goal)
Fig. 6. User Study Setup.
notion of comfort matches ours. If people are actually myopic about the goal
when selecting a grasping conﬁguration, then we expect results for this second
part to match those from the ﬁrst part.
Dependent Measures. We used both objective and subjective measures.
Objective: We annotated for each condition which of the 6 TSRs for the
mug the person selected. From this, we computed expected cost over all IK
solutions at the goal, for all grasps that were feasible at handover time (i.e.
had feasible IK solutions), making 2 assumptions: 1) the person follows our
ergonomic model, and 2) we know the human kinematics:
OM1: E[C(qgoal)],l=10,8qgoal2IK(g),8g2TSR feas. at handover (5)
To alleviate bias in our results induced by the two assumptions, we intro-
duce 3 additional metrics that break each assumption separately as well as
both assumptions together: we break the ﬁrst assumption by computing aver-
age cost (which is the expected cost using a uniform distribution, i.e. l=0)
instead of expected ergonomic cost , and we break the second assumption by
allowing infeasible grasps that a person with different kinematics might have
chosen:
OM2: E[C(qgoal)],l=0,8qgoal2IK(g),8g2TSR feas. at handover (6)
OM3: E[C(qgoal)],l=10,8qgoal2IK(g),8g2TSR (7)
OM4: E[C(qgoal)],l=0,8qgoal2IK(g),8g2TSR (8)
We did not estimate cost at handover time, because we were speciﬁcally
interested in whether the robot successfully inﬂuences users to select grasps
that are good at the goal. Indeed, we might see lower handover time costs for
the baseline condition because it restricts the users less.
Subjective. After each complete experiment, the subject answered a series of
1-7 Likert-scale survey questions about which program they preferred, which
program made their goal easier to accomplish, and which program inspired
the most trust in the robot. These capture each subject’s subjective opinion
about which metric was more effective at making interaction with the robot
comfortable and effective.
Hypotheses.
H1.IF humans are actually myopic when selecting grasping conﬁgurations (e.g.
when they are not even aware of the goal), our method successfully inﬂuences them to
select conﬁgurations with lower cost at the goal compared to the baseline.
10 Bestick, Bajcsy, Dragan
Table 2. Estimated Human Ergonomic Costs at Goal (Part 1: Users not aware of goal)
Objective Measure minjQj min E[C]
E[C(qgoal)],l=10,8qgoal2IK(g),8g2TSR feas. at handover 12.43 6.02
E[C(qgoal)],l=0,8qgoal2IK(g),8g2TSR feas. at handover 12.41 6.30
E[C(qgoal)],l=10,8qgoal2IK(g),8g2TSR 12.18 11.26
E[C(qgoal)],l=0,8qgoal2IK(g),8g2TSR 12.28 11.45
Table 3. Estimated Human Ergonomic Costs at Goal (Part 2: Users aware of the goal)
Objective Measure minjQj min E[C]
E[C(qgoal)],l=10,8qgoal2IK(g),8g2TSR feas. at handover 11.42 5.37
E[C(qgoal)],l=0,8qgoal2IK(g),8g2TSR feas. at handover 11.52 5.61
E[C(qgoal)],l=10,8qgoal2IK(g),8g2TSR 11.72 11.02
E[C(qgoal)],l=0,8qgoal2IK(g),8g2TSR 11.84 11.23
H2.Our method inﬂuences people to select conﬁgurations with lower cost at the
goal compared to the baseline, even when they are aware of the goal.
H3.People prefer to work and are more comfortable with a robot using our method
compared to the baseline.
6.2 Analysis
H1. We used results for part 1 of the study, when users are not aware of the
goal, to test H1. We ﬁrst computed Cronbach’s afor the four objective mea-
sures, which was high at .9036. We thus computed an aggregate goal cost using
all four measures.
We then ran a repeated-measures factorial ANOVA on this aggregate, with
goal and metric as factors. We found a signiﬁcant main effect for metric, as
expected ( F(1, 179 ) =377.83, p<.0001), and a signiﬁcant main effect for goal
(F(9, 171 ) = 26.79, p<.0001). However, there was also a signiﬁcant interac-
tion effect, and so we conducted a Tukey HSD post-hoc, comparing all pairs
but compensating for multiple comparisons. The analysis revealed that the ex-
pected cost (our) metric resulted in signiﬁcantly lower cost at the goal than the
baseline for 7 out of the 10 goals, all with p<.03.
This supports our hypothesis H1, but suggests that the beneﬁt of our
method does depend on the choice of the goal pose, with the maximum feasi-
bility baseline being sufﬁcient for some goals.
Table 2 shows the goal ergonomic costs estimated by each of the four mea-
sures, averaged across all nine study participants for this part of the study.
It shows that pose optimization with min E[C]gives consistently lower er-
gonomic cost at the goal than optimization with min jQj. This difference is
particularly marked for the ﬁrst two measures, which consider only grasps
feasible at the handover. These results suggest that expected ergonomic cost
can be used to inﬂuence humans to choose grasps with good ergonomic prop-
erties even when they are completely unaware of the goal.
H2. For part 2, when users were given speciﬁc goals, our objective measures
again had high item reliability, Cronbach’s a=.8830. We again computed an
Implicitly Assisting Humans to Choose Good Grasps 11
Table 4. Post-Study Survey Results
Statement min jQjmin E[C]t(8) p
“I prefer Program __” 2.0 6.2 9.73 <.0001
“The robot was helpful when running Program __” 3.4 6.4 6.80 <.0001
“I trust the robot running Program __” 3.7 6.1 4.4 <.01
“The robot understood my goal when running
Program __” 2.8 6.4 5.33 <.001
“It was physically easy to do the task when
the robot was running Program __” 2.8 6.2 6.50 <.001
“The robot running Program __ handed me objects
in a way that made the task easier” 2.0 6.3 9.19 <.0001
“If you had to choose a program you prefer,
which would it be?” 0% 100% - -
aggregate cost. We again ran a factorial repeated-measures ANOVA, and the
results, as expected, were analogous to the results from part 1. We again saw
signiﬁcant main effects, but also a signiﬁcant interaction between the factors.
As before, a post-hoc with Tukey HSD corrections showed that 7 out of the 10
goals saw signiﬁcantly lower costs at the goal with our method than with the
baseline. The set of these 7 goals was almost identical to the one in part 1, with
the exception of one goal no longer showing a signiﬁcant difference, and one
goal starting to show a signiﬁcant difference.
This supports out hypothesis H2: our method does not only help users
improve performance when we force them to be myopic by not making them
aware of the goal – it helps in realistic situations, when users have a goal
that they are aware of . This suggests that people are indeed myopic in their
selections of a grasp conﬁguration.
Table 3 shows the goal ergonomic costs estimated by each of the four objec-
tive metrics, averaged across all nine study participants for Part 2 of the study,
where subjects were instructed to place the object on a pictoral marker at the
goal pose after each handover. We see a similar improvement in ergonomic
costs when minimizing E[C]versus maximizing jQj.
Here, we found it interesting that the costs dropped slightly across the
board. This suggests that perhaps when people are aware of the goal they
perform slightly better, but that still our method can signiﬁcantly help them
to further improve their performance.
H3.Table 4 summarizes users’ subjective ratings. t-tests showed that our method
outperformed the baseline in user overall preference, how helpful they thought
the robot was, how much they trusted the robot, and how easy it was to do the
task. Users thought that the robot understood their goal and that it handed
them objects in a way that made their task easier.
The users’ comments were particularly enlightening (here Program 1 refers
to the baseline and Program 2 refers to our method):
“With Program 2, I could move straight from grip to the target with a natural
motion. With Program 1, I would sometimes have to contort my arm unnaturally to
place the mug correctly.”; “Program 1 made it easier to pick up objects but harder to
achieve the goal. Program 2 sometimes made it more difﬁcult to pick up objects but
achieving the goal was easier.”; “Program 1 is an a**hole.”
12 Bestick, Bajcsy, Dragan
7 Discussion
Summary. We introduced a model of how people take an object from the robot,
and used it to select robot actions that lead to better outcomes for the person.
Especially when the person has a goal for the object after the handover, but
they are myopic or greedy in their selection of their grasp and do not account
for the goal, we have shown that the robot can inﬂuence the person’s grasp to
help them achieve better comfort across the task – at the handover time, but
also at the goal time.
Limitations and Future Work. Our work is limited in many ways. We optimize
for total ergonomic cost to the person, but it is not clear what this ergonomic
cost should be, and it will likely differ from human to human. Furthermore,
our study did not measure exactly the ergonomic cost at the goal. Future work
might address this by instrumenting the person and the object. Nonetheless,
we are encouraged by the subjective results, which align well with our objec-
tive estimates. Thus far we only looked at cost at the handover and at the goal,
but not at the trajectory the human would plan from one to the other.
Conclusion. Despite these limitations, we are encouraged to see robots being
able to inﬂuence human actions in a helpful way, making it more likely for
them to ﬁnd good solutions for the task. We are excited to explore further
applications of this idea beyond handovers, to human plans more broadly.
References
1. A. Bestick, S. Burden, G. Willits, N. Naikal, S. S. Sastry, and R. Bajcsy, “Personal-
ized kinematics for human-robot collaborative manipulation,” in IEEE International
Conference on Intelligent Robots and Systems , 2015.
2. M. Cakmak, S. S. Srinivasa, M. K. Lee, J. Forlizzi, and S. Kiesler, “Human prefer-
ences for robot-human hand-over conﬁgurations,” in IEEE International Conference
on Intelligent Robots and Systems , pp. 1986–1993, IEEE, 2011.
3. C.-m. Huang, M. Cakmak, and B. Mutlu, “Adaptive coordination strategies for
human-robot handovers,” in Robotics: Science and Systems (RSS) , 2011.
4. J. Kim, J. Park, Y. Hwang, and M. Lee, “Advanced grasp planning for handover
operation between human and robot: three handover methods in esteem etiquettes
using dual arms and hands of home-service robot,” 2nd International Conference on
Autonomous Robots and Agents , pp. 34–39, 2004.
5. J. Mainprice, E. A. Sisbot, T. Siméon, and R. Alami, “Planning safe and legible
hand-over motions for human-robot interaction,” vol. 2, p. 7, 2010.
6. V . Micelli, K. Strabala, and S. Srinivasa, “Perception and control challenges for
effective human-robot handoffs,” RSS 2011 RGB-D Workshop , 2011.
7. A. H. Quispe, H. B. Amor, and M. Stilman, “Handover planning for every occa-
sion,” IEEE-RSJ International Conference on Humanoid Robots , 2014.
8. A. H. Quispe, H. Ben Amor, H. Christensen, and M. Stilman, “It takes two hands to
clap: towards handovers in bimanual manipulation planning,” in Robotics: Science
and Systems (RSS) , 2014.
9. E. Sisbot and R. Alami, “A human-aware manipulation planner,” IEEE Transactions
on Robotics , vol. 28, pp. 1045–1057, oct 2012.
10. K. Strabala, M. K. Lee, A. Dragan, J. Forlizzi, S. S. Srinavasa, M. Cakmak, and
V . Micelli, “Towards seamless human-robot handovers,” Journal of Human-Robot In-
teraction , vol. 1, no. 1, pp. 112–132, 2012.
11. D. Berenson, S. Srinivasa, and J. Kuffner, “Task Space Regions: A framework for
pose-constrained manipulation planning,” The International Journal of Robotics Re-
search , vol. 30, no. 12, pp. 1435–1460, 2011.


Information Gathering Actions Over Human Internal State.

Information Gathering Actions over Human Internal State
Dorsa Sadigh S. Shankar Sastry Sanjit A. Seshia Anca Dragan
Abstract — Much of estimation of human internal state
(goal, intentions, activities, preferences, etc.) is passive : an
algorithm observes human actions and updates its estimate
of human state. In this work, we embrace the fact that
robot actions affect what humans do, and leverage it to
improve state estimation. We enable robots to do active
information gathering , by planning actions that probe the
user in order to clarify their internal state. For instance, an
autonomous car will plan to nudge into a human driver’s
lane to test their driving style. Results in simulation and
in a user study suggest that active information gathering
signiﬁcantly outperforms passive state estimation.
I.introduction
Imagine driving on the highway. Another driver is
in the lane next to you, and you need to switch lanes.
Some drivers are aggressive and they will never brake
to let you in. Others are more defensive and would
gladly make space for you. You don’t know what kind
of driver this is, so you decide to gently nudge in
towards the other lane to test their reaction. At an
intersection, you might nudge in to test if the other
driver is distracted and they might just let you go
through (Fig.1 bottom left). Our goal in this work is
to give robots the capability to plan such actions as
well.
In general, human behavior is affected by internal
states that a robot would not have direct access to:
intentions, goals, preferences, objectives, driving style,
etc. Work in robotics and perception has focused thus
far on estimating these internal states by providing
algorithms with observations of humans acting, be it
intent prediction [23], [13], [6], [3], [4], [16], Inverse
Reinforcement Learning [1], [12], [15], [22], [18], driver
style prediction [11], affective state prediction [10], or
activity recognition [20].
Human state estimation has also been studied in
the context of human-robot interaction tasks. Here, the
robot’s reward function depends (directly or indirectly)
on the human internal state, e.g., on whether the robot
is able to adapt to the human’s plan or preferences.
Work in assistive teleoperation or in human assistance
has cast this problem as a Partially Observable Markov
Decision Process, in which the robot does observe the
physical state of the world but not the human internal
state — that it has to estimate from human actions. Be-
cause POMDP solvers are not computationally efﬁcient,
the solutions proposed thus far use the current estimate
Department of Electrical Engineering and Computer
Sciences, University of California, Berkeley, Berkeley 94720
dsadigh,sastry,sseshia,dragan@eecs.berkeley.edu
Fig. 1: We enable robots to generate actions that actively probe
humans in order to ﬁnd out their internal state. We apply this to
autonomous driving. In this example, the robot car (yellow) decides
to inch forward in order to test whether the human driver (white)
is attentive. The robot expects drastically different reactions to this
action (bottom right shows attentive driver reaction in light orange,
and distracted driver reaction in dark orange). We conduct a user
study in which we let drivers pay attention or distract them with
cellphones in order to put this state estimation algorithm to the test.
of the internal state to plan (either using the most likely
estimate, or the entire current belief), and adjust this
estimate at every step [8], [7], [5]. Although efﬁcient,
these approximations sacriﬁce an important aspect of
POMDPs: the ability to actively gather information .
Our key insight is that robots can leverage their
own actions to help estimation of human internal
state.
Rather than relying on passive observations, robots can
actually account for the fact that humans will react
to their actions: they can use this knowledge to select
actions that will trigger human reactions which in turn
will clarify the internal state.
We make two contributions:
An Algorithm for Active Information Gathering
over Human Internal State. We introduce an algo-
rithm for planning robot actions that have high ex-
pected information gain. Our algorithm uses a reward-
maximization model of how humans plan their actions
in response to those of the robot’s [19], and leverages
the fact that different human internal states will lead
to different human reactions to speed up estimation.
Fig.1 shows an example of the anticipated difference in
reaction between a distracted and an attentive driver.
Application to Driver Style Estimation. We apply
our algorithm to estimating a human driver’s style
during the interaction of an autonomous vehicle with a
human-driven vehicle. Results in simulation as well as
from a user study suggest that our algorithm’s ability
to leverage robot actions for estimation leads to signiﬁ-
cantly higher accuracy in identifying the correct human
internal state. The autonomous car plans actions like
inching forward at an intersection (Fig.1), nudging into
another car’s lane, or braking slightly in front of a
human-driven car, all to estimate whether the human
driver is attentive.
Overall, we are excited to have taken a step towards
giving robots the ability to actively probe end-users
through their actions in order to better estimate their
goals, preferences, styles, and so on. Even though we
chose driving as our application domain in this pa-
per, our algorithm is general across different domains
and types of human internal state. We anticipate that
applying it in the context of human goal inference
during shared autonomy, for instance, will lead to the
robot purposefully committing to a particular goal in
order to trigger a reaction from the user, either positive
or negative, in order to clarify the desired goal. Of
course, further work is needed in order to evaluate
how acceptant end-users are of different kinds of such
probing actions.
II. Information Gathering Actions
We start with a general formulation of the problem of
a robot needing to maximize its reward by acting in an
environment where a human is also acting. The human
is choosing its actions in a manner that is responsive
to the robot’s actions, and also inﬂuenced by some
internal variables. While most methods that have ad-
dressed such problems have proposed approximations
based on passively estimating the internal variable and
exploiting that estimate, here we propose a method
foractive information gathering that enables the robot
to purposefully take actions that probe the human.
Finally, we discuss our implementation in practice,
which trades off between exploration and exploitation.
A. General Formulation
We deﬁne a human-robot system in which the hu-
man’s actions depend on some human internal state j
that the robot does not directly observe. In a driving
scenario, jmight correspond to driving style : aggressive
or timid, attentive or distracted. In collaborative manip-
ulation scenarios, jmight correspond to the human’s
current goal, or their preference about the task.
We let x2Xbe a continuous physical state of
our system. For our running example of autonomous
cars, this includes position, velocity and heading of the
autonomous and human driven vehicles. Let j2Fbe
the hidden variable, e.g., human driver’s driving style.We assume the robot observes the current physical state
xt, but not the human internal state j.
The robot and human can apply continuous controls
uRand uH. The dynamics of the system evolves as
robot’s and human’s control inputs arrive at each step:
xt+1=fH(fR(xt,ut
R),ut
H). (1)
Here, fRand fHrepresent how the actions of the robot
and of the human respectively affect the dynamics,
and can be applied synchronously or asynchronously.
We assume that while xchanges via (1) based on the
human and robot actions, jdoes not. For instance, we
assume that the human maintains their preferences or
driving style throughout the interaction.
The robot’s reward function in the task depends on
the current state, the robot’s action, as well as the
action that the human takes at that step in response,
rR(xt,ut
R,ut
H).
If the robot has access to the human’s policy
pH(x,uR,j), maximizing robot reward can be modeled
as a POMDP [9] with states (x,j), actions uR, and
reward rR. In this POMDP , the dynamics model can be
computed directly from (1) with ut
H=pH(xt,ut
R,j).
The human’s actions can serve as observations of j
via some P(uHjx,uR). In Sec. II-C we introduce a
model for both pHand P(uHjx,uR,j)based on the
assumption that the human is maximizing their own
reward function.
If we were able to solve the POMDP , the robot would
estimate jbased on the human’s actions, and optimally
trade off between exploiting its current belief over j,
and actively taking information gathering actions meant
to cause human reactions that give the robot a better
estimate of the hidden variable j.
Because POMDPs cannot be solved tractably, several
approximations have been proposed for similar prob-
lem formulations [8], [11], [7]. These approximations
arepassively estimating the human internal state, and
exploiting the belief to plan robot actions.1
In this work, we take the opposite approach: we focus
explicitly on active information gathering. We enable
the robot to decide to actively probe the person to get
a better estimate of j. Our method can be leveraged
in conjunction with exploitation methods, or be used
alone when human state estimation is robot’s primary
objective.
B. Reduction to Information Gathering
At every step, the robot can update its belief over j
via:
bt+1(j)µbt(j)P(uHjxt,uR,j). (2)
To explicitly focus on taking actions to estimate j,
we redeﬁne the robot’s reward function to capture the
1One exception is Nikolaidis et al. [17], who propose to solve the
full POMDP , albeit for discrete and not continuous state and action
spaces.
information gain at every step:
rR(xt,uR,uH) =H(bt)H(bt+1) (3)
with H(b)being the entropy over the belief:
H(b) =åjb(j)log(b(j))
åjb(j). (4)
Optimizing expected reward now entails reasoning
about the effects that the robot actions will have on
what observations the robot will get, i.e., the actions
that the human will take in response, and how use-
ful these observations will be in shattering ambiguity
about j.
C. Solution: Human Model & Model Predictive Control
We solve the information gathering planning prob-
lem via Model Predictive Control (MPC) [14]. At every
time step, we ﬁnd the optimal actions of the robot u
R
by maximizing the expected reward function over a
ﬁnite horizon.
Notation. Let x0be the state at the current time
step, i.e., at the beginning of the horizon. uR=
(u0
R, . . . , uN1
R)be a ﬁnite sequence of the robot’s con-
tinuous actions, and uH= (u0
H, . . . , uN1
H)be a ﬁnite
sequence of human’s continuous actions. Further, let
RR(x0,uR,uH)denote the reward over the ﬁnite hori-
zon, if the agents started in x0and executed uRand
uH, which can be computed via the dynamics in (1).
MPC Maximization Objective. At every time step, the
robot is computing the best actions for the horizon:
u
R=arg max
uREj
RR(x0,uR,uj
H(x0,uR))
(5)
where uj
H(x0,uR)corresponds to the actions the hu-
man would take from state x0over the horizon of
Nsteps if the robot executed actions uR. Here, the
expectation is taken over the current belief over j,b0.
Simplifying (5) using the deﬁnition of reward from
(3), we get:
u
R=arg max
uREj
H(b0)H(bN)
, (6)
u
R=arg max
uREjH(bN)
, (7)
where the expectation remains with respect to b0.
Human Model. We assume that the human maxi-
mizes their own reward function at every step. We let
rj
H(xt,ut
R,ut
H)represent human’s reward function at
time t, which is parametrized by the human internal
state j. Then, the sum of human rewards over horizon
Nis:
Rj
H(x0,uR,uH) =N1
å
t=0rj
H(xt,ut
R,ut
H) (8)Building on our previous work [19], which showed
how the robot can plan using such a reward func-
tion when there are no hidden variables, we compute
uj
H(x0,uR)through an approximation. We model the
human as having access to uRa priori, and compute
the ﬁnite horizon human actions that maximize the
human’s reward:
uj
H(x0,uR) =arg max
uHRj
H(x0,uR,uH) (9)
One can ﬁnd rj
Hthrough Inverse Reinforcement
Learning (IRL) [1], [12], [15], [22], by getting demon-
strations of human behavior associated with a direct
measurement of j.
The robot can use this reward in the dynamics model
in order to compute human actions via (9). To update
the belief band compute expected reward in (7), we
still need an observation model. We assume that ac-
tions with lower reward are exponentially less likely,
building on the principle of maximum entropy [22]:
P(uHjx,uR,j)µexp(jH(x,uR,uH)) (10)
Optimization Procedure. To solve (5) (or equivalently
(7)), we use a gradient descent optimization method,
L-BFGS, designed for unconstrained nonlinear prob-
lems [2]. Therefore, we would like to ﬁnd the gradient
of the objective in equation (5) with respect to uR.
Since the objective is the expectation of RR, we can
reformulate this gradient as:
¶Ej
RR(x0,uR,uj
H(x0,uR))
¶uR
=å
j¶RR(x0,uR,uj
H(x0,uR))
¶uRb0(j)(11)
Then, we only need to ﬁnd¶RR
¶uR, which is equivalent
to:
¶RR(x0,uR,uj
H(x0,uR))
¶uR=
¶RR(x0,uR,uH)
¶uH¶uj
H
¶uR+¶RR(x0,uR,uH)
¶uRjuH=uj
H(x0,uR)
(12)
Because RR, as indicated by (7), simpliﬁes to the
negative entropy of the updated belief, we can com-
pute both¶RR(x0,uR,uH)
¶uHand¶RR(x0,uR,uH)
¶uRjuH=uj
H(x0,uR)
symbolically.
This leaves¶uj
H
¶uR. We use the fact that the gradient of
RHwill evaluate to zero at uj
H:
¶RH
¶uH(x0,uR,uj
H(x0,uR)) = 0 (13)
Now, differentiating this expression with respect to uR
will result in:
¶2RH
¶u2
H¶uj
H
¶uR+¶2RH
¶uH¶uR¶uR
¶uR=0 (14)
Then, solving for¶uj
H
¶uRenables us to ﬁnd the following
symbolic expression:
¶uj
H
¶uR= [¶2RH
¶uH¶uR][¶2RH
¶u2
H]1. (15)
This expression allows ﬁnding a symbolic expression
for the gradient in equation (11).
D. Explore-Exploit Trade-Off
In practice, we use information gathering in conjunc-
tion with exploitation. We do not solely optimize the
reward from Sec. II-B, but optimize it in conjunction
with the robot’s actual reward function assuming the
current estimate of j:
rR(xt,uR,uH) =H(bt)H(bt+1) (16)
+lrgoal(xt,uR,uH,bt)
At the very least, we do this as a measure of safety,
e.g., we want an autonomous car to keep avoiding colli-
sions even when it is actively probing a human driver to
test their reactions. We choose lexperimentally, though
existing techniques that can better adapt lover time
[21].
Despite optimizing this trade-off, we do not claim
that our method as-is can better solve the general
POMDP formulation from Sec. II-A: only that it can
be used to get better estimates of human internal state.
The next sections test this in simulation and in practice,
in a user study, and future work will look at how
to leverage this ability to better solve human-robot
interaction problems.
III. S imulation Results
In this section, we show simulation results that use
the method from the previous section to estimate hu-
man driver type in the interaction between an au-
tonomous vehicle and a human-driven vehicle.
In this section, we consider three different au-
tonomous driving scenarios. In these scenarios, the
human is either distracted or attentive during different
driving experiments. The scenarios are shown in Fig.2,
where the yellow car is the autonomous vehicle, and
the white car is the human driven vehicle. Our goal is
to plan to actively estimate the human’s driving style in
each one of these scenarios, by using the robot’s actions.
A. Attentive vs. Distracted Human Driver Models
Our technique requires reward functions rj
Hthat
model the human behavior for a particular internal
state j. We obtain a generic driver model via Con-
tinuous Inverse Optimal Control with Locally Opti-
mal Examples [12] from demonstrated trajectories in
a driving simulator in an environment with multiple
autonomous cars, which followed precomputed routes.
We parametrize the human reward function as a
linear combination of features, and learn weights onthe features. We use various features including features
for bounds on the control inputs, features that keep
the vehicles within the road boundaries and close to
the center of their lanes. Further, we use quadratic
functions of speed to capture reaching the goal, and
Gaussians around other vehicles on the road to enforce
collision avoidance as part of the feature set.
We then adjust the learned weights to model atten-
tive vs. distractive drivers. Speciﬁcally, we modify the
weights of the collision avoidance features, so the dis-
tracted human model has less weight for these features.
Therefore, the distracted driver is more likely to collide
with the other cars while the attentive driver has high
weights for the collision avoidance feature.
B. Manipulated Factors
We manipulate the reward function that the robot is
optimizing. In the passive condition, the robot opti-
mizes a simple reward function for collision avoidance
based on the current belief estimate. It then updates
this belief passively, by observing the outcomes of its
actions at every time step. In the active condition,
the robot trades off between this reward function and
the Information Gain from (3) in order to explore the
human’s driving style.
We also manipulate the human internal state to be
attentive ordistracted . The human is simulated to
follow the ideal model of reward maximization for our
two rewards.
C. Driving Simulator
We use a simple point-mass model for the dynamics
of the vehicle, where x=
x y qv>is the state
of the vehicle. Here, xand yare the coordinates of
the vehicle, qis the heading, and vis the speed. Each
vehicle has two control inputs u=
u1u2>, where
u1is the steering input, and u2is acceleration. Further,
we let abe a friction coefﬁcient. Then, the dynamics of
each vehicle is formalized as:
[˙x˙y˙q˙v] = [ vcos(q)vsin(q)vu1u2av].
(17)
D. Scenarios and Qualitative Results
Scenario 1: Nudging In to Explore on a Highway.
In this scenario, we show an autonomous vehicle ac-
tively exploring the human’s driving style in a high-
way driving setting. We contrast the two conditions
in Fig.2(a). In the passive condition, the autonomous
car drives on its own lane without interfering with
the human throughout the experiment, and updates
its belief based on passive observations gathered from
the human car. However, in the active condition, the
autonomous car actively probes the human by nudging into
her lane in order to infer her driving style. An attentive
human signiﬁcantly slows down (timid driver) or speeds up
(aggressive driver) to avoid the vehicle, while a distracted
Autonomous VehicleHuman Driven VehiclePassive EstimationActive Info GatheringSpeed: 1.00Speed: 1.00Speed: 1.00Speed: 0.64
Speed: 0.45Speed: 0.50Speed: 0.58
Speed: 0.01(a) Scenario 1: Nudging in to Explore on a Highway(b) Scenario 2: Braking to Explore on a Highway(c) Scenario 3: Nudging in to Explore at an Intersection
Speed: 1.00
Speed: 1.00
Speed: 0.67
Speed: 1.00
Speed: 0.64
Speed: 0.45
Speed: 0.50
Speed: 0.58
Speed: 0.01
Fig. 2: Our three scenarios, along with a comparison of robot plans for passive estimation (gray) vs active information gathering (orange). In
the active condition, the robot is purposefully nudging in or braking to test human driver attentiveness. The color of the autonomous car in
the initial state is yellow, but changes to either gray or orange in cases of passive and active information gathering respectively.
driver might not realize the autonomous actions and main-
tain their velocity, getting closer to the autonomous vehicle .
It is this difference in reactions that enables the robot
to better estimate j.
Scenario 2: Braking to Explore on a Highway. In
the second scenario, we show the driving style can be
explored by the autonomous car probing the human
driver behind it. The two vehicles start in the same
lane as shown in Fig.2(b), where the autonomous car is
in the front. In the passive condition, the autonomous
car drives straight without exploring or enforcing any
interactions with the human driven vehicle. In the active
condition, the robot slows down to actively probe the human
and ﬁnd out her driving style. An attentive human would
slow down and avoid collisions while a distracted human
will have a harder time to keep safe distance between the
two cars.
Scenario 3: Nudging In to Explore at an Intersec-
tion. In this scenario, we consider the two vehicles
at an intersection, where the autonomous car actively
tries to explore human’s driving style by nudging into
the intersection. The initial conditions of the vehicles
are shown in Fig.2(c). In the passive condition, the
autonomous car stays at its position without probing
the human, and only optimizes for collision avoidance.
This provides limited observations from the human
car resulting in a low conﬁdence belief distribution.
In the active condition, the autonomous car nudges into
the intersection to probe the driving style of the human.
An attentive human would slow down to stay safe at the
intersection while a distracted human will not slow down.
E. Quantitative Results
Throughout the remainder of the paper, we use a
common color scheme to plot results for our exper-
imental conditions. We show this common scheme
in Fig.3: darker colors (black and red) correspond to
Active RobotPassive RobotAttentive HumanDistracted HumanReal User (solid line)Ideal User Model (dotted line)Real User (solid line)Ideal User Model (dotted line)Real User (solid line)Ideal User Model (dotted line)Real User (solid line)Ideal User Model (dotted line)Fig. 3: Legends indicating active/passive robots, attentive/distracted
humans, and real user/ideal model used for Fig.4, Fig. ??, Fig.5, and
Fig.6.
attentive humans, and lighter colors (gray and orange)
correspond to distracted humans. Further, the shades
of orange correspond to active information gathering,
while the shades of gray indicate passive information
gathering. We also use solid lines for real users, and
dotted lines for scenarios with an ideal user model
learned through inverse reinforcement learning. This
table is representative for the legends of Fig.4, Fig. ??,
Fig.5, and Fig.6.
Fig.4 plots, using dotted lines, the beliefs over time
for the attentive (left) and distracted (right) condi-
tions, comparing in each the passive (dotted black
and gray respectively) with the active method (dotted
dark orange and light orange respectively). In every
situation, the active method achieves a more accurate
belief (higher values for attentive on the left, when
the true jis attentive, and lower values on the right,
when the true jis distracted). In fact, passive estima-
tion sometimes incorrectly classiﬁes drivers as attentive
when they are distracted and vice-versa.
The same ﬁgure also shows (in solid lines) results
from our user study of what happens when the robot
no longer interacts with an ideal model. We discuss
these in the next section.
Fig.5 and Fig.6 plot the corresponding robot and
time
(a) Scenario 1. Attentive0.00.20.40.60.81.0b(ϕ=attentive)
time
(b) Scenario 1. Distracted0.00.20.40.60.81.0b(ϕ=attentive)
time
(c) Scenario 2. Attentive0.00.20.40.60.81.0b(ϕ=attentive)
time
(d) Scenario 2. Distracted0.00.20.40.60.81.0b(ϕ=attentive)
time
(e) Scenario 3. Attentive0.00.20.40.60.81.0b(ϕ=attentive)
time
(f) Scenario 3. Distracted0.00.20.40.60.81.0b(ϕ=attentive)Fig. 4: The probability that the robot assigns to attentive as a function
of time, for the attentive (left) and distracted (right). Each plot
compares the active algorithm to passive estimation, showing that
active information gathering leads to more accurate state estimation,
in simulation and with real users.
human trajectories for each scenario. The important
takeaway from these ﬁgures is that there tends to be
a larger gap between attentive and distracted human
trajectories in the active condition (orange shades) than
in the passive condition (gray shades), especially in
scenarios 2 and 3. It is this difference that helps the
robot better estimate j:the robot in the active condition
is purposefully choosing actions that will lead to large
differences in human reactions , in order to more easily
determine the human driving style.
IV . U serStudy
In the previous section, we explored planning for
an autonomous vehicle that actively probes a human’s
driving style, by braking or nudging in and expecting
to cause reactions from the human driver that would
be different depending on their style. We showed that
active exploration does signiﬁcantly better at distin-
guishing between attentive and distracted drivers using
simulated (ideal) models of drivers. Here, we show
the results of a user study that evaluates this active
exploration for attentive and distracted human drivers.A. Experimental Design
We use the same three scenarios discussed in the
previous section.
Manipulated Factors. We manipulated the same two
factors as in our simulation experiments: the reward
function that the robot is optimizing (whether it is
optimizing its reward through passive state estimation,
or whether it is trading off with active information
gathering), and the human internal state (whether the
user is attentive or distracted). We asked our users to
pay attention to the road and avoid collisions for the
attentive case, and asked our users to play a game on
a mobile phone during the distracted driving experi-
ments.
Dependent Measure. We measured the probability that
the robot assigned along the way to the human internal
state.
Hypothesis. The active condition will lead to more accu-
rate human internal state estimation, regardless of the true
human internal state.
Subject Allocation. We recruited 8 participants (2 fe-
male, 6 male) in the age range of 21-26 years old.
All participants owned a valid driver license and had
at least 2 years of driving experience. We ran the
experiments using a 2D driving simulator with the
steering input and acceleration input provided through
a steering wheel and a pedals as shown in Fig.1. We
used a within-subject experiment design with counter-
balanced ordering of the four conditions.
B. Analysis
We ran a factorial repeated-measures ANOVA on
the probability assigned to “attentive”, using reward
(active vs passive) and human internal state (attentive
vs distracted) as factors, and time and scenario as
covariates. As a manipulation check, attentive drivers
had signiﬁcantly higher estimated probability of “at-
tentive” associated than distracted drivers (.66 vs .34,
F=3080.3, p<.0001). More importantly, there
was a signiﬁant interaction effect between the factors
(F=1444.8, p<.000). We ran a post-hoc analysis
with Tukey HSD corrections for multiple comparisons,
which showed all four conditions to be signiﬁcantly
different from each other, all contrasts with p<.0001.
In particular, the active information gathering did end
up with higher probability mass on “attentive” than
the passive estimation for the attentive users, and lower
probability mass for the distracted user. This supports
our hypothesis that our method works, and active in-
formation gathering is better at identifying the correct
state.
Fig.4 compares passive (grays and blacks) and ac-
tive (light and dark oranges) across scenarios and for
attentive (left) and distracted (right) users. It plots
the probability of attentive over time, and the shaded
regions correspond to standard error. From the ﬁrst
column, we can see that our algorithm in all cases
(a)Scenario 1: R nudges inxtimetimexspeedy(b) Scenario 2: R brakes(c) Scenario 3: R inches forwardFig. 5: Robot trajectories for each scenario in the active information
gathering condition. The robot acts differently when the human
is attentive (dark orange) vs. when the human is distracted (light
orange) due to the trade-off with safety.
detects human’s attentiveness with much higher prob-
ably than the passive information gathering technique
shown in black. From the second column, we see that
our algorithm places signiﬁcantly lower probability on
attentiveness, which is correct because those users were
distracted users. These are in line with the statistical
analysis, with active information gathering doing a
better job estimating the true human internal state.
Fig.5 plots the robot trajectories for the active in-
formation gathering setting. Similar to Fig.4, the solid
lines are the mean of robot trajectories and the shaded
regions show the standard error. We plot a representa-
tive dimension of the robot trajectory (like position or
speed) for attentive (dark orange) or distracted (light
orange) cases. The active robot probed the user, but
ended up taking different actions when the user was
attentive vs. distracted in order to maintain safety. For
example, in Scenario 1, the trajectories show the robot
is nudging into the human’s lane, but the robot decides
to move back to its own lane when the human drivers
are distracted (light orange) in order to stay safe. In
Scenario 2, the robot brakes in front of the human, but
it brakes less when the human is distracted. Finally, in
Scenario 3, the robot inches forward, but again it stops
when if the human is distracted, and even backs up to
make space for her.
Fig.6 plots the user trajectories for both active infor-
mation gathering (ﬁrst row) and passive information
gathering (second row) conditions. We compare the
reactions of distracted (light shades) and attentive (dark
shades) users. There are large differences directly ob-
servable, with user reactions tending to indeed cluster
according to their internal state. These differences are
much smaller in the passive case (second row, where
distracted is light gray and attentive is black). For
example, in Scenario 1 and 2, the attentive users (dark
orange) keep a larger distance to the car that nudges
in front of them or brakes in front of them, while
the distracted drivers (light orange) tend to keep a
smaller distance. In Scenario 3, the attentive drivers
tend to slow down and do not cross the intersec-
tion, when the robot actively inches forward. None of
these behaviors can be detected clearly in the passiveinformation gathering case (second row). This is the
core advantage of active information gathering: the
actions are purposefully selected by the robot such that
users would behave drastically differently depending
on their internal state, clarifying to the robot what this
state actually is.
Overall, these results support our simulation ﬁnd-
ings, that our algorithm performs better at estimating
the true human internal state by leveraging purposeful
information gathering actions.
V .discussion
Summary. In this paper, we formalized the problem of
active information gathering between robot and human
agents, where the robot plans to actively explore and
gather information about the human’s internal state
by leveraging the effects of its actions on the human
actions. The generated strategy for the robot actively
probes the human by taking actions that impact the
human’s action in such a way that they reveal her
internal state. The robot generates strategies for in-
teraction that we would normally need to hand-craft,
like inching forward at a 4-way stop. We evaluated
our method in simulation and through a user study
for various autonomous driving scenarios. Our results
suggest that robots are indeed able to construct a more
accurate belief over the human’s driving style with
active exploration than with passive estimation.
Limitations and Future Work. Our work is limited in
many ways. First, state estimation is not the end goal,
and ﬁnding how to trade off exploration and exploita-
tion is still a challenge. Second, our optimization is
close to real-time, but higher computational efﬁciency
is still needed. Further, our work relies on a model
(reward function) of the human for each j, which
might be difﬁcult to acquire, and might not be accurate.
Thus far, we have assumed a static j, but in reality
jmight change over time (e.g. the human adapts her
preferences), or might even be inﬂuenced by the robot
(e.g. a defensive driver becomes more aggressive when
the robot probes her).
We also have not tested the users’ acceptance of
information gathering actions. Although these actions
are useful, people might not always react positively to
being probed.
Last but not least, exploring safely will be of crucial
importance.
Conclusion. We are encouraged by the fact that
robots can generate useful behavior for interaction
autonomously, and are excited to explore information-
gathering actions on human state further, including
beyond autonomous driving scenarios.
Acknowledgements
This work was partially supported by Berkeley
DeepDrive, NSF grants CCF-1139138 and CCF-1116993,
ONR N00014-09-1-0230, and an NDSEG Fellowship.
HumanActive RobotPassive Robot
HumanActive RobotPassive Robot
HumanActive RobotPassive Robot
stops to let R throughFig. 6: The user trajectories for each scenario. The gap between attentive and distracted drivers’ actions is clear in the active information
gathering case (ﬁrst row).
References
[1] P . Abbeel and A. Y. Ng. Exploration and apprenticeship learning
in reinforcement learning. In Proceedings of the 22nd international
conference on Machine learning , pages 1–8. ACM, 2005.
[13] M. Liebner, M. Baumann, F. Klanner, and C. Stiller. Driver intent
[2] G. Andrew and J. Gao. Scalable training of l 1-regularized log-
linear models. In Proceedings of the 24th international conference
on Machine learning , pages 33–40. ACM, 2007.
[3] M. Awais and D. Henrich. Human-robot collaboration by
intention recognition using probabilistic state machines. In
Robotics in Alpe-Adria-Danube Region (RAAD), 2010 IEEE 19th
International Workshop on , pages 75–80. IEEE, 2010.
[4] C. L. Baker, R. Saxe, and J. B. Tenenbaum. Action understanding
as inverse planning. Cognition , 113(3):329–349, 2009.
[5] T. Bandyopadhyay, K. S. Won, E. Frazzoli, D. Hsu, W. S. Lee,
and D. Rus. Intention-aware motion planning. In Algorithmic
Foundations of Robotics X , pages 475–491. Springer, 2013.
[6] A. D. Dragan and S. S. Srinivasa. Formalizing assistive teleopera-
tion. MIT Press, July, 2012.
[7] A. Fern, S. Natarajan, K. Judah, and P . Tadepalli. A decision-
theoretic model of assistance. In IJCAI , pages 1879–1884, 2007.
[8] S. Javdani, J. A. Bagnell, and S. Srinivasa. Shared autonomy via
hindsight optimization. arXiv preprint arXiv:1503.07619 , 2015.
[9] L. P . Kaelbling, M. L. Littman, and A. R. Cassandra. Planning
and acting in partially observable stochastic domains. Artiﬁcial
intelligence , 101(1):99–134, 1998.
[10] D. Kulic and E. A. Croft. Affective state estimation for human–
robot interaction. Robotics, IEEE Transactions on , 23(5):991–1000,
2007.
[11] C.-P . Lam, A. Y. Yang, and S. S. Sastry. An efﬁcient algorithm for
discrete-time hidden mode stochastic hybrid systems. In Control
Conference (ECC), 2015 European , pages 1212–1218. IEEE, 2015.
[12] S. Levine and V . Koltun. Continuous inverse optimal control
with locally optimal examples. arXiv preprint arXiv:1206.4617 ,
2012.inference at urban intersections using the intelligent driver
model. In Intelligent Vehicles Symposium (IV), 2012 IEEE , pages
1162–1167. IEEE, 2012.
[14] M. Morari, C. Garcia, J. Lee, and D. Prett. Model predictive control .
Prentice Hall Englewood Cliffs, NJ, 1993.
[15] A. Y. Ng, S. J. Russell, et al. Algorithms for inverse reinforcement
learning. In Proceedings of the 17th international conference on
Machine learning , pages 663–670, 2000.
[16] T.-H. D. Nguyen, D. Hsu, W.-S. Lee, T.-Y. Leong, L. P . Kael-
bling, T. Lozano-Perez, and A. H. Grant. Capir: Collabora-
tive action planning with intention recognition. arXiv preprint
arXiv:1206.5928 , 2012.
[17] S. Nikolaidis, A. Kuznetsov, D. Hsu, and S. Srinivasa. Formal-
izing human-robot mutual adaptation via a bounded memory
based model. In Human-Robot Interaction , March 2016.
[18] D. Ramachandran and E. Amir. Bayesian inverse reinforcement
learning. Urbana , 51:61801, 2007.
[19] D. Sadigh, S. S. Sastry, S. A. Seshia, and A. Dragan. Anonymous
title.
[20] T. Van Kasteren, A. Noulas, G. Englebienne, and B. Kröse.
Accurate activity recognition in a home setting. In Proceedings
of the 10th international conference on Ubiquitous computing , pages
1–9. ACM, 2008.
[21] H. P . Vanchinathan, I. Nikolic, F. De Bona, and A. Krause.
Explore-exploit in top-n recommender systems via gaussian
processes. In Proceedings of the 8th ACM Conference on Recom-
mender systems , pages 225–232. ACM, 2014.
[22] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum
entropy inverse reinforcement learning. In AAAI , pages 1433–
1438, 2008.
[23] B. D. Ziebart, N. Ratliff, G. Gallagher, C. Mertz, K. Peterson, J. A.
Bagnell, M. Hebert, A. K. Dey, and S. Srinivasa. Planning-based
prediction for pedestrians. In Intelligent Robots and Systems, 2009.
IROS 2009. IEEE/RSJ International Conference on , pages 3931–
3936. IEEE, 2009.

Dorsa Sadigh
S
Shankar Sastry
Sanjit A
Seshia
Anca Dragan