Preference learning along multiple criteria: A game-theoretic perspective.

Preference learning along multiple criteria:
A game-theoretic perspective
Kush Bhatia
EECS, UC Berkeley
kush@cs.berkeley.eduAshwin Pananjady
Simons Institute, UC Berkeley
ashwinpm@berkeley.edu
Peter L. Bartlett
EECS and Statistics, UC Berkeley
peter@berkeley.eduAnca D. Dragan
EECS, UC Berkeley
anca@berkeley.eduMartin J. Wainwright
EECS and Statistics, UC Berkeley
wainwrig@berkeley.edu
Abstract
The literature on ranking from ordinal data is vast, and there are several ways to
aggregate overall preferences from pairwise comparisons between objects. In par-
ticular, it is well-known that any Nash equilibrium of the zero-sum game induced
by the preference matrix deﬁnes a natural solution concept (winning distribution
over objects) known as a von Neumann winner. Many real-world problems, how-
ever, are inevitably multi-criteria, with different pairwise preferences governing the
different criteria. In this work, we generalize the notion of a von Neumann winner
to the multi-criteria setting by taking inspiration from Blackwell’s approachability.
Our framework allows for non-linear aggregation of preferences across criteria, and
generalizes the linearization-based approach from multi-objective optimization.
From a theoretical standpoint, we show that the Blackwell winner of a multi-criteria
problem instance can be computed as the solution to a convex optimization problem.
Furthermore, given random samples of pairwise comparisons, we show that a
simple, “plug-in” estimator achieves (near-)optimal minimax sample complexity.
Finally, we showcase the practical utility of our framework in a user study on
autonomous driving, where we ﬁnd that the Blackwell winner outperforms the von
Neumann winner for the overall preferences.
1 Introduction
Economists, social scientists, engineers, and computer scientists have long studied models for
human preferences, under the broad umbrella of social choice theory [10, 7]. Learning from human
preferences has found applications in interactive robotics for learning reward functions [ 45,39],
in medical domains for personalizing assistive devices [ 59,9], and in recommender systems for
optimizing search engines [ 15,28]. The recent focus on safety in AI has popularized human-in-the-
loop learning methods that use human preferences in order to promote value alignment [16, 46, 6].
The most popular form of preference elicitation is to make pairwise comparisons [ 51,13,33].
Eliciting such feedback involves showing users a pair of objects and asking them a query: Do you
prefer object A or object B? Depending on the application, an object could correspond to a product in
a search query, or a policy or reward function in reinforcement learning. A vast body of classical work
dating back to Condorcet and Borda [ 17,12] has focused on deﬁning and producing a “winning"
object from the result of a set of pairwise comparisons.
In relatively recent work, Dudik et al. [ 22] proposed the concept of a von Neumann winner, corre-
sponding to a distribution over objects that beats or ties every other object in the collection. They
showed that under an expected utility assumption, such a randomized winner always exists and
34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.
(a) (b)
Figure 1. (a) Policy A focuses on optimizing comfort and policy B on speed, and these are compared
pairwise in different environments. (b) Preference matrices, where entry (i; j)of the matrix contains the
proportion of comparisons between the pair (i; j)that are won by object i. (The diagonals are set to
half by convention). The overall pairwise comparisons are given by the matrix POverall
ex , and preferences
along each of the criteria by matrices PComfort
ex andPSpeed
ex (the numbers here are illustrative of our
user-study in Section 4). Policy R is a randomized policy1=2A+1=2B. While the preference matrices
satisfy the linearity assumption individually along speed and comfort, the assumption is violated overall,
wherein R is preferred over both A and B.
overcomes limitations of existing winning concepts—the Condorcet winner does not always exist,
while the Borda winner fails an independence of clones test [ 47]. However, the assumption of
expected utility relies on a strong hypothesis about how humans evaluate distributions over objects: it
posits that the probability with which any distribution over objects beats an object is linear in .
Consequences of assuming linearity: In order to better appreciate these consequences, consider
as an example the task of deciding between two policies (say A and B) to deploy in an autonomous
vehicle. Suppose that these policies have been obtained by optimizing two different objectives, with
policy A optimized for comfort and policy B optimized for speed. Figure 1(a) shows a snapshot of
these two policies. When compared overall, 60% of the people preferred Policy A over B – making it
the von Neumann winner. The linearity assumption then posits that a randomized policy that mixes
between A and B can never be better than both A and B; but we see that the Policy R = 1=2A+1=2B
is actually preferred by a majority over both A and B! Why is the linearity assumption violated here?
One possible explanation for such a violation is that the comparison problem is actually multi-criteria
in nature. If we look at the preferences for the criterion speed and comfort individually in Figure 1(b),
we see that Policy A does quite poorly on the speed axis while B lags behind in comfort. In contrast,
Policy R does acceptably well along both the criteria and hence is preferred overall to both Policies A
and B. It is indeed impossible to come to this conclusion by only observing the overall comparisons.
This observation forms the basis of our main proposal: decompose the single overall comparison and
ask humans to provide preferences along simpler criteria. This decomposition of the comparison
task allows us to place structural assumptions on comparisons along each criterion. For instance,
we may now posit the linearity assumption along each criterion separately rather than on the overall
comparison task. In addition to allowing for simpliﬁed assumptions, breaking up the task into such
simpler comparisons allows us to obtain richer and more accurate feedback as compared to the single
overall comparison. Indeed, such a motivation for eliciting simpler feedback from humans ﬁnds its
roots in the the study of cognitive biases in decision making, which suggests that the human mind
resorts to simple heuristics when faced with a complicated questions [53].
Contributions: In this paper, we formalize these insights and propose a new framework for
preference learning when pairwise comparisons are available along multiple, possibly conﬂicting,
criteria. As shown by our example in Figure 1, a single distribution which is the von Neumann
winner along every criteria might not exist. To counter this, we formulate the problem of ﬁnding
the “best” randomized policy by drawing on tools from the literature on vector valued pay-offs in
game theory. Speciﬁcally, we take inspiration from Blackwell’s approachability [ 11] and introduce
the notion of a Blackwell winner. This solution concept strictly generalizes the concept of a von
Neumann winner, and recovers the latter when there is only a single criterion present. Section 2
describes this framework in detail, and Section 3 collects our statistical and computational guarantees
for learning the Blackwell winner from data. Section 4 describes a user study with an autonomous
driving environment, in which we ask human subjects to compare self-driving policies along multiple
2
criteria such as safety, aggressiveness, and conservativeness. Our experiment demonstrates that the
Blackwell winner is able to better trade off utility along these criteria and produces randomized
policies that outperform the von Neumann winner for the overall preferences.
Related work. Most closely related to our work is the ﬁeld of computational social choice, which
has focused on deﬁning notions of winners from overall pairwise comparisons (see the survey [ 37] for
a review). Amongst them, three deterministic notions of a winner—the Condorcet [ 17], Borda [ 12],
and Copeland [ 18] winners—have been widely studied. In addition, Dudik et al. [ 22] recently
introduced the notion of a (randomized) von Neumann winner. Starting with the work of Yue et al. [ 57],
there have been several research papers studying an online version of preference learning, called the
Dueling Bandits problem. Algorithms have been proposed to compete with Condorcet [ 60,62,4],
Copeland [61, 56], Borda [30] and von Neumann [22] winners.
The theoretical foundations of decision making based on multiple criteria have been widely studied
within the operations research community . This sub-ﬁeld—called multiple-criteria decision analysis—
has focused largely on scoring, classiﬁcation, and sorting based on multiple-criteria feedback. See
the surveys [ 44,63] for thorough overviews of existing methods and their associated guarantees. The
problem of eliciting the user’s relative weighting of the various criteria has also been considered [ 20].
However, relatively less attention has been paid to the study of randomized decisions and statistical
inference, both of which form the focus of our work. From an applied perspective, the combination
of multi-criteria assessments has received attention in disparate ﬁelds such as psychometrics [ 40,35],
healthcare [ 50], and recidivism prediction [ 55]. In many of these cases, a variety of approaches—both
linear and non-linear—have been empirically evaluated [ 19]. Justiﬁcation for non-linear aggregation
of scores along the criteria has a long history in psychology and the behavioral sciences [ 27,24,54].
In the game theory literature, Blackwell [ 11] introduced the notion of approachability as a general-
ization of a zero-sum game with vector-valued payoffs (for a detailed discussion see Appendix A).
Blackwell’s approachability and its connections with no-regret learning and calibrated forecasting
have been extensively studied [ 1,42,34]. These connections have enabled applications of Blackwell’s
results to problems ranging from constrained reinforcement learning [ 36] to uncertainty estimation
for question-answering tasks [ 31]. In contrast, our framework for preference learning along multiple
criteria deals with a single shot game and uses the idea of the target set to deﬁne the concept of a
Blackwell winner. Another body of literature related to our work studies Nash equilibria in games
with perturbed payoffs, under both robust [ 3,32] and uncertain (or Bayesian) [ 25] formulations
(see the recent survey by Perchet [ 43]). Perturbation theory for Nash equilibria has been derived in
these contexts, and it is well-known that the Nash equilibrium is not (at least in general) stable to
perturbations of the payoff matrix. On the other hand, the results of [ 22] consider Nash equilibria of
perturbed, symmetric, zero-sum games, but show that the payoff of the perturbed Nash equilibrium is
indeed stable. Our work provides a similar characterization for the multi-criteria setting.
2 Framework for preference learning along multiple criteria
We now set up our framework for preference learning along multiple criteria. We consider a collection
ofdobjects over which comparisons can be elicited along kdifferent criteria. We index the objects
by the set [d] : =f1;:::;dgand the criteria by the set [k].
2.1 Probabilistic model for comparisons
Since human responses to comparison queries are typically noisy, we model the pairwise preferences
as random variables drawn from an underlying population distribution. In particular, the result of a
comparison between a pair of objects (i1;i2)along criterion jis modeled as a draw from a Bernoulli
distribution, with p(i1;i2;j) =P(i1i2along criterion j):By symmetry, we must have
p(i2;i1;j) = 1p(i1;i2;j)for each triple i12[d]; i22[d];andj2[k]: (1)
We let1;22drepresent1two distributions over the dobjects. With a slight abuse of notation, let
p(1;2;j)denote the probability with which an object drawn from distribution 1beats an object
1We let ddenote the d-dimensional simplex.
3
drawn from distribution 2along criterion j. We assume for each individual criterion jthat the
probabilityp(1;2;j)is linear in the distributions 1and2, i.e. that it satisﬁes the relation
p(1;2;j) : =Ei11;i22[p(i1;i2;j)]: (2)
Equation (2)encodes the per-criterion linearity assumption highlighted in Section 1. We collect the
probabilitiesfp(i1;i2;j)ginto a preference tensor P2[0;1]ddkand denote byPd;kthe set of all
preference tensors that satisfy the symmetry condition (1). Speciﬁcally, we have
Pd;k=fP2[0;1]ddkjP(i1;i2;j) = 1P(i2;i1;j)for all (i1;i2;j)g: (3)
LetPjdenote theddmatrix corresponding to the comparisons along criterion j, so that
p(1;2;j) =>
1Pj2. Also note that a comparison between a pair of objects (i1;i2)induces
ascore vector containingksuch probabilities. Denote this vector by P(i1;i2)2[0;1]k, whosej-th
entry is given by p(i1;i2;j). Denote by P(1;2)the score vector for a pair of distribution (1;2).
In the single criterion case when k= 1, each comparison between a pair of objects is along an overall
criterion. We let Pov2[0;1]ddrepresent such an overall comparison matrix. As mentioned in
Section 1, most preference learning problems are multi-objective in nature, and the overall preference
matrix Povis derived as a non-linear combination of per-criterion preference matrices fPjgk
j=1.
Therefore, even when the linearity assumption (2)holds across each criterion, it might not hold for
theoverall preference Pov. In contrast, when the matrices Pjare aggregated linearly to obtain the
overall matrix Pov, we recover the assumptions of Dudik et al. [22].
2.2 Blackwell winner
(a) (b)
Figure 2. Two target sets S1andS2for our exam-
ple from Figure 1 that capture trade-offs between
comfort and speed. Set S1requires feasible score
vectors to satisfy 40% of the population along both
comfort and speed. Set S2requires both scores to
be greater than 0:3but with a linear trade-off: the
combined score must be at least 0:9.Given our probabilistic model for pairwise
comparisons, we now describe our notion of
a Blackwell winner. When deﬁning a win-
ning distribution for the multi-criteria case, it
would be ideal to ﬁnd a distribution that
is a von Neumman winner along each of the
criteria separately. However, as shown in our
example from Figure 1, such a distribution
need not exist. We thus need a generalization
of the von Neumann winner that explicitly ac-
counts for conﬂicts between the criteria.
Blackwell [ 11] asked a related question for
the theory of zero-sum games: how can one
generalize von Neumann’s minimax theorem
to vector-valued games? He proposed the no-
tion of a target set : a set of acceptable payoff vectors that the ﬁrst player in a zero-sum game seeks
to attain. Within this context, Blackwell proposed the notion of approachability, i.e. how the player
might obtain payoffs in a repeated game that are close to the target set on average. We take inspiration
from these ideas to deﬁne a solution concept for the multi-criteria preference problem.Our notion of a
winner also relies on a target set , which we denote by S[0;1]k, and which in our setting contains
score vectors . This set provides a way to combine different criteria by specifying combinations of
preference scores that are acceptable. Figure 2 provides an example of two such sets.
Observe that for our preference learning problem, the target set Sis by deﬁnition monotonic with
respect to the orthant ordering, that is, if z1z2coordinate-wise, then z22Simpliesz12S. Our
goal is to then produce a distribution that can achieve a target score vector for any distribution
with which it is compared—that is P(;)2Sfor all2d:When such a distribution exists,
we say that the problem instance (P;S)isachievable . On the other hand, it is clear that there are
problem instances (P;S)that are not achievable. While Blackwell’s workaround was to move to the
setting of repeated games, preference aggregation is usually a one-shot problem. Consequently, our
relaxation instead introduces the notion of a worst-case distance to the target set. In particular, we
measure the distance between any pair of score vectors u;v2[0;1]kas(u;v) =kuvkfor some
normkk. Using the shorthand (u;S) : = infv2Skuvk, the Blackwell winner for an instance
(P;S;kk)is now deﬁned as the one which minimizes the maximum distance to the set S, i.e.,
(P;S;kk)2argmin
2d[v(;P;S;kk)];wherev(;P;S;kk) : = max
02d(P(;0);S):(4)
4
Observe that equation (4)has an interpretation as a zero-sum game, where the objective of the
minimizing player is to make the score vector P(;0)as close as possible to the target set S.
We now look at commonly studied frameworks for single criterion preference aggregation and
multi-objective optimization and show how these can be naturally derived from our framework.
Example: Preference learning along a single criterion. A particular special case of our frame-
work is when we have a single criterion ( k= 1) and the preferences are given by a matrix Pov. The
scorePov(i1;i2)is a scalar representing the probability with which object i1beats object i2in an
overall comparison. As a consequence of the von Neumann minimax theorem, we have
max
12dmin
22dPov(1;2)= min
22dmax
12dPov(1;2)=1
2; (5)
with any maximizer above called the von Neumann winner [ 22]. Thus, for anypreference matrix
Pov, a von Neumann winner is preferred to any other object with probability at least1
2.
Let us show how this uni-criterion formulation can be derived as a special case of our framework.
Consider the target set S= [1
2;1]and choose the distance function (a;b) =jabj. By equation (5),
the target set S= [1
2;1]is achievable for all preference matrices Pov, and so the von Neumann
winner and the Blackwell winner (Pov;[1
2;1];jj)coincide. |
Example: Weighted combinations of a multi-criterion problem. One of the common ap-
proaches used in multi-objective optimization to reduce a multi-dimensional problem to a uni-
dimensional counterpart is by introducing a weighted combinations of objectives. Formally, consider
a weight vector w2kand the corresponding preference matrix P(w) : =P
j2[k]wjPjobtained
by combining the preference matrices along the different criteria. A winning distribution can then
be obtained by solving for the von Neumann winner of P(w)given by(P(w);[1
2;1];jj). The
following proposition establishes that such an approach is a particular special case of our framework.
Proposition 1. (a) For every weight vector w2k, there exists a target set Sw2[0;1]ksuch that
for any normkk, we have
(P;Sw;kk) =(P(w);[1=2;1];jj)for all P2Pd;k:
(b) Conversely, there exists a set Sand a preference tensor Pwith a unique Blackwell winner
such that for all w2k, exactly one of the following is true:
(P(w);[1=2;1];jj)6=orargmax
2dmin
i2[d]P(;i) = d:
Thus, while the Blackwell winner is always able to recover any linear combination of criteria, the
converse is not true. Speciﬁcally, part (b) of the proposition shows that for a choice of preference
tensorPand target set S, either the von Neumann winner for P(w)is not equal to the Blackwell
winner, or it degenerates to the entire simplex dand is thus uninformative. Consequently, our
framework is strictly more general that weighting the individual criteria. |
3 Statistical guarantees and computational approaches
In this section, we provide theoretical results on computing the Blackwell winner from samples of
pairwise comparisons along the various criteria.
Observation model and evaluation metrics. We operate in the natural passive observation model,
where a sample consists of a comparison between two randomly chosen objects along a randomly cho-
sen criterion. Speciﬁcally, we assume access to an oracle that when queried with a tuple = (i1;i2;j)
comprising a pair of objects (i1;i2)and a criterion j, returns a comparison y()Ber(p(i1;i2;j)).
Each query to the oracle constitutes one sample. In the passive sampling model, the tuple of objects
and criterion is sampled uniformly, with replacement, that is (i1;i2)Uniff[d]
2
gandjUniff[k]g
where UniffAgdenotes the uniform distribution over the elements of a set A. Given access to sam-
plesfy1(1);:::;yn(n)gfrom this observation model, we deﬁne the empirical preference tensor
(speciﬁcally the upper triangular part)
bPn(i1;i2;j) : =Pn
`=1y`(`)I[`= (i1;i2;j)]
1_P
`I[`= (i1;i2;j)]fori1<i2; (6)
5
where each entry of the upper-triangular tensor is estimated using a sample average and the remaining
entries are calculated to ensure the symmetry relations implied by the inclusion bPn2Pd;k.
As mentioned before, we are interested in computing the solution : =(P;S;kk)to the
optimization problem (4), but with access only to samples from the passive observation model. For
any estimatorb2dobtained from these samples, we evaluate its error based on its value with
respect to the tensor P, i.e.,
P(b;) : =v(b;S;P;kk)v(;S;P;kk): (7)
Note that the error Pimplicitly also depends on the set Sand the normkk, but we have chosen
our notation to be explicit only in the preference tensor P. For the rest of this section, we restrict our
attention to convex target sets Sand refer them to as valid sets . Having established the background,
we are now ready to provide sample complexity bounds on the estimation error P(b;).
3.1 Upper bounds on the error of the plug-in estimator
While, our focus in this section is to provide upper bounds on the error of the plug-in estimator
bplug=(bP;S;kk), we ﬁrst state a general perturbation bound which relates the error of the
optimizer(eP;S;kk)to the deviation of the tensor ePfrom the true tensor P. We use P(;i)2
[0;1]dkto denote a matrix formed by viewing the i-th slice of Palong its second dimension.
Theorem 1. Suppose the distance is induced by the norm kkqfor someq1. Then for each
valid target set Sand preference tensor eP, we have
P((eP);)2 max
i2[d]keP(;i)P(;i))k1;q: (8)
Note that this theorem is entirely deterministic: it bounds the deviation in the optimal solution to the
problem (4)as a function of perturbations to the tensor P. It also applies uniformly to all valid target
setsS. In particular, this result generalizes the perturbation result of Dudik et al. [ 22, Lemma 3]
which obtained such a deviation bound for the single criterion problem with as the von Neumann
winner. Indeed, one can observe that by setting the distance (u;v) =juvjin Theorem 1 for the
uni-criterion setup, we have the error P((eP);)2kePPk1;1, matching the bound of [ 22].
Let us now illustrate a consequence of this theorem by specializing it to the plug-in estimator, and
with the distances given by the `1norm.
Corollary 1. Suppose that the distance is induced by the `1-normkk1. Then there exists a
universal constant c>0such that given a sample size n>cd2klog(cdk
), we have for each valid
target setS
E[P(bplug;)]cs
d2k
nlogcdk

; (9)
with probability greater than 1.
The bound (9)implies that the plug-in estimator bplugis an-approximate solution whenever the
number of samples scales as n=eO(d2k
2). Observe that this sample complexity scales quadratically
in the number of objects dand linearly in the number of criteria k. This scaling represents the
effective dimensionality of the problem instance, since the underlying preference tensor Phas
O(d2k)unknown parameters. Notice that the corollary holds for sample size n=eO(d2k); this
should not be thought of as restrictive, since otherwise, the bound (9) is vacuous.
3.2 Information-theoretic lower bounds
While Corollary 1 provides an upper bound on the error of the plug-in estimator that holds for all
valid target sets S, it is natural to ask if this bounds is sharp, i.e., whether there is indeed a target set
Sfor which one can do no better than the plug-in estimator. In this section, we address this question
by providing lower bounds on the minimax risk Mn;d;k(S;kk1) : = infbsupP2PE[P(b;)];
where the inﬁmum is taken over allestimators that can be computed from nsamples from our
observation model. It is important to note that the error Pis computed using the `1norm and for
the setS. Our lower bound will apply to the particular choice of target set S0= [1=2;1]k.
6
Theorem 2. There is a universal constant csuch that for all d4,k2, andncd4k, we have
Mn;d;k(S0;kk1)cr
d2k
n: (10)
Comparing equations and (9)and(10), we see that for the `1-norm and the set S0, we have provided
upper and lower bounds that match up to a logarithmic factor in the dimension. Thus, the plug-in
estimator is indeed optimal for this pair (kk1;S0).Further, observe that the above lower bound is
non-asymptotic, and holds for all values of n&d4k. This condition on the sample size arises as a
consequence of the speciﬁc packing set used for establishing the lower bound, and improving it is an
interesting open problem.
However, this raises the question of whether the set S0is special, or alternatively, whether one can
obtain anS-dependent lower bound. The following proposition shows that at least asymptotically ,
the sample complexity for anypolyhedral set Sobeys a similar lower bound.
Proposition 2 (Informal) .Suppose that we have a valid polyhedral target set S, and thatd4.
There exists a positive integer n0(d;k;S )such that for all nn0(d;k;S )we have
Mn;d;k(S;kk1)&r
d2k
n: (11)
We defer the formal statement and proof of this proposition to Appendix B. This proposition estab-
lishes that the plugin estimator bplugis indeed asymptotically optimal in the `1norm for broad class
of setsS.
3.3 Computing the plug-in estimator
In the last few sections, we discussed the statistical properties of the plug-in estimator, and showed
that its sample complexity was optimal in a minimax sense. We now turn to the algorithmic question:
how can the plug-in estimator bplugbe computed? Our main result in this direction is the following
theorem that characterizes properties of the objective function v(;P;S;kk).
Theorem 3. Suppose that the distance function is given by an `qnormkkqfor someq1. Then
for each valid target set S, the objective function v(;P;S;kkq)is convex in , and Lipschitz in
the`1norm, i.e.,
jv(1;P;S;kkq)v(2;P;S;kkq)jk1
qk12k1for each1;22d:
Theorem 3 establishes that the plug-in estimator can indeed be computed as the solution to a
(constrained) convex optimization problem. In Appendix C, we discuss a few speciﬁc algorithms
based on zeroth-order and ﬁrst-order methods for obtaining such a solution and an analysis of the
corresponding iteration complexity for these methods; see Propositions 5 and 6 in the appendix.
4 Autonomous driving user study
In order to evaluate the proposed framework, we applied it to an autonomous driving environment. The
objective is to study properties of the randomized policies obtained by our multi-criteria framework—
the Blackwell winner for speciﬁc choices of the target set—and compare them with the alternative
approaches of linear combinations of criteria and the single-criterion (overall) von Neumann winner.
We brieﬂy describe the components of the experiment here; see Appendix D for more details.
Self-driving Environment. Figure 1(a) shows a snapshot of one of the worlds in this environment
with the autonomous car shown in orange. We construct three different worlds in this environment:
W1: The ﬁrst world comprises an empty stretch of road with no obstacles (20 steps).
W2: The second world consists of a sequence of cones placed in certain sequences (80 steps).
W3: The third world has additional cars driving at varying speeds in their ﬁxed lanes (80 steps).
7
Policies. For our base policies , we design ﬁve different reward functions encoding different self-
driving behaviors. These polices, named Policy A-E, are then set to be the model predictive control
based policies based on these reward functions wherein we ﬁx the planning horizon to 6. We defer
the details of these reward functions to Appendix D. A randomized policy 25is given by a
distribution over the base policies A-E. Such a randomized policy is implemented in our environment
by randomly sampling a base policy from the mixture distribution after every H= 18 time steps and
executing this selected policy for that duration. To account for the randomization, we execute each
such policy for 5independent runs in each of the worlds and record these behaviors.
Subjective Criteria. We selected ﬁve subjective criteria to compare the policies, with questions
asking which of the two policies was C1: Less aggressive, C2: More predictable, C3: More quick,
C4: More conservative, and had C5: Less collision risk. Such a framing of question ensures that
higher score value along any of C1-C5 is preferred; thus a higher score along C1 would imply less
aggressive while along C2 would mean more predictable. In addition to the these base criteria, we
also consider an Overall Preference which compares any pair of policies in an aggregate manner.
Additionally, we also asked the users to rate the importance of each criterion in their overall preference.
Main Hypotheses. The central focus of the main hypotheses is on comparing the randomized
policies given by the Blackwell winner, the overall von Neumann winner, and those given by
weighing the criteria linearly.
MH1 There exists a set Ssuch that the Blackwell winner with respect to Sand`1-norm produced
by our framework outperforms the overall von Neumann winner.
MH2 The Blackwell winner for oblivious score sets Soutperforms both oblivious2and data-driven
weights for linear combination of criteria.
Independent Variables. The independent variable of our experiment is the choice of algorithms
for producing the different randomized winners. These comprise the von Neumann winner based
on overall comparisons, Blackwell winners based on two oblivious target sets, and 9 different linear
combinations weights (3 data-driven and 6 oblivious).
We begin with the two target sets S1andS2for our evaluation of the Blackwell winner which
were selected in a data-oblivious manner. Set S1is an axis-aligned set promoting the use of safer
policies with score vector constrained to have a larger value along the collision risk axis. Similar to
Figure 2(b), the set S2adds a linear constraint along aggressiveness and collision risk. This target set
thus favors policies which are less aggressive and have lower collision risk. For evaluating hypothesis
MH2, we considered several weight vectors, both oblivious and data-dependent, comprising average
of the users’ self-reported weights, that obtained by regressing the overall criterion on C1-C5, and a
set of oblivious weights. See Appendix D for details of the sets S1andS2, and the weights w1:9.
Data collection. The experiment was conducted in two phases, both of which involved human sub-
jects on Amazon Mechanical Turk (Mturk) (see Appendix D for an illustration of the questionnaire).
The ﬁrst phase of the experiment involved preference elicitation for the ﬁve base policies A-E. Each
user was asked to provide comparison data for all ten combinations of policies. The cumulative
comparison data is given in Appendix D, and the average weight vector elicited from the users was
found to bew1= [0:21;0:19;0:20;0:18;0:22]. We ran this study with 50 subjects.
In the overall preference elicitation, we saw an approximate ordering amongst the base policies:
CE%D%BA. Thus, Policy C was the von Neumann winner along the overall criterion.
For each of the linear combination weights w1throughw9, Policy C was the weighted winner.
The Blackwell winners R1 and R2 for the sets S1andS2with the`1distance were found to be
R1= [0:09;0:15;0:30;0:15;0:31]and R2 = [0:01;0:01;0:31;0:02;0:65].
In the second phase, we obtained preferences from a set of 41 subjects comparing the randomized
polices R1 and R2 with the baseline policies A-E. The results are aggregated in Table 1 in Appendix D.
Analysis for main hypotheses. Given that the overall von Neumann winner and those correspond-
ing to weights w1:9were all Policy C, hypotheses MH1 and MH2 reduced whether users prefer at least
one of {R1, R2} to the deterministic policy C, that is whether Pov(C;R1)<0:5orPov(C;R2)<0:5.
2We use the term oblivious to denote variables that were ﬁxed before the data collection phase and data-driven
to denote those which are based on collected data.
8
Policies C and E were preferred to R1 by 0:71and0:61fraction of the respondents, respectively.
On the other hand, R2 was preferred to the von Neumann winner C by 0:66fraction of the subjects.
Using the data, we conducted a hypothesis test with the null and alternative hypotheses given by
H0:Pov(C;R2)0:5;andH1:Pov(C;R2)<0:5:
Among the hypotheses that make up the (composite) null, our samples have the highest likelihood
for the distribution Ber(0:5). We therefore perform a one-sided hypothesis test with the Binomial
distribution with number of samples n= 41 , success probability p= 0:5and number of successes
x= 14 (indicating number of subjects which preferred Policy C to R2). The p-value for this test was
obtained to be 0:0298 . This supports both our claimed hypotheses MH1 and MH2.
5 Discussion and future work
In this paper, we considered the problem of eliciting and learning from preferences along multiple
criteria, as a way to obtain rich feedback under weaker assumptions. We introduced the notion of a
Blackwell winner, which generalizes many known winning solution concepts. We showed that the
Blackwell winner was efﬁciently computable from samples with a simple and optimal procedure, and
also that it outperformed the von Neumann winner in a user study on autonomous driving. Our work
raises many interesting follow-up questions: How does the sample complexity vary as a function of
the preference tensor P? Can the process of choosing a good target set be automated? What are the
analogs of our results in the setting where pairwise comparisons can be elicited actively ?
Broader impact
An important step towards deploying AI systems in the real world involves aligning their objectives
with human values. Examples of such objectives include safety for autonomous vehicles, fairness
for recommender systems, and effectiveness of assistive medical devices. Our paper takes a step
towards accomplishing this goal by providing a framework to aggregate human preferences along
such subjective criteria, which are often hard to encode mathematically. While our framework is
quite expressive and allows for non-linear aggregation across criteria, it leaves the choice of the target
set in the hands of the designer. As a possible negative consequence, getting this choice wrong could
lead to incorrect inferences and unexpected behavior in the real world.
Acknowledgments and Disclosure of Funding
We would like to thank Niladri Chatterji, Robert Kleinberg and Karthik Sridharan for helpful
discussions, and Andreea Bobu, Micah Carroll, Lawrence Chan and Gokul Swamy for helping with
the user study setup.
AP is supported by a Swiss Re research fellowship at the Simons Institute for the Theory of Computing
and KB is supported by a JP Morgan AI Fellowship. This work was partially supported by Ofﬁce
of Naval Research Young Investigator Award and a AFOSR grant to ADD, and by Ofﬁce of Naval
Research Grant DOD ONR-N00014-18-1-2640 to MJW.
Additional revenue: ADD is employed as a consultant at Waymo, LLC and PLB is employed as a
consultant at Google.
9
References
[1]J. Abernethy, P. L. Bartlett, and E. Hazan. Blackwell approachability and no-regret learning are
equivalent. In Proceedings of the 24th Annual Conference on Learning Theory , pages 27–46,
2011.
[2]A. Agarwal, O. Dekel, and L. Xiao. Optimal algorithms for online convex optimization with
multi-point bandit feedback. In COLT , 2010.
[3]M. Aghassi and D. Bertsimas. Robust game theory. Mathematical Programming , 107(1-2):
231–273, 2006.
[4]N. Ailon, Z. Karnin, and T. Joachims. Reducing dueling bandits to cardinal bandits. In
International Conference on Machine Learning , pages 856–864, 2014.
[5]A. R. Alimov and I. Tsar’kov. Connectedness and other geometric properties of suns and
chebyshev sets. Fundamentalnaya i Prikladnaya Matematika , 19(4):21–91, 2014.
[6]S. Amershi, M. Cakmak, W. B. Knox, and T. Kulesza. Power to the people: The role of humans
in interactive machine learning. Ai Magazine , 35(4):105–120, 2014.
[7] K. J. Arrow et al. Social choice and individual values. 1951.
[8]V . Balestro, H. Martini, and R. Teixeira. Convex analysis in normed spaces and metric projec-
tions onto convex bodies. arXiv preprint arXiv:1908.08742 , 2019.
[9]E. Bıyık, N. Huynh, M. J. Kochenderfer, and D. Sadigh. Active preference-based gaussian
process regression for reward learning. arXiv preprint arXiv:2005.02575 , 2020.
[10] D. Black. On the rationale of group decision-making. Journal of political economy , 56(1):
23–34, 1948.
[11] D. Blackwell. An analog of the minimax theorem for vector payoffs. Paciﬁc Journal of
Mathematics , 6(1):1–8, 1956.
[12] J. d. Borda. Mémoire sur les élections au scrutin. Histoire de l’Academie Royale des Sciences
pour 1781 (Paris, 1784) , 1784.
[13] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. The method of
paired comparisons. Biometrika , 39(3/4):324–345, 1952.
[14] S. Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends R
in
Machine Learning , 8, 2015.
[15] O. Chapelle, T. Joachims, F. Radlinski, and Y . Yue. Large-scale validation and analysis of
interleaved search evaluation. ACM Transactions on Information Systems (TOIS) , 30(1):1–41,
2012.
[16] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement
learning from human preferences. In Advances in Neural Information Processing Systems ,
pages 4299–4307, 2017.
[17] M. d. Condorcet. Essai sur l’application de l’analyse a la probabilité des decisions rendues a la
pluralité des voix. 1785.
[18] A. H. Copeland. A reasonable social welfare function. Technical report, mimeo, 1951. University
of Michigan, 1951.
[19] K. M. Douglas and R. J. Mislevy. Estimating classiﬁcation accuracy for complex decision rules
based on multiple scores. Journal of Educational and Behavioral Statistics , 35(3):280–306,
2010.
[20] M. Doumpos and C. Zopounidis. Regularized estimation for preference disaggregation in
multiple criteria decision making. Computational Optimization and Applications , 38(1):61–80,
2007.
[21] J. C. Duchi, M. I. Jordan, M. J. Wainwright, and A. Wibisono. Optimal rates for zero-order
convex optimization: The power of two function evaluations. IEEE Transactions on Information
Theory , 61(5), 2015.
[22] M. Dudík, K. Hofmann, R. E. Schapire, A. Slivkins, and M. Zoghi. Contextual dueling bandits.
InConference on Learning Theory , 2015.
10
[23] A. D. Flaxman, A. T. Kalai, and H. B. McMahan. Online convex optimization in the bandit
setting: gradient descent without a gradient. In Proceedings of the sixteenth annual ACM-SIAM
symposium on Discrete algorithms , 2005.
[24] D. Frisch and R. T. Clemen. Beyond expected utility: Rethinking behavioral decision research.
Psychological bulletin , 116(1):46, 1994.
[25] D. Fudenberg and D. K. Levine. Self-conﬁrming equilibrium. Econometrica: Journal of the
Econometric Society , pages 523–545, 1993.
[26] S. Ghadimi and G. Lan. Stochastic ﬁrst-and zeroth-order methods for nonconvex stochastic
programming. SIAM Journal on Optimization , 23(4), 2013.
[27] W. M. Goldstein and J. Beattie. Judgments of relative importance in decision making: The
importance of interpretation and the interpretation of importance. In Frontiers of mathematical
psychology , pages 110–137. Springer, 1991.
[28] K. Hofmann, S. Whiteson, and M. De Rijke. A probabilistic method for inferring preferences
from clicks. In Proceedings of the 20th ACM international conference on Information and
knowledge management , pages 249–258, 2011.
[29] E. Hüllermeier, J. Fürnkranz, W. Cheng, and K. Brinker. Label ranking by learning pairwise
preferences. Artiﬁcial Intelligence , 172(16-17), 2008.
[30] K. G. Jamieson, S. Katariya, A. Deshpande, and R. D. Nowak. Sparse dueling bandits. 2015.
[31] V . Kuleshov and S. Ermon. Estimating uncertainty online against an adversary. In Thirty-First
AAAI Conference on Artiﬁcial Intelligence , 2017.
[32] E. Lehrer. Partially speciﬁed probabilities: Decisions and games. American Economic Journal:
Microeconomics , 4(1):70–100, 2012.
[33] R. D. Luce. Individual choice behavior. 1959.
[34] S. Mannor, V . Perchet, and G. Stoltz. Approachability in unknown games: Online learning
meets multi-objective optimization. In Conference on Learning Theory , pages 339–355, 2014.
[35] M. T. McBee, S. J. Peters, and C. Waterman. Combining scores in multiple-criteria assessment
systems: The impact of combination rule. Gifted Child Quarterly , 58(1):69–89, 2014.
[36] S. Miryooseﬁ, K. Brantley, H. Daume III, M. Dudik, and R. E. Schapire. Reinforcement
learning with convex constraints. In Advances in Neural Information Processing Systems , pages
14070–14079, 2019.
[37] H. Moulin. Handbook of Computational Social Choice . Cambridge University Press, 2016.
[38] Y . Nesterov and V . Spokoiny. Random gradient-free minimization of convex functions. Founda-
tions of Computational Mathematics , 17(2), 2017.
[39] M. Palan, N. C. Landolﬁ, G. Shevchuk, and D. Sadigh. Learning reward functions by integrating
human demonstrations and preferences. arXiv preprint arXiv:1906.08928 , 2019.
[40] J. P. Papay. Different tests, different answers: The stability of teacher value-added estimates
across outcome measures. American Educational Research Journal , 48(1):163–193, 2011.
[41] J.-P. Penot and R. Ratsimahalo. Characterizations of metric projections in Banach spaces and
applications. In Abstract and Applied Analysis , volume 3, 1970.
[42] V . Perchet. Approachability, regret and calibration; implications and equivalences. arXiv
preprint arXiv:1301.2663 , 2013.
[43] V . Perchet. A note on robust nash equilibria with uncertainties. RAIRO-Operations Research ,
48(3):365–371, 2014.
[44] J.-C. Pomerol and S. Barba-Romero. Multicriterion decision in management: principles and
practice , volume 25. Springer Science & Business Media, 2012.
[45] D. Sadigh, A. D. Dragan, S. Sastry, and S. A. Seshia. Active preference-based learning of
reward functions. In Robotics: Science and Systems , 2017.
[46] W. Saunders, G. Sastry, A. Stuhlmueller, and O. Evans. Trial without error: Towards safe
reinforcement learning via human intervention. In Proceedings of the 17th International
Conference on Autonomous Agents and MultiAgent Systems , pages 2067–2069, 2018.
11
[47] M. Schulze. A new monotonic, clone-independent, reversal symmetric, and Condorcet-
consistent single-winner election method. Social Choice and Welfare , 36(2):267–303, 2011.
[48] O. Shamir. On the complexity of bandit and derivative-free stochastic convex optimization. In
Conference on Learning Theory , 2013.
[49] O. Shamir. An optimal algorithm for bandit and zero-order convex optimization with two-point
feedback. The Journal of Machine Learning Research , 18(1), 2017.
[50] A. Teixeira-Pinto and S.-L. T. Normand. Statistical methodology for classifying units on the
basis of multiple-related measures. Statistics in medicine , 27(9):1329–1350, 2008.
[51] L. L. Thurstone. A law of comparative judgment. Psychological review , 34(4):273, 1927.
[52] A. B. Tsybakov. Introduction to nonparametric estimation . Springer Science & Business Media,
2008.
[53] A. Tversky and D. Kahneman. Judgment under uncertainty: Heuristics and biases. science , 185
(4157):1124–1131, 1974.
[54] A. Tversky and D. Kahneman. Prospect theory: An analysis of decision under risk. Economet-
rica, 47(2):263–291, 1979.
[55] G. D. Walters. Taking the next step: Combining incrementally valid indicators to improve
recidivism prediction. Assessment , 18(2):227–233, 2011.
[56] H. Wu and X. Liu. Double thompson sampling for dueling bandits. In Advances in Neural
Information Processing Systems , pages 649–657, 2016.
[57] Y . Yue, J. Broder, R. Kleinberg, and T. Joachims. The k-armed dueling bandits problem. Journal
of Computer and System Sciences , 78(5):1538–1556, 2012.
[58] L. Zají ˇcek. On the fréchet differentiability of distance functions. Proceedings of the 12th Winter
School on Abstract Analysis , pages 161–165, 1984.
[59] J. Zhang, P. Fiers, K. A. Witte, R. W. Jackson, K. L. Poggensee, C. G. Atkeson, and S. H.
Collins. Human-in-the-loop optimization of exoskeleton assistance during walking. Science ,
356(6344):1280–1284, 2017.
[60] M. Zoghi, S. Whiteson, R. Munos, and M. De Rijke. Relative upper conﬁdence bound for the
k-armed dueling bandit problem. arXiv preprint arXiv:1312.3393 , 2013.
[61] M. Zoghi, Z. S. Karnin, S. Whiteson, and M. De Rijke. Copeland dueling bandits. In Advances
in Neural Information Processing Systems , pages 307–315, 2015.
[62] M. Zoghi, S. Whiteson, and M. de Rijke. Mergerucb: A method for large-scale online ranker
evaluation. In Proceedings of the Eighth ACM International Conference on Web Search and
Data Mining , pages 17–26, 2015.
[63] C. Zopounidis and M. Doumpos. Multicriteria classiﬁcation and sorting methods: A literature
review. European Journal of Operational Research , 138(2):229–246, 2002.
12

Kush Bhatia
Ashwin Pananjady
Peter L
Bartlett
Anca D
Dragan
Martin J
Wainwright


SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards..

Published as a conference paper at ICLR 2020
SQIL: I MITATION LEARNING VIA REINFORCEMENT
LEARNING WITH SPARSE REWARDS
Siddharth Reddy, Anca D. Dragan, Sergey Levine
Department of Electrical Engineering and Computer Science
University of California, Berkeley
fsgr,anca,svlevine g@berkeley.edu
ABSTRACT
Learning to imitate expert behavior from demonstrations can be challenging, es-
pecially in environments with high-dimensional, continuous observations and un-
known dynamics. Supervised learning methods based on behavioral cloning (BC)
suffer from distribution shift: because the agent greedily imitates demonstrated
actions, it can drift away from demonstrated states due to error accumulation. Re-
cent methods based on reinforcement learning (RL), such as inverse RL and gen-
erative adversarial imitation learning (GAIL), overcome this issue by training an
RL agent to match the demonstrations over a long horizon. Since the true reward
function for the task is unknown, these methods learn a reward function from the
demonstrations, often using complex and brittle approximation techniques that in-
volve adversarial training. We propose a simple alternative that still uses RL, but
does not require learning a reward function. The key idea is to provide the agent
with an incentive to match the demonstrations over a long horizon, by encourag-
ing it to return to demonstrated states upon encountering new, out-of-distribution
states. We accomplish this by giving the agent a constant reward of r= +1 for
matching the demonstrated action in a demonstrated state, and a constant reward
ofr= 0for all other behavior. Our method, which we call soft Q imitation learn-
ing(SQIL), can be implemented with a handful of minor modiﬁcations to any
standard Q-learning or off-policy actor-critic algorithm. Theoretically, we show
that SQIL can be interpreted as a regularized variant of BC that uses a sparsity
prior to encourage long-horizon imitation. Empirically, we show that SQIL out-
performs BC and achieves competitive results compared to GAIL, on a variety of
image-based and low-dimensional tasks in Box2D, Atari, and MuJoCo. This pa-
per is a proof of concept that illustrates how a simple imitation method based on
RL with constant rewards can be as effective as more complex methods that use
learned rewards.
1 I NTRODUCTION
Many sequential decision-making problems can be tackled by imitation learning: an expert demon-
strates near-optimal behavior to an agent, and the agent attempts to replicate that behavior in novel
situations (Argall et al., 2009). This paper considers the problem of training an agent to imitate
an expert, given expert action demonstrations and the ability to interact with the environment. The
agent does not observe a reward signal or query the expert, and does not know the state transition
dynamics.
Standard approaches based on behavioral cloning (BC) use supervised learning to greedily imitate
demonstrated actions, without reasoning about the consequences of actions (Pomerleau, 1991). As
a result, compounding errors cause the agent to drift away from the demonstrated states (Ross et al.,
2011). The problem with BC is that, when the agent drifts and encounters out-of-distribution states,
the agent does not know how to return to the demonstrated states. Recent methods based on in-
verse reinforcement learning (IRL) overcome this issue by training an RL agent not only to imitate
demonstrated actions, but also to visit demonstrated states (Ng et al., 2000; Wulfmeier et al., 2015;
Finn et al., 2016b; Fu et al., 2017). This is also the core idea behind generative adversarial imi-
tation learning (GAIL) (Ho & Ermon, 2016), which implements IRL using generative adversarial
1
Published as a conference paper at ICLR 2020
networks (Goodfellow et al., 2014; Finn et al., 2016a). Since the true reward function for the task
is unknown, these methods construct a reward signal from the demonstrations through adversarial
training, making them difﬁcult to implement and use in practice (Kurach et al., 2018).
The main idea in this paper is that the effectiveness of adversarial imitation methods can be achieved
by a much simpler approach that does not require adversarial training, or indeed learning a reward
function at all. Intuitively, adversarial methods encourage long-horizon imitation by providing the
agent with (1) an incentive to imitate the demonstrated actions in demonstrated states, and (2) an
incentive to take actions that lead it back to demonstrated states when it encounters new, out-of-
distribution states. One of the reasons why adversarial methods outperform greedy methods, such as
BC, is that greedy methods only do (1), while adversarial methods do both (1) and (2). Our approach
is intended to do both (1) and (2) without adversarial training, by using constant rewards instead of
learned rewards. The key idea is that, instead of using a learned reward function to provide a reward
signal to the agent, we can simply give the agent a constant reward of r= +1 for matching the
demonstrated action in a demonstrated state, and a constant reward of r= 0for all other behavior.
We motivate this approach theoretically, by showing that it implements a regularized variant of BC
that learns long-horizon imitation by (a) imposing a sparsity prior on the reward function implied
by the imitation policy, and (b) incorporating information about the state transition dynamics into
the imitation policy. Intuitively, our method accomplishes (a) by training the agent using an ex-
tremely sparse reward function – +1for demonstrations, 0everywhere else – and accomplishes (b)
by training the agent with RL instead of supervised learning.
We instantiate our approach with soft Q-learning (Haarnoja et al., 2017) by initializing the agent’s
experience replay buffer with expert demonstrations, setting the rewards to a constant r= +1 in
the demonstration experiences, and setting rewards to a constant r= 0in all of the new experiences
the agent collects while interacting with the environment. Since soft Q-learning is an off-policy
algorithm, the agent does not necessarily have to visit the demonstrated states in order to experience
positive rewards. Instead, the agent replays the demonstrations that were initially added to its buffer.
Thus, our method can be applied in environments with stochastic dynamics and continuous states,
where the demonstrated states are not necessarily reachable by the agent. We call this method soft
Q imitation learning (SQIL).
The main contribution of this paper is SQIL: a simple and general imitation learning algorithm that is
effective in MDPs with high-dimensional, continuous observations and unknown dynamics. We run
experiments in four image-based environments – Car Racing, Pong, Breakout, and Space Invaders –
and three low-dimensional environments – Humanoid, HalfCheetah, and Lunar Lander – to compare
SQIL to two prior methods: BC and GAIL. We ﬁnd that SQIL outperforms BC and achieves com-
petitive results compared to GAIL. Our experiments illustrate two key beneﬁts of SQIL: (1) that it
can overcome the state distribution shift problem of BC without adversarial training or learning a re-
ward function, which makes it easier to use, e.g., with images, and (2) that it is simple to implement
using existing Q-learning or off-policy actor-critic algorithms.
2 S OFT Q IMITATION LEARNING
SQIL performs soft Q-learning (Haarnoja et al., 2017) with three small, but important, modiﬁcations:
(1) it initially ﬁlls the agent’s experience replay buffer with demonstrations, where the rewards are
set to a constant r= +1 ; (2) as the agent interacts with the world and accumulates new experiences,
it adds them to the replay buffer, and sets the rewards for these new experiences to a constant r= 0;
and (3) it balances the number of demonstration experiences and new experiences (50% each) in each
sample from the replay buffer.1These three modiﬁcations are motivated theoretically in Section 3,
via an equivalence to a regularized variant of BC. Intuitively, these modiﬁcations create a simple
reward structure that gives the agent an incentive to imitate the expert in demonstrated states, and to
take actions that lead it back to demonstrated states when it strays from the demonstrations.
1SQIL resembles the Deep Q-learning from Demonstrations (DQfD) (Hester et al., 2017) and Normalized
Actor-Critic (NAC) algorithms (Gao et al., 2018), in that it initially ﬁlls the agent’s experience replay buffer
with demonstrations. The key difference is that DQfD and NAC are RL algorithms that assume access to a
reward signal, while SQIL is an imitation learning algorithm that does not require an extrinsic reward signal
from the environment. Instead, SQIL automatically constructs a reward signal from the demonstrations.
2
Published as a conference paper at ICLR 2020
Algorithm 1 Soft Q Imitation Learning (SQIL)
1: Requiresamp2R0;Ddemo
2: InitializeDsamp ;
3:whileQnot converged do
4: r(2(Ddemo;1) +samp2(Dsamp;0))fSee Equation 1g
5: Sample transition (s;a;s0)with imitation policy (ajs)/exp (Q(s;a))
6:Dsamp D samp[f(s;a;s0)g
7:end while
Crucially, since soft Q-learning is an off-policy algorithm, the agent does not necessarily have to
visit the demonstrated states in order to experience positive rewards. Instead, the agent replays
the demonstrations that were initially added to its buffer. Thus, SQIL can be used in stochastic
environments with high-dimensional, continuous states, where the demonstrated states may never
actually be encountered by the agent.
SQIL is summarized in Algorithm 1, where Qis the soft Q function, Ddemoare demonstrations, 2
is the squared soft Bellman error,
2(D;r),1
jDjX
(s;a;s0)2D
Q(s;a)
r+
log X
a02Aexp (Q(s0;a0))!!! 2
; (1)
andr2f0;1gis a constant reward.2The experiments in Section 4 use a convolutional neural
network or multi-layer perceptron to model Q, where are the weights of the neural network.
Section A.3 in the appendix contains additional implementation details, including values for the
hyperparameter samp; note that the simple default value of samp= 1works well across a variety of
environments.
As the imitation policy in line 5 of Algorithm 1 learns to behave more like the expert, a growing
number of expert-like transitions get added to the buffer Dsampwith an assigned reward of zero. This
causes the effective reward for mimicking the expert to decay over time. Balancing the number of
demonstration experiences and new experiences (50% each) sampled for the gradient step in line
4 ensures that this effective reward remains at least 1=(1 +samp), instead of decaying to zero. In
practice, we ﬁnd that this reward decay does not degrade performance if SQIL is halted once the
squared soft Bellman error loss converges to a minimum (e.g., see Figure 8 in the appendix). Note
that prior methods also require similar techniques: both GAIL and adversarial IRL (AIRL) (Fu et al.,
2017) balance the number of positive and negative examples in the training set of the discriminator,
and AIRL tends to require early stopping to avoid overﬁtting.
3 I NTERPRETING SQIL ASREGULARIZED BEHAVIORAL CLONING
To understand why SQIL works, we sketch a surprising theoretical result: SQIL is equivalent to a
variant of behavioral cloning (BC) that uses regularization to overcome state distribution shift.
BC is a simple approach that seeks to imitate the expert’s actions using supervised learning – in
particular, greedily maximizing the conditional likelihood of the demonstrated actions given the
demonstrated states, without reasoning about the consequences of actions. Thus, when the agent
makes small mistakes and enters states that are slightly different from those in the demonstrations,
the distribution mismatch between the states in the demonstrations and those actually encountered
by the agent leads to compounding errors (Ross et al., 2011). We show that SQIL is equivalent to
augmenting BC with a regularization term that incorporates information about the state transition
dynamics into the imitation policy, and thus enables long-horizon imitation.
3.1 P RELIMINARIES
Maximum entropy model of expert behavior. SQIL is built on soft Q-learning, which assumes
that expert behavior follows the maximum entropy model (Ziebart et al., 2010; Levine, 2018). In
2Equation 1 assumes discrete actions, but SQIL can also be used with continuous actions, as shown in
Section 4.3.
3
Published as a conference paper at ICLR 2020
an inﬁnite-horizon Markov Decision Process (MDP) with a continuous state space Sand discrete
action spaceA,3the expert is assumed to follow a policy that maximizes reward R(s;a). The
policyforms a Boltzmann distribution over actions,
(ajs),exp (Q(s;a))P
a02Aexp (Q(s;a0)); (2)
whereQis the soft Q function. The soft Q values are a function of the rewards and dynamics, given
by the soft Bellman equation,
Q(s;a),R(s;a) +
Es0"
log X
a02Aexp (Q(s0;a0))!#
: (3)
In our imitation setting, the rewards and dynamics are unknown. The expert generates a ﬁxed set of
demonstrationsDdemo, by rolling out their policy in the environment and generating state transi-
tions (s;a;s0)2D demo.
Behavioral cloning (BC). Training an imitation policy with standard BC corresponds to ﬁtting a
parametric model that minimizes the negative log-likelihood loss,
`BC(),X
(s;a)2D demolog(ajs): (4)
In our setting, instead of explicitly modeling the policy , we can represent the policy in terms
of a soft Q function Qvia Equation 2:
(ajs),exp (Q(s;a))P
a02Aexp (Q(s;a0)): (5)
Using this representation of the policy, we can train Qvia the maximum-likelihood objective in
Equation 4:
`BC(),X
(s;a)2D demo
Q(s;a)log X
a02Aexp (Q(s;a0))!!
: (6)
However, optimizing the BC loss in Equation 6 does not in general yield a valid soft Q function
Q– i.e., a soft Q function that satisﬁes the soft Bellman equation (Equation 3) with respect to
the dynamics and some reward function. The problem is that the BC loss does not incorporate any
information about the dynamics into the learning objective, so Qlearns to greedily assign high
values to demonstrated actions, without considering the state transitions that occur as a consequence
of actions. As a result, Qmay output arbitrary values in states that are off-distribution from the
demonstrationsDdemo.
In Section 3.2, we describe a regularized BC algorithm that adds constraints to ensure that Qis
a valid soft Q function with respect to some implicitly-represented reward function, and further
regularizes the implicit rewards with a sparsity prior. In Section 3.3, we show that this approach
recovers an algorithm similar to SQIL.
3.2 R EGULARIZED BEHAVIORAL CLONING
Under the maximum entropy model described in Section 3.1, expert behavior is driven by a reward
function, a soft Q function that computes expected future returns, and a policy that takes actions with
high soft Q values. In the previous section, we used these assumptions to represent the imitation
policy in terms of a model of the soft Q function Q(Equation 5). In this section, we represent the
reward function implicitly in terms of Q, as shown in Equation 7. This allows us to derive SQIL as
a variant of BC that imposes a sparsity prior on the implicitly-represented rewards.
Sparsity regularization. The issue with BC is that, when the agent encounters states that are out-
of-distribution with respect to Ddemo,Qmay output arbitrary values. One solution from prior work
3Assuming a discrete action space simpliﬁes our analysis. SQIL can be applied to continuous control tasks
using existing sampling methods (Haarnoja et al., 2017; 2018), as illustrated in Section 4.3.
4
Published as a conference paper at ICLR 2020
(Piot et al., 2014) is to regularize Qwith a sparsity prior on the implied rewards – in particular, a
penalty on the magnitude of the rewardsP
s2S;a2AjRq(s;a)jimplied byQvia the soft Bellman
equation (Equation 3), where
Rq(s;a),Q(s;a)
Es0"
log X
a02Aexp (Q(s0;a0))!#
: (7)
Note that the reward function Rqis not explicitly modeled in this method. Instead, we directly
minimize the magnitude of the right-hand side of Equation 7, which is equivalent to minimizing
jRq(s;a)j.
The purpose of the penalty on jRq(s;a)jis two-fold: (1) it imposes a sparsity prior motivated by
prior work (Piot et al., 2013), and (2) it incorporates information about the state transition dynamics
into the imitation learning objective, since Rq(s;a)is a function of an expectation over next state s0.
(2) is critical for learning long-horizon behavior that imitates the demonstrations, instead of greedy
maximization of the action likelihoods in standard BC. For details, see Piot et al. (2014).
Approximations for continuous states. Unlike the discrete environments tested in Piot et al.
(2014), we assume the continuous state space Scannot be enumerated. Hence, we approximate
the penaltyP
s2S;a2AjRq(s;a)jby estimating it from samples: transitions (s;a;s0)observed in
the demonstrations Ddemo, as well as additional rollouts Dsampperiodically sampled during training
using the latest imitation policy. This approximation, which follows the standard approach to con-
straint sampling (Calaﬁore & Dabbene, 2006), ensures that the penalty covers the state distribution
actually encountered by the agent, instead of only the demonstrations.
To make the penalty continuously differentiable, we introduce an additional approximation: instead
of penalizing the absolute value jRq(s;a)j, we penalize the squared value (Rq(s;a))2. Note that
since the reward function Rqis not explicitly modeled, but instead deﬁned via Qin Equation 7,
the squared penalty (Rq(s;a))2is equivalent to the squared soft Bellman error 2(Ddemo[D samp;0)
from Equation 1.
Regularized BC algorithm. Formally, we deﬁne the regularized BC loss function adapted from
Piot et al. (2014) as
`RBC(),`BC() +2(Ddemo[D samp;0); (8)
where2R0is a constant hyperparameter, and 2denotes the squared soft Bellman error de-
ﬁned in Equation 1. The BC loss encourages Qto output high values for demonstrated actions at
demonstrated states, and the penalty term propagates those high values to nearby states. In other
words,Qoutputs high values for actions that lead to states from which the demonstrated states are
reachable. Hence, when the agent ﬁnds itself far from the demonstrated states, it takes actions that
lead it back to the demonstrated states.
The RBC algorithm follows the same procedure as Algorithm 1, except that in line 4, RBC takes a
gradient step on the RBC loss from Equation 8 instead of the SQIL loss.
3.3 C ONNECTION BETWEEN SQIL AND REGULARIZED BEHAVIORAL CLONING
The gradient of the RBC loss in Equation 8 is proportional to the gradient of the SQIL loss in line
4 of Algorithm 1, plus an additional term that penalizes the soft value of the initial state s0(full
derivation in Section A.1 of the appendix):
r`RBC()/r
2(Ddemo;1) +samp2(Dsamp;0) +V(s0)
: (9)
In other words, SQIL solves a similar optimization problem to RBC. The reward function in SQIL
also has a clear connection to the sparsity prior in RBC: SQIL imposes the sparsity prior from RBC,
by training the agent with an extremely sparse reward function – r= +1 at the demonstrations, and
r= 0everywhere else. Thus, SQIL can be motivated as a practical way to implement the ideas for
regularizing BC proposed in Piot et al. (2014).
The main beneﬁt of using SQIL instead of RBC is that SQIL is trivial to implement, since it only
requires a few small changes to any standard Q-learning implementation (see Section 2). Extending
SQIL to MDPs with a continuous action space is also easy, since we can simply replace Q-learning
5
Published as a conference paper at ICLR 2020
with an off-policy actor-critic method (Haarnoja et al., 2018) (see Section 4.3). Given the difﬁculty
of implementing deep RL algorithms correctly (Henderson et al., 2018), this ﬂexibility makes SQIL
more practical to use, since it can be built on top of existing implementations of deep RL algorithms.
Furthermore, the ablation study in Section 4.4 suggests that SQIL actually performs better than RBC.
4 E XPERIMENTAL EVALUATION
Our experiments aim to compare SQIL to existing imitation learning methods on a variety of tasks
with high-dimensional, continuous observations, such as images, and unknown dynamics. We
benchmark SQIL against BC and GAIL4on four image-based games – Car Racing, Pong, Break-
out, and Space Invaders – and three state-based tasks – Humanoid, HalfCheetah, and Lunar Lander
(Brockman et al., 2016; Bellemare et al., 2013; Todorov et al., 2012). We also investigate which
components of SQIL contribute most to its performance via an ablation study on the Lunar Lander
game. Section A.3 in the appendix contains additional experimental details.
4.1 T ESTING GENERALIZATION IN IMAGE -BASED CARRACING
The goal of this experiment is to study not only how well each method can mimic the expert demon-
strations, but also how well they can acquire policies that generalize to new states that are not seen
in the demonstrations. To do so, we train the imitation agents in an environment with a different
initial state distribution Strain
0than that of the expert demonstrations Sdemo
0, allowing us to system-
atically control the mismatch between the distribution of states in the demonstrations and the states
actually encountered by the agent. We run experiments on the Car Racing game from OpenAI Gym.
To createStrain
0, the car is rotated 90 degrees so that it begins perpendicular to the track, instead of
parallel to the track as in Sdemo
0. This intervention presents a signiﬁcant generalization challenge to
the imitation learner, since the expert demonstrations do not contain any examples of states where
the car is perpendicular to the road, or even signiﬁcantly off the road axis. The agent must learn to
make a tight turn to get back on the road, then stabilize its orientation so that it is parallel to the road,
and only then proceed forward to mimic the expert demonstrations.
Domain Shift (Strain
0) No Shift (Sdemo
0)
Random2156684
BC4518 69810
GAIL-DQL973668
SQIL (Ours) 37519 704 6
Expert 48011 704 79
Figure 1: Average reward on 100 episodes after training.
Standard error on three random seeds.The results in Figure 1 show that
SQIL and BC perform equally well
when there is no variation in the ini-
tial state. The task is easy enough
that even BC achieves a high reward.
Note that, in the unperturbed condi-
tion (right column), BC substantially
outperforms GAIL, despite the well-
known shortcomings of BC. This in-
dicates that the adversarial optimiza-
tion in GAIL can substantially hinder
learning, even in settings where standard BC is sufﬁcient. SQIL performs much better than BC when
starting fromStrain
0, showing that SQIL is capable of generalizing to a new initial state distribution,
while BC is not. SQIL learns to make a tight turn that takes the car through the grass and back
onto the road, then stabilizes the car’s orientation so that it is parallel to the track, and then proceeds
forward like the expert does in the demonstrations. BC tends to drive straight ahead into the grass
instead of turning back onto the road.
4For all the image-based tasks, we implement a version of GAIL that uses deep Q-learning (GAIL-DQL)
instead of TRPO as in the original GAIL paper (Ho & Ermon, 2016), since Q-learning performs better than
TRPO in these environments, and because this allows for a head-to-head comparison of SQIL and GAIL:
both algorithms use the same underlying RL algorithm, but provide the agent with different rewards – SQIL
provides constant rewards, while GAIL provides learned rewards. We use the standard GAIL-TRPO method as
a baseline for all the low-dimensional tasks, since TRPO performs better than Q-learning in these environments.
The original GAIL method implicitly encodes prior knowledge – namely, that terminating an episode is either
always desirable or always undesirable. As pointed out in Kostrikov et al. (2019), this makes comparisons to
alternative methods unfair. We implement the unbiased version of GAIL proposed by Kostrikov et al. (2019),
and use this in all of the experiments. Comparisons to the biased version with implicit termination knowledge
are included in Section A.2 in the appendix.
6
Published as a conference paper at ICLR 2020
0 2000 4000 6000 8000
Number of On-Policy Rollouts0100200300400RewardBreakout
SQIL (Ours)
GAIL-DQL
BC (P’91)
Expert
0 500 1000 1500 2000
Number of On-Policy Rollouts−20−1001020RewardPong
0 1000 2000 3000 4000
Number of On-Policy Rollouts200400600RewardSpace Invaders
Figure 2: Image-based Atari. Smoothed with a rolling window of 100 episodes. Standard error
on three random seeds. X-axis represents amount of interaction with the environment (not expert
demonstrations).
SQIL outperforms GAIL in both conditions. Since SQIL and GAIL both use deep Q-learning for RL
in this experiment, the gap between them may be attributed to the difference in the reward functions
they use to train the agent. SQIL beneﬁts from providing a constant reward that does not require
ﬁtting a discriminator, while GAIL struggles to train a discriminator to provide learned rewards
directly from images.
4.2 I MAGE -BASED EXPERIMENTS ON ATARI
The results in Figure 2 show that SQIL outperforms BC on Pong, Breakout, and Space Invaders
– additional evidence that BC suffers from compounding errors, while SQIL does not. SQIL also
outperforms GAIL on all three games, illustrating the difﬁculty of using GAIL to train an image-
based discriminator, as in Section 4.1.
4.3 I NSTANTIATING SQIL FOR CONTINUOUS CONTROL IN LOW-DIMENSIONAL MUJOCO
0 20 40
Number of Demonstration Rollouts05001000Reward
HalfCheetah-v1
Expert
GAIL-TRPO (HE’16)
BC (P’91)
SQIL (Ours)
0 20 40
Number of Demonstration Rollouts0200400600800Reward
Humanoid-v1
Figure 3: SQIL: best per-
formance on 10 consecutive
training episodes. BC, GAIL:
results from Dhariwal et al.
(2017).The experiments in the previous sections evaluate SQIL on MDPs
with a discrete action space. This section illustrates how SQIL can
be adapted to continuous actions. We instantiate SQIL using soft
actor-critic (SAC) – an off-policy RL algorithm that can solve con-
tinuous control tasks (Haarnoja et al., 2018). In particular, SAC
is modiﬁed in the following ways: (1) the agent’s experience re-
play buffer is initially ﬁlled with expert demonstrations, where re-
wards are set to r= +1 , (2) when taking gradient steps to ﬁt the
agent’s soft Q function, a balanced number of demonstration ex-
periences and new experiences (50% each) are sampled from the
replay buffer, and (3) the agent observes rewards of r= 0 during
its interactions with the environment, instead of an extrinsic reward
signal that speciﬁes the desired task. This instantiation of SQIL is
compared to GAIL on the Humanoid (17 DoF) and HalfCheetah (6
DoF) tasks from MuJoCo.
The results show that SQIL outperforms BC and performs compa-
rably to GAIL on both tasks, demonstrating that SQIL can be suc-
cessfully deployed on problems with continuous actions, and that
SQIL can perform well even with a small number of demonstra-
tions. This experiment also illustrates how SQIL can be run on top
of SAC or any other off-policy value-based RL algorithm.
4.4 A BLATION STUDY ON LOW-DIMENSIONAL LUNAR LANDER
We hypothesize that SQIL works well because it combines information about the expert’s policy
from demonstrations with information about the environment dynamics from rollouts of the imi-
tation policy periodically sampled during training. We also expect RBC to perform comparably to
SQIL, since their objectives are similar. To test these hypotheses, we conduct an ablation study using
the Lunar Lander game from OpenAI Gym. As in Section 4.1, we control the mismatch between the
7
Published as a conference paper at ICLR 2020
distribution of states in the demonstrations and the states encountered by the agent by manipulating
the initial state distribution. To create Strain
0, the agent is placed in a starting position never visited in
the demonstrations.
In the ﬁrst variant of SQIL, sampis set to zero, to prevent SQIL from using additional samples drawn
from the environment (see line 4 of Algorithm 1). This comparison tests if SQIL really needs to
interact with the environment, or if it can rely solely on the demonstrations. In the second condition,

is set to zero to prevent SQIL from accessing information about state transitions (see Equation 1
and line 4 of Algorithm 1). This comparison tests if SQIL is actually extracting information about
the dynamics from the samples, or if it can perform just as well with a na ¨ıve regularizer (setting

to zero effectively imposes a penalty on the L2-norm of the soft Q values instead of the squared
soft Bellman error). In the third condition, a uniform random policy is used to sample additional
rollouts, instead of the imitation policy (see line 6 of Algorithm 1). This comparison tests how
important it is that the samples cover the states encountered by the agent during training. In the
fourth condition, we use RBC to optimize the loss in Equation 8. instead of using SQIL to optimize
the loss in line 4 of Algorithm 1. This comparison tests the effect of the additional V(s0)term in
RBC vs. SQIL (see Equation 9).
Domain Shift (Strain
0) No Shift (Sdemo
0)
Random 0:100:30 0 :040:02
BC 0:070:03 0:930:03
GAIL-TRPO 0:670:04 0:930:03
SQIL (Ours) 0:890:02 0:880:03Ablationsamp= 0 0:120:02 0:870:02

= 0 0 :410:02 0:840:02
= Unif 0 :470:02 0 :820:02
RBC 0:660:02 0:890:01
Expert 0:930:03 0 :890:31
Figure 4: Best success rate on 100 consecutive episodes dur-
ing training. Standard error on ﬁve random seeds. Perfor-
mance bolded if at least within one standard error of expert.The results in Figure 4 show that all
methods perform well when there is
no variation in the initial state. When
the initial state is varied, SQIL per-
forms signiﬁcantly better than BC,
GAIL, and the ablated variants of
SQIL. This conﬁrms our hypothesis
that SQIL needs to sample from the
environment using the imitation pol-
icy, and relies on information about
the dynamics encoded in the samples.
Surprisingly, SQIL outperforms RBC
by a large margin, suggesting that the
penalty on the soft value of the initial
stateV(s0), which is present in RBC but not in SQIL (see Equation 9), degrades performance.
5 D ISCUSSION AND RELATED WORK
Related work. Concurrently with SQIL, two other imitation learning algorithms that use constant
rewards instead of a learned reward function were developed (Sasaki et al., 2019; Wang et al., 2019).
We see our paper as contributing additional evidence to support this core idea, rather than proposing
a competing method. First, SQIL is derived from sparsity-regularized BC, while the prior meth-
ods are derived from an alternative formulation of the IRL objective (Sasaki et al., 2019) and from
support estimation methods (Wang et al., 2019), showing that different theoretical approaches inde-
pendently lead to using RL with constant rewards as an alternative to adversarial training – a sign
that this idea may be a promising direction for future work. Second, SQIL is shown to outperform
BC and GAIL in domains that were not evaluated in Sasaki et al. (2019) or Wang et al. (2019) – in
particular, tasks with image observations and signiﬁcant shift in the state distribution between the
demonstrations and the training environment.
Summary. We contribute the SQIL algorithm: a general method for learning to imitate an expert
given action demonstrations and access to the environment. Simulation experiments on tasks with
high-dimensional, continuous observations and unknown dynamics show that our method outper-
forms BC and achieves competitive results compared to GAIL, while being simple to implement on
top of existing off-policy RL algorithms.
Limitations and future work. We have not yet proven that SQIL matches the expert’s state occu-
pancy measure in the limit of inﬁnite demonstrations. One direction for future work would be to
rigorously show whether or not SQIL has this property. Another direction would be to extend SQIL
to recover not just the expert’s policy, but also their reward function; e.g., by using a parameterized
reward function to model rewards in the soft Bellman error terms, instead of using constant rewards.
This could provide a simpler alternative to existing adversarial IRL algorithms.
8
Published as a conference paper at ICLR 2020
REFERENCES
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from
demonstration. Robotics and autonomous systems , 57(5):469–483, 2009.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An
evaluation platform for general agents. Journal of Artiﬁcial Intelligence Research , 47:253–279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech
Zaremba. Openai gym, 2016.
Giuseppe Calaﬁore and Fabrizio Dabbene. Probabilistic and randomized methods for design under uncertainty .
Springer, 2006.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John
Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://github.com/
openai/baselines , 2017.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversar-
ial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852 ,
2016a.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy
optimization. In International Conference on Machine Learning , pp. 49–58, 2016b.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement
learning. arXiv preprint arXiv:1710.11248 , 2017.
Yang Gao, Ji Lin, Fisher Yu, Sergey Levine, Trevor Darrell, et al. Reinforcement learning from imperfect
demonstrations. arXiv preprint arXiv:1802.05313 , 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron
Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information process-
ing systems , pp. 2672–2680, 2014.
David Ha and J ¨urgen Schmidhuber. Recurrent world models facilitate policy evolution. arXiv preprint
arXiv:1809.01999 , 2018.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-
based policies. arXiv preprint arXiv:1702.08165 , 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum
entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290 , 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep rein-
forcement learning that matters. In Thirty-Second AAAI Conference on Artiﬁcial Intelligence , 2018.
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan,
Andrew Sendonaris, Gabriel Dulac-Arnold, et al. Deep q-learning from demonstrations. arXiv preprint
arXiv:1704.03732 , 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information
Processing Systems , pp. 4565–4573, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980 , 2014.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson.
Discriminator-actor-critic: Addressing sample inefﬁciency and reward bias in adversarial imitation learn-
ing. In International Conference on Learning Representations , 2019. URL https://openreview.
net/forum?id=Hk4fpoA5Km .
Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. The gan landscape: Losses,
architectures, regularization, and normalization. arXiv preprint arXiv:1807.04720 , 2018.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv
preprint arXiv:1805.00909 , 2018.
9
Published as a conference paper at ICLR 2020
V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex
Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep
reinforcement learning. Nature , 518(7540):529, 2015.
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, pp. 663–670,
2000.
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Learning from demonstrations: Is it worth estimating a reward
function? In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pp.
17–32. Springer, 2013.
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted and reward-regularized classiﬁcation for apprentice-
ship learning. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent
systems , pp. 1249–1256. International Foundation for Autonomous Agents and Multiagent Systems, 2014.
Dean A Pomerleau. Efﬁcient training of artiﬁcial neural networks for autonomous navigation. Neural Compu-
tation , 3(1):88–97, 1991.
St´ephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured pre-
diction to no-regret online learning. In Proceedings of the fourteenth international conference on artiﬁcial
intelligence and statistics , pp. 627–635, 2011.
Fumihiro Sasaki, Tetsuya Yohira, and Atsuo Kawaguchi. Sample efﬁcient imitation learning for continuous
control. In International Conference on Learning Representations , 2019. URL https://openreview.
net/forum?id=BkN5UoAqF7 .
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In In-
telligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on , pp. 5026–5033. IEEE,
2012.
Ruohan Wang, Carlo Ciliberto, Pierluigi Amadori, and Yiannis Demiris. Random expert distillation: Imitation
learning via expert policy support estimation. arXiv preprint arXiv:1905.06750 , 2019.
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforcement learn-
ing. arXiv preprint arXiv:1507.04888 , 2015.
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum
causal entropy. In Proceedings of the 27th International Conference on International Conference on Machine
Learning , pp. 1255–1262. Omnipress, 2010.
10
Published as a conference paper at ICLR 2020
A A PPENDIX
A.1 D ERIVATION OF RBC G RADIENT
Let= (s0;a0;s1;:::;sT)denote a rollout, where sTis an absorbing state. Let Vdenote the soft
value function,
V(s),log X
a2Aexp (Q(s;a))!
: (10)
Splitting up the squared soft Bellman error terms for DdemoandDsampin Equation 8,
r`RBC() =X
2D demoT1X
t=0(rQ(st;at)rV(st))
+demoX
2D demoT1X
t=0r(Q(st;at)
V(st+1))2+sampr2(Dsamp;0)
=X
2D demoT1X
t=0rV(st)
rV(st+1)
+demor2
Ddemo;1
2demo
+sampr2(Dsamp;0): (11)
Setting
,1turns the inner sum in the ﬁrst term into a telescoping sum:
(11) =X
2D demo(rV(s0)rV(sT)) +demor2
Ddemo;1
2demo
+sampr2(Dsamp;0):(12)
SincesTis assumed to be absorbing, V(sT)is zero. Thus,
(12) =X
s02D demorV(s0) +demor2
Ddemo;1
2demo
+sampr2(Dsamp;0); (13)
In our experiments, we have that all the demonstration rollouts start at the same initial state s0.5
Thus,
(13)/r
2(Ddemo;1) +samp2(Dsamp;0) +V(s0)
; (14)
wheresamp2R0is a constant hyperparameter.
A.2 C OMPARING THE BIASED AND UNBIASED VARIANTS OF GAIL
As discussed in Section 4, to correct the original GAIL method’s biased handling of rewards at
absorbing states, we implement the suggested changes to GAIL in Section 4.2 of Kostrikov et al.
(2019): adding a transition to an absorbing state and a self-loop at the absorbing state to the end of
each rollout sampled from the environment, and adding a binary feature to the observations indicat-
ing whether or not a state is absorbing. This enables GAIL to learn a non-zero reward for absorbing
states. We refer to the original, biased GAIL method as GAIL-DQL-B and GAIL-TRPO-B, and the
unbiased version as GAIL-DQL-U and GAIL-TRPO-U.
The mechanism for learning terminal rewards proposed in Kostrikov et al. (2019) does not apply to
SQIL, since SQIL does not learn a reward function. SQIL implicitly assumes a reward of zero at
absorbing states in demonstrations. This is the case in all our experiments, which include some envi-
ronments where terminating the episode is always undesirable (e.g., walking without falling down)
and other environments where success requires terminating the episode (e.g., landing at a target),
suggesting that SQIL is not sensitive to the choice of termination reward, and neither signiﬁcantly
beneﬁts nor is signiﬁcantly harmed by setting the termination reward to zero.
5Demonstration rollouts may still vary due to the stochasticity of the expert policy.
11
Published as a conference paper at ICLR 2020
Domain Shift (Strain
0) No Shift (Sdemo
0)
Random2156684
BC4518 69810
GAIL-DQL-B9143421
GAIL-DQL-U973668
SQIL (Ours) 37519 704 6
Expert 48011 704 79
Figure 5: Image-based Car Racing. Average reward on 100 episodes after training. Standard error
on three random seeds.
0 2000 4000 6000 8000
Number of On-Policy Rollouts0200400RewardBreakout
SQIL (Ours)
GAIL-DQL-B
GAIL-DQL-U
BC (P’91)
Expert
0 500 1000 1500 2000
Number of On-Policy Rollouts−20−1001020RewardPong
0 2000 4000 6000
Number of On-Policy Rollouts200400600RewardSpace Invaders
Figure 6: Image-based Atari. Smoothed with a rolling window of 100 episodes. Standard error
on three random seeds. X-axis represents amount of interaction with the environment (not expert
demonstrations).
Car Racing. The results in Figure 5 show that both the biased (GAIL-DQL-B) and unbiased (GAIL-
DQL-U) versions of GAIL perform equally poorly. The problem of training an image-based discrim-
inator for this task may be difﬁcult enough that even with an unfair bias toward avoiding crashes that
terminate the episode, GAIL-DQL-B does not perform better than GAIL-DQL-U.
Atari. The results in Figure 6 show that SQIL outperforms both variants of GAIL on Pong and the
unbiased version of GAIL (GAIL-DQL-U) on Breakout and Space Invaders, but performs compara-
bly to the biased version of GAIL (GAIL-DQL-B) on Space Invaders and worse than it on Breakout.
This may be due to the fact that in Breakout and Space Invaders, the agent has multiple lives –
ﬁve in Breakout, and three in Space Invaders – and receives a termination signal that the episode
has ended after losing each life. Thus, the agent experiences many more episode terminations than
in Pong, exacerbating the bias in the way the original GAIL method handles rewards at absorb-
ing states. Our implementation of GAIL-DQL-B in this experiment provides a learned reward of
r(s;a) =log (1D(s;a)), whereDis the discriminator (see Section A.3 in the appendix for
details). The learned reward is always positive, while the implicit reward at an absorbing state is
zero. Thus, the agent is inadvertently encouraged to avoid terminating the episode. For Breakout
and Space Invaders, this just happens to be the right incentive, since the objective is to stay alive as
long as possible. GAIL-DQL-B outperforms SQIL in Breakout and performs comparably to SQIL
in Space Invaders because GAIL-DQL-B is accidentally biased in the right way.
Lunar Lander. The results in Figure 7 show that when the initial state is varied, SQIL outperforms
the unbiased variant of GAIL (GAIL-TRPO-U), but underperforms against the biased version of
GAIL (GAIL-TRPO-B). The latter result is likely due to the fact that the implementation of GAIL-
TRPO-B we used in this experiment provides a learned reward of r(s;a) = log (D(s;a)), where
Dis the discriminator (see Section A.3 in the appendix for details). The learned reward is always
negative, while the implicit reward at an absorbing state is zero. Thus, the agent is inadvertently
encouraged to terminate the episode quickly. For the Lunar Lander game, this just happens to be
the right incentive, since the objective is to land on the ground and thereby terminate the episode.
As in the Atari experiments, GAIL-TRPO-B performs better than SQIL in this experiment because
GAIL-TRPO-B is accidentally biased in the right way.
12
Published as a conference paper at ICLR 2020
Domain Shift (Strain
0) No Shift (Sdemo
0)
Random 0:100:30 0 :040:02
BC 0:070:03 0:930:03
GAIL-TRPO-B (HE’16) 0:980:01 0:950:02
GAIL-TRPO-U 0:670:04 0:930:03
SQIL (Ours) 0:890:02 0:880:03Ablationsamp= 0 0 :120:02 0:870:02

= 0 0 :410:02 0:840:02
= Unif 0 :470:02 0 :820:02
RBC 0:660:02 0:890:01
Expert 0:930:03 0 :890:31
Figure 7: Low-dimensional Lunar Lander. Best success rate on 100 consecutive episodes during
training. Standard error on ﬁve random seeds. Performance bolded if at least within one standard
error of expert.
0 50000 100000 150000
Number of Training Steps50100150Squared Bellman Error
0200400600
RewardHumanoid-v1
Figure 8: Standard error over two random seeds. No smoothing across training steps.
A.3 I MPLEMENTATION DETAILS
To ensure fair comparisons, the same network architectures were used to evaluate SQIL, GAIL, and
BC. For Lunar Lander, we used a network architecture with two fully-connected layers containing
128 hidden units each to represent the Q network in SQIL, the policy and discriminator networks in
GAIL, and the policy network in BC. For Car Racing, we used four convolutional layers (following
(Ha & Schmidhuber, 2018)) and two fully-connected layers containing 256 hidden units each. For
Humanoid and HalfCheetah, we used two fully-connected layers containing 256 hidden units each.
For Atari, we used the convolutional neural network described in (Mnih et al., 2015) to represent the
Q network in SQIL, as well as the Q network and discriminator network in GAIL.
To ensure fair comparisons, the same demonstration data were used to train SQIL, GAIL, and BC.
For Lunar Lander, we collected 100 demonstration rollouts. For Car Racing, Pong, Breakout, and
Space Invaders, we collected 20 demonstration rollouts. Expert demonstrations were generated
from scratch for Lunar Lander using DQN (Mnih et al., 2015), and collected from open-source pre-
trained policies for Car Racing (Ha & Schmidhuber, 2018) as well as Humanoid and HalfCheetah
(Dhariwal et al., 2017). The Humanoid demonstrations were generated by a stochastic expert policy,
while the HalfCheetah demonstrations were generated by a deterministic expert policy; both experts
were trained using TRPO.6We used two open-source implementations of GAIL: (Fu et al., 2017)
for Lunar Lander, and (Dhariwal et al., 2017) for MuJoCo. We adapted the OpenAI Baselines
implementation of GAIL to use soft Q-learning for Car Racing and Atari. Expert demonstrations
were generated from scratch for Atari using DQN.
For Lunar Lander, we set samp= 106. For Car Racing, we set samp= 0:01. For all other
environments, we set samp= 1.
6https://drive.google.com/drive/folders/1h3H4AY_ZBx08hz-Ct0Nxxus-V1melu1U
13
Published as a conference paper at ICLR 2020
SQIL was not pre-trained in any of the experiments. GAIL was pre-trained using BC for HalfChee-
tah, but was not pre-trained in any other experiments.
In standard implementations of soft Q-learning and SAC, the agent’s experience replay buffer typ-
ically has a ﬁxed size, and once the buffer is full, old experiences are deleted to make room for
new experiences. In SQIL, we never delete demonstration experiences from the replay buffer, but
otherwise follow the standard implementation.
We use Adam (Kingma & Ba, 2014) to take the gradient step in line 4 of Algorithm 1.
The BC and GAIL performance metrics in Section 4.3 are taken from (Dhariwal et al., 2017).7
The GAIL and SQIL policies in Section 4.3 are set to be deterministic during the evaluation rollouts
used to measure performance.
7https://github.com/openai/baselines/blob/master/baselines/gail/result/
gail-result.md
14

Siddharth Reddy
Anca D
Dragan
Sergey Levine