Responsive safety in reinforcement learning by pid lagrangian methods

Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
Adam Stooke1 2Joshua Achiam1 2Pieter Abbeel1
Abstract
Lagrangian methods are widely used algorithms
for constrained optimization problems, but their
learning dynamics exhibit oscillations and over-
shoot which, when applied to safe reinforcement
learning, leads to constraint-violating behavior
during agent training. We address this shortcom-
ing by proposing a novel Lagrange multiplier up-
date method that utilizes derivatives of the con-
straint function. We take a controls perspective,
wherein the traditional Lagrange multiplier up-
date behaves as integral control; our terms intro-
duce proportional andderivative control, achiev-
ing favorable learning dynamics through damp-
ing and predictive measures. We apply our PID
Lagrangian methods in deep RL, setting a new
state of the art in Safety Gym, a safe RL bench-
mark. Lastly, we introduce a new method to ease
controller tuning by providing invariance to the
relative numerical scales of reward and cost. Our
extensive experiments demonstrate improved per-
formance and hyperparameter robustness, while
our algorithms remain nearly as simple to derive
and implement as the traditional Lagrangian ap-
proach.
1. Introduction
Reinforcement learning has solved sequential decision tasks
of impressive difﬁculty by maximizing reward functions
through trial and error. Recent examples using deep learn-
ing range from robotic locomotion ( Schulman et al. ,2015 ;
Gu et al. ,2016 ;Schulman et al. ,2017 ;Levine et al. ,2016 )
to sophisticated video games ( Mnih et al. ,2013 ;Schulman
et al. ,2017 ;OpenAI ,2018 ;Jaderberg et al. ,2019 ). While
errors during training in these domains come without cost,
in some learning scenarios it is important to limit the rates of
hazardous outcomes. One example would be wear and tear
1University of California, Berkeley2OpenAI. Correspondence
to: Adam Stooke .
Proceedings of the 37thInternational Conference on Machine
Learning , Vienna, Austria, PMLR 119, 2020. Copyright 2020 by
the author(s).on a robot’s components or its surroundings. It may not be
possible to impose such limits by prescribing constraints in
the action or state space directly; instead, hazard-avoiding
behavior must be learned. For this purpose, we use the
well-known framework of the constrained Markov decision
process (CMDP) ( Altman ,1999 ), which limits the accumu-
lation of a “cost” signal which is analogous to the reward.
The optimal policy is one which maximizes the usual re-
turn while satisfying the cost constraint. In safe RL the
agent must avoid hazards not only at convergence, but also
throughout exploration and learning.
Lagrangian methods are a classic approach to solving con-
strained optimization problems. For example, the equality-
constrained problem over the real vector x:
min
xf(x)s.t.g(x) = 0 (1)
is transformed into an unconstrained one by introduction
of a dual variable–the Lagrange multiplier, λ–to form the
Lagrangian:L(x,λ) =f(x)+λg(x), which is used to ﬁnd
the solution as:
(x∗,λ∗) = argmax
λmin
xL(x,λ) (2)
Gradient-based algorithms iteratively update the primal and
dual variables:
−∇xL(x,λ) =−∇xf(x)−λ∇xg(x) (3)
∇λL(x,λ) =g(x) (4)
so thatλacts as a learned penalty coefﬁcient in the objective,
leading eventually to a constraint-satisfying solution (see
e.g.Bertsekas (2014 )). The Lagrangian multiplier method
is readily adapted to the constrained RL setting ( Altman ,
1998 ;Geibel & Wysotzki ,2011 ) and has become a popular
baseline in deep RL ( Achiam et al. ,2017 ;Chow et al. ,2019 )
for its simplicity and effectiveness.
Although they have been shown to converge to optimal,
constraint-satisfying policies ( Tessler et al. ,2018 ;Paternain
et al. ,2019 ), a shortcoming of gradient Lagrangian methods
for safe RL is that intermediate iterates often violate con-
straints. Cost overshoot and oscillations are in fact inherent
to the learning dynamics ( Platt & Barr ,1988 ;Wah et al. ,
2000 ), and we witnessed numerous problematic cases in our
own experiments. Figure 1(left) shows an example from a
Responsive Safety in RL by PID Lagrangian Methods
deep RL setting, where the cost and multiplier values oscil-
lated throughout training. Our key insight in relation to this
deﬁciency is that the traditional Lagrange multiplier update
in (4) amounts to integral control on the constraint. The
90-degree phase shift between the curves is characteristic
of ill-tuned integral controllers.
Our contribution is to expand the scope of possible Lagrange
multiplier update rules beyond ( 4), by interpreting the over-
all learning algorithm as a dynamical system. Speciﬁcally,
we employ the next simplest mechanisms, proportional and
derivative control, to λ, by adding terms corresponding to
derivatives of the constraint function into ( 4) (derivatives
with respect to learning iteration). To our knowledge, this
is the ﬁrst time that an expanded update rule has been con-
sidered for a learned Lagrange multiplier. PID control is
an appealing enhancement, evidenced by the fact that it is
one of the most widely used and studied control techniques
(˚Astr¨om & H ¨agglund ,2006 ). The result is a more respon-
sive safety mechanism, as demonstrated in Figure 1(right ),
where the cost oscillations have been damped, dramatically
reducing violations.
0
 2
 4
Agent Steps 1e7
180
200
220
240g(x)Δ ≈ 90∘Traditional Lagrangian
0
 2
 4
Agent Steps 1e7
180
200
220
240g(x)PID Lagrangian
0.5
1.0
1.5
λ
0.5
1.0
1.5
λ
Figure 1. Left: The traditional Lagrangian method exhibits oscilla-
tions with 90◦phase shift between the constraint function and the
Lagrange multiplier, characteristic of integral control. Right : PID
control on the Lagrange multiplier damps oscillations and obeys
constraints. Environment: D OGGO BUTTON 1, cost limit 200.
Our contributions in this paper are outlined as follows. First,
we provide further context through related works and pre-
liminary deﬁnitions. In Section 4, we propose modiﬁed
Lagrangian multiplier methods and analyze their beneﬁts
in the learning dynamics. Next, in Section 5, we cast con-
strained RL as a dynamical system with the Lagrange multi-
plier as a control input, to which we apply PID control as
a new algorithm. In Section 6, we adapt a leading deep RL
algorithm, Proximal Policy Optimization (PPO) ( Schulman
et al. ,2017 ) with our methods and achieve state of the art
performance in the OpenAI Safety-Gym suite of environ-
ments ( Ray et al. ,2019 ). Finally, in Section 7we introduce
another novel technique that makes tuning easier by provid-
ing invariance to the relative numerical scales of rewards and
costs, and we demonstrate it in a further set of experiments.
Our extensive empirical results show that our algorithms,which are intuitive and simple to implement, improve cost
performance and promote hyperparameter robustness in a
deep RL setting.
2. Related Work
Constrained Deep RL. Adaptations of the Lagrange multi-
plier method to the actor-critic RL setting have been shown
to converge to the optimal, constraint-satisfying solution
under certain assumptions ( Tessler et al. ,2018 ). Conver-
gence proofs have relied upon updating the multiplier more
slowly than the policy parameters ( Tessler et al. ,2018 ;Pa-
ternain et al. ,2019 ), implying many constraint-violating
policy iterations may occur before the penalty comes into
full effect.
Several recent works have aimed at improving constraint
satisfaction in RL over the Lagrangian method, but they
tend to incur added complexity. Achiam et al. (2017 ) in-
troduced Constrained Policy Optimization (CPO), a policy
search algorithm with near-constraint satisfaction guaran-
tees at every iteration, based on a new bound on the expected
returns of two nearby policies. CPO includes a projection
step on the policy parameters, which in practice requires
a time-consuming backtracking line search. Yet, simple
Lagrangian-based algorithms performed as well or better in
a recent empirical comparison in Safety Gym ( Ray et al. ,
2019 ). Approaches to safe RL based on Lyapunov func-
tions have been developed in a series of studies ( Chow et al. ,
2018 ;2019 ), resulting in algorithms that combine a projec-
tion step, as in CPO, with action-layer interventions like the
safety layer of Dalal et al. (2018 ). Experimentally, this line
of work showed mixed performance gains over Lagrangian
methods, at a nontrivial cost to implement and without clear
guidance for tuning. Liu et al. (2019 ) developed interior
point methods for RL, which augment the objective with
logarithmic barrier functions. These methods are shown
theoretically to provide suboptimal solutions. Furthermore,
they require tuning of the barrier strength and typically
assume already feasible iterates, the latter point possibly
being problematic for random agent initializations or under
noisy cost estimates. Most recently, Yang et al. (2020 ) ex-
tended CPO with a two-step projection-based optimization
approach. In contrast to these techniques, our method re-
mains nearly as simple to implement and compute as the
baseline Lagrangian method.
Dynamical Systems View of Optimization. Several recent
works have proposed different dynamical systems view-
points to analyze optimization algorithms, including those
often applied to deep learning. Hu & Lessard (2017 ) rein-
terpreted ﬁrst-order gradient optimization as a dynamical
system; they likened the gradient of the objective, ∇xf,
to the plant, which the controller aims to drive to zero to
arrive at the optimal parameters, x∗. Basic gradient de-
Responsive Safety in RL by PID Lagrangian Methods
scent then matches the form of integral control (on ∇xf).
They extend the analogy to momentum-based methods, for
example linking Nesterov momentum to PID control with
lag compensation. In another example, An et al. (2018 )
interpreted SGD as P-control and momentum methods as
PI-control. They introduced a derivative term, based on
the change in the gradient, and applied their resulting PID
controller to improve optimization of deep convolutional
networks. Other recent works bring yet other perspectives
from dynamical systems to deep learning and optimization,
see for example ( Lessard et al. ,2014 ;Nishihara et al. ,2015 ;
Liu & Theodorou ,2019 )). None of these works address con-
strained RL, however, necessitating our distinct formulation
for that problem.
Constrained Optimization. Decades’ worth of literature
have accumulated on Lagrangian methods. But even recent
textbooks on the topic ( Bertsekas ,2014 ;Nocedal & Wright ,
2006 ) only consider updating the Lagrange multiplier us-
ing the value of the constraint function, g(x), and miss
ever using its derivatives, ˙g(x)or¨g(x), which we introduce.
The modiﬁcation to the Lagrangian method most similar
in effect to our proportional control term (here using ˙g(x))
is the quadratic penalty method ( Hestenes (1969 );Powell
(1969 ) see also e.g.Bertsekas (1976 )), which we compare
in Section 4.Song & Leland (1998 ) proposed a controls
viewpoint (continuous-time) of optimizing neural networks
for constrained problems and arrived at proportional control
rules only. Related to our ﬁnal experiments on reward-scale
invariance, Wah et al. (2000 ) developed an adaptive weight-
ing scheme for continuous-time Lagrangian objectives, but
it is an intricate procedure which is not straightforwardly
applied to safe RL.
3. Preliminaries
Constrained Reinforcement Learning Constrained
Markov Decision Processes (CMDP) ( Altman ,1998 )
extend MDPs (see Sutton & Barto (1998 )) to incorporate
constraints into reinforcement learning. A CMDP is the
expanded tuple (S,A,R,T,µ,C 0,C1,...,d0,d1,...), with
the cost functions Ci:S×A×S→Rdeﬁned by the
same form as the reward, and di:Rdenoting limits on the
costs. For ease of notation, we will only consider a single,
all-encompassing cost.
The expected sum of discounted rewards over tra-
jectories, τ= (s0,a0,s1,a1,...), induced by the
policyπ(a|s)is a common performance objective:
J(π) =Eτ∼π[/summationtext∞
t=0γtR(st,at,st+1)]. The analo-
gous value function for cost is deﬁned as: JC(π) =
Eτ∼π[/summationtext∞
t=0γtC(st,at,st+1)]. The constrained RL prob-
lem is to solve for the best feasible policy:
π∗= argmax
πJ(π)s.t.JC(π)≤d (5)Deep reinforcement learning uses a deep neural network
for the policy, πθ=π(·|s;θ)with parameter vector θ, and
policy gradient algorithms improve the policy iteratively
by gathering experience in the task to estimate the reward
objective gradient, ∇θJ(πθ). Thus our problem of interest
is better expressed as maximizing score at some iterate, πk,
while ideally obeying constraints at each iteration:
max
πJ(πk)
s.t.JC(πm)≤d m∈{0,1,...,k}(6)
Practical settings often allow trading reward performance
against some constraint violations ( e.g.the constraints them-
selves may include a safety margin). For this purpose we
introduce a constraint ﬁgure of merit with our experiments.
3.1. Dynamical Systems and Optimal Control
Dynamical systems are processes which can be subject to
an external inﬂuence, or control . A general formulation for
discrete-time systems with feedback control is:
xk+1=F(xk,uk)
yk=Z(xk)
uk=h(y0,...,yk)(7)
with state vector x, dynamics function F, measurement
outputsy, applied control u, and the subscript denoting
the time step. The feedback rule hhas access to past and
present measurements. A problem in optimal control is to
design a control rule, h, that results in a sequence y0:T.=
{y0,...,yT}(orx0:Tdirectly) that scores well according to
some cost function C. Examples include simply reaching
a goal condition, C=|yT−y|, or following close to a
desired trajectory, y0:T.
Systems with simpler dependence on the input are generally
easier to analyze and control ( i.e.simplerhperforms well),
even if the dependence on the state is complicated ( Skelton ,
1988 ). Control-afﬁne systems are a broad class of dynamical
systems which are especially amenable to analysis ( Isidori
et al. ,1995 ). They take the form:
F(xk,uk) =f(xk)+g(xk)uk (8)
wherefandgmay be nonlinear in state, and are possibly
uncertain , meaning unknown. We will seek control-afﬁne
form for ease of control and to support future analysis.
4. Modiﬁed Lagrangian Methods for
Constrained Optimization
Lagrangian methods are a classic family of approaches to
solving constrained optimization problems. We propose
an intuitive, previously overlooked form for the multiplier
Responsive Safety in RL by PID Lagrangian Methods
update and derive its beneﬁcial effect on the learning dy-
namics. We begin by reviewing a prior formulation for the
equality-constrained problem.1
4.1. Review: “Basic Differential Multiplier Method”
We follow the development of Platt & Barr (1988 ), who
analyzed the dynamics of a continuous-time neural learning
system applied to this problem (our result can similarly be
derived for iterative gradient methods). They begin with the
component-wise differential equations:
˙xi=−∂L(x,λ)
∂xi=−∂f
∂xi−λ∂g
∂xi(9)
˙λ=α∂L(x,λ)
∂λ=αg(x) (10)
where we have inserted the scalar constant αas a learning
rate onλ. Differentiating ( 9) and substituting with ( 10) leads
to the second-order dynamics, written in vector format:
¨x+A˙x+αg(x)∇g= 0 (11)
which is a forced oscillator with damping matrix equal to
the weighted sum of Hessians:
Aij=∂2f
∂xi∂xj+λ∂2g
∂xi∂xj,or,A=∇2f+λ∇2g(12)
Platt & Barr (1988 ) showed that if Ais positive deﬁnite,
then the system ( 11) converges to a solution that satisﬁes the
constraint. Platt & Barr (1988 ) also noted that the system
(9)-(10) is prone to oscillations as it converges into the
feasible region, with frequency and settling time depending
onα. We provide complete derivations of the dynamics in
(11) and for our upcoming methods in an appendix.
4.2. Proportional-Integral Multiplier Method
In (10),λsimply integrates the constraint. To improve the
dynamics towards more rapid and stable satisfaction of con-
straints, we introduce a new term in λthat is proportional to
the current constraint value. In the differential equation for
λ, this term appears as the time-derivative of the constraint:
˙λ=αg(x)+β˙g(x) =αg(x)+β/summationdisplay
j∂g
∂xj˙xj (13)
with strength coefﬁcient, β. Replacing ( 10) by ( 13) and
combining with ( 9) yields similar second-order dynamics as
(11), except with an additional term in the damping matrix:
¨x+/parenleftbig
A+β∇g∇⊤g/parenrightbig˙x+αg(x)∇g= 0 (14)
1Standard techniques extend our results to inequality con-
straints, and multiple constraints, as in Platt & Barr (1988 ), and
notation is simplest for an equality constraint.The new term is beneﬁcial because it is positive semi-
deﬁnite—being the outer product of a vector with itself—so
it can increase the damping eignevalues, boosting conver-
gence. The results of ( Platt & Barr ,1988 ) hold under ( 13,
14), because the conditions of the solution, namely ˙x= 0
andg(x) = 0 , remain unaffected and extend immediately to
˙g(x) = 0 (and for the sequel, to ¨g(x) = 0 ). To our knowl-
edge, this is the ﬁrst time that a proportional-integral update
rule has been considered for a learned Lagrange multiplier.
The well-known penalty method ( Hestenes ,1969 ;Powell ,
1969 ) augments the Lagrangian with an additional term,
c
2g(x)2, which produces a similar effect on the damping
matrix, as shown in ( Platt & Barr ,1988 ):
Apenalty=A+c∇g∇⊤g+cg(x)∇2g (15)
Our approach appears to provide the same beneﬁt, with-
out the following two complications of the penalty method.
First, the penalty term must be implemented in the deriva-
tive˙x, whereas our methods do not modify the Lagrangian
nor the derivative in ( 9). Second, the penalty introduces
another instance of the hessian ∇2gin the damping matrix,
which might not be positive semi-deﬁnite but shares the
proportionality factor, c, with the desired term.
4.3. Integral-Derivative Multiplier Method
A similar analysis extends to the addition of a term in λ
based on the derivative of the constraint value. It appears in
˙λas the second derivative of the constraint:
˙λ=αg(x)+γ¨g(x) (16)
with strength coefﬁcient γ. The resulting dynamics are:
¨x+B−1A˙x+/parenleftbig
αg(x)+γ˙x⊤∇2g˙x/parenrightbig
B−1∇g= 0 (17)
withB=/parenleftbig
I+γ∇g∇⊤g/parenrightbig
, andIthe identity matrix.
The effects of the derivative update method are two-fold.
First, since the eigenvalues of the matrix B−1will be less
than1, both the damping ( A) and forcing (∇g) terms are
weakened (and rotated, generally). Second, the new forcing
term can be interpreted as a drag quadratic in the speed and
modulated by the curvature of the constraint along the direc-
tion of motion. To illustrate cases, if the curvature of gis
positive along the direction of travel, then this term becomes
a force for decreasing g. If at the same time g(x)>0, then
the traditional force will also be directed to decrease g, so
the two will add. On the other hand, if gcurves negatively
along the velocity, then the new force promotes increasing
g; ifg(x)>0, then the two forces subtract, weakening the
acceleration ¨x. By using curvature, the derivative method
acts predictively, but may be prone to instability.
Theproportional-integral-derivative multiplier method
is the combination of the previous two developments, which
Responsive Safety in RL by PID Lagrangian Methods
induced independent changes in the dynamics ( i.e.insert
the damping matrix of ( 14) into ( 17)). We leave for future
work a more rigorous analysis of the effects of the new
terms, along with theoretical considerations of the values of
coefﬁcients α,β, andγ. In the next section, we carry the
intuitions from our analysis to make practical enhancements
to Lagrangian-based constrained RL algorithms.
5. Feedback Control for Constrained RL
We advance the broader consideration of possible multiplier
update rules by reinterpreting constrained RL as a dynam-
ical system; the adaptive penalty coefﬁcient is a control
input, and the cost threshold is a setpoint which the system
should maintain. As the agent learns for rewards, the up-
ward pressure on costs from reward-learning can change,
requiring dynamic response. In practical Lagrangian RL,
the iterates λkmay deviate from the optimal value, even for
lucky initialization λ0=λ∗, as the policy is only partially
optimized at each iteration. Adaptive sequences λ0,...,λK
other than those prescribed by the Lagrangian method may
achieve superior cost control for Problem ( 6). In this section
we relate the Lagrangian method to a dynamical system,
formalizing how to incorporate generic update rules using
feedback. We return to the case of an inequality constrained
CMDP to present our main algorithmic contribution—the
use of PID control to adapt the penalty coefﬁcient.
5.1. Constrained RL as a Dynamical System
We write constrained RL as the ﬁrst-order dynamical sys-
tem:
θk+1=F(θk,λk)
yk=JC(πθk)
λk=h(y0,...,yk,d)(18)
whereFis an unknown nonlinear function2corresponding
to the RL algorithm policy update on the agent’s parameter
vector,θ. The cost-objective serves as the system measure,
y, which is supplied to the feedback control rule, h, along
with cost limit, d. From this general starting point, both the
RL algorithm, F, and penalty coefﬁcient update rule, h, can
be tailored for solving Problem ( 6).
The reward and cost policy gradients of the ﬁrst-order3
Lagrangian method, ∇θL(θ,λ) =∇θJ(πθ)−λ∇θJC(πθ),
can be organized into the form of ( 18) as:
F(θk,λk) =f(θk)+g(θk)λk (19)
f(θk) =θk+η∇θJ(πθk) (20)
2Known as an “uncertain” nonlinear function in the control
literature, meaning we lack an analytical expression for it.
3We discuss only the ﬁrst-order case, which provides sufﬁcient
clarity for our developments.g(θk) =−η∇θJC(πθk) (21)
with SGD learning rate η. The role of the controller is to
drive inequality constraint violations (Jc−d)+to zero in
the presence of drift from reward-learning due to f. The
Lagrange multiplier update rule for an inequality constraint
uses subgradient descent:
λk+1= (λk+KI(JC−d))+ (22)
with learning rate KIand projection into λ≥0. This
update step is clearly an integral control rule, for h.
5.2. Constraint-Controlled RL
Our general procedure, constraint-controlled RL, is given in
Algorithm 1. It follows the typical minibatch-RL scheme,
and sampled estimates of the cost criterion, ˆJCare fed back
to control the Lagrange multiplier. In contrast to prior work
(Tessler et al. ,2018 ;Paternain et al. ,2019 ) which uses a
single value approximator and treats r+λcas the reward,
we use separate value- and cost-value approximators, since
λmay change rapidly.
Whenλis large, the update in ( 19) can cause excessively
large change in parameters, θ, destabilizing learning. To
maintain consistent step size, we use a re-scaled objective
for theθ-learning loop:
θ∗(λ) = argmax
θJ−λJC= argmax
θ1
1+λ(J−λJC)
This convex combination of objectives yields the policy
gradient used in Algorithm 1. Our experiments use this
re-scaling, including for traditional Lagrangian baselines.
Algorithm 1
Constraint-Controlled Reinforcement Learning
1:procedure CONSTRAINED RL(πθ0(·|s),d)
2: Initialize control rule (as needed)
3:JC←{} ⊲cost measurement history
4: repeat
5: Sample environment: ⊲a minibatch
6: a∼π(·|s;θ),s′∼T(s,a),
7: r∼R(s,a,s′),c∼C(s,a,s′)
8: Apply feedback control:
9: Store sample estimate ˆJCintoJC
10: λ←h(JC,d), λ≥0
11: Updateπby RL: ⊲by Lagrangian objective
12: Update critics, Vφ(s),VC,ψ(s)⊲if using
13:∇θL=1
1+λ/parenleftBig
∇θˆJ(πθ)−λ∇θˆJC(πθ)/parenrightBig
14: until converged
15: returnπθ
16:end procedure
As an aside, we note that it is possible to maintain the
control-afﬁne form of ( 19) with this re-scaling, by reparam-
Responsive Safety in RL by PID Lagrangian Methods
eterizing the control as 0≤u=λ
1+λ≤1and substituting
for (21) with:
g(θk) =−η∇θ(J(πθk)+JC(πθk)) (23)
This parameterization simply weights the reward and cost
gradients in the Lagrangian objective as:
∇θL(θ,λ) = (1−u)∇θJ(πθ)−u∇θJC(πθ) (24)
It may provide superior performance in some cases, as it
will behave differently in relation to the nonlinearity in
control which arises from the inequality constraint. We
leave experimentation with direct control on u∈[0,1]to
future work.
5.3. The PID Lagrangian Method
We now specify a new control rule for use in Algorithm
1. To overcome the shortcomings of integral-only control,
we follow the developments of the previous section and
introduce the next simplest components: proportional and
derivative terms. Our PID update rule to replace ( 22) is
shown in Algorithm 2. The proportional term will hasten
the response to constraint violations and dampen oscilla-
tions, as derived in Section 4. Unlike the Lagrangian update,
derivative control can act in anticipation of violations. It
can both prevent cost overshoot and limit the rate of cost
increases within the feasible region, useful when monitor-
ing a system for further safety interventions. Our derivative
term is projected as (·)+so that it acts against increases in
cost but does not impede decreases. Overall, PID control
provides a much richer set of controllers while remaining
nearly as simple to implement; setting KP=KD= 0re-
covers the traditional Lagrangian method. The integral term
remains necessary for eliminating steady-state violations
at convergence. Our experiments mainly focus on the ef-
fects of proportional and derivative control of the Lagrange
multiplier in constrained deep RL.
Algorithm 2 PID-Controlled Lagrange Multiplier
1:Choose tuning parameters: KP,KI,KD≥0
2:Integral:I←0
3:Previous Cost: JC,prev←0
4:repeat at each iteration k
5: Receive cost JC
6:∆←JC−d
7:∂←(JC−JC,prev)+
8:I←(I+∆)+
9:λ←(KP∆+KII+KD∂)+
10:JC,prev←JC
11: returnλ6. PID Control Experiments
We investigated the performance of our algorithms on Prob-
lem ( 6) in a deep RL setting. In particular, we show the
effectiveness of PID control at reducing constraint viola-
tions from oscillations and overshoot present in the base-
line Lagrangian method. Both maximum performance and
robustness to hyperparameter selection are considered. Al-
though many methods exist for tuning PID parameters, we
elected to do so manually, demonstrating ease of use.
6.1. Environments: Safety-Gym
We use the recent Safety-Gym suite ( Ray et al. ,2019 ), which
consists of robot locomotion tasks built on the MuJoCo sim-
ulator ( Todorov et al. ,2012 ). The robots range in complexity
from a simple Point robot to the 12-jointed Doggo, and they
move in an open arena ﬂoor. Rewards have a small, dense
component encouraging movement toward the goal, and a
large, sparse component for achieving it. When a goal is
achieved, a new goal location is randomly generated, and
the episode continues until the time limit at 1,000 steps.
Each task has multiple difﬁculty levels corresponding to
density and type of hazards, which induce a cost when
contacted by the robot (without necessarily hindering its
movement). Hazards are placed randomly at each episode
and often lay in the path to the goal. Hence the aims of
achieving high rewards and low costs are in opposition. The
robot senses the position of hazards and the goal through a
coarse, LIDAR-like mode. The output of this sensor, along
with internal readings like the joint positions and velocities,
comprises the state fed to the agent. Figure 2displays a
scene from the D OGGO GOAL1 environment.
Figure 2. Rendering from the D OGGO GOAL1 environment from
Safety Gym. The red, four-legged robot must walk to the green
cylinder while avoiding other objects, and receives coarse egocen-
tric sensor readings of their locations.
6.2. Algorithm: Constraint-Controlled PPO
We implemented Algorithm 1on top of Proximal Policy Op-
timization (PPO) ( Schulman et al. ,2017 ) to make constraint-
controlled PPO (CPPO). CPPO uses an analogous clipped
surrogate objective for the cost as for the reward. Our
Responsive Safety in RL by PID Lagrangian Methods
policy is a 2-layer MLP followed by an LSTM with a
skip connection. We applied smoothing to proportional
and derivative controls to accommodate noisy estimates.
The environments’ ﬁnite horizons allowed use of non-
discounted episodic costs as the constraint and input to the
controller. Additional training details can be found in sup-
plementary materials, and our implementation is available
athttps://github.com/astooke/safe-rlpyt .
6.3. Main Results
We compare PID controller performance against the La-
grangian baseline under a wide range of settings. Plots
showing the performance of the unconstrained analogue
conﬁrm that constraints are not trivially satisﬁed, and they
appear in supplementary material.
6.3.1. R OBUST SAFETY WITH PI C ONTROL
We observed cost oscillations or overshoot with slow settling
time in a majority of Safety Gym environments when using
the Lagrangian method. Figure 3shows an example where
PI-control eliminated this behavior while maintaining good
reward performance, in the challenging D OGGO BUTTON 1
environment. Individual runs are plotted for different cost
limits.
0
 2
 4
 6
 8
1e7
50
100
150
200Cost
0
 2
 4
 6
 8
Environment Steps 1e7
0
5
10
15Return
KP= 0
KP= 1 (ours)
Figure 3. Oscillations in episodic costs (and returns) from the La-
grangian method, KP= 0,KI= 10−2, are damped by propor-
tional control, KP= 1 (ours), at cost limits 50,100,150,200
(curves shaded) in D OGGO BUTTON 1.
As predicted in ( Platt & Barr ,1988 ), we found the severity
of cost overshoot and oscillations to depend on the penalty
coefﬁcient learning rate, KI. The top left panel of Figure
4shows example cost curves from D OGGO GOAL2 under
I-control, over a wide range of values for KI(we refer to
varyingKI, assuming KI= 1; the two are interchangeable
in our design). With increasing KI, the period and ampli-tude of cost oscillations decrease and eventually disappear.
The bottom left of Figure 4, however, shows that larger KI
also brings diminishing returns. We study this effect in the
next section. The center and right columns of Figure 4show
the cost and return when using PI-control, with KP= 0.25
andKP= 1, respectively. Proportional control stabilized
the cost, with most oscillations reduced to the noise ﬂoor
forKI>10−4. Yet returns remained relatively high over
a wide range, KI<10−1. Similar curves for other Safety
Gym environments are included in an appendix.
0.0
 0.5
 1.0
1e8
30
50
70CostKP= 0
0.0
 0.5
 1.0
1e8
30
50
70KP= 0.25
0.0
 0.5
 1.0
1e8
30
50
70KP= 1
0.0
 0.5
 1.0
Env Steps 1e8
0
5
10
15
20Return
0.0
 0.5
 1.0
1e8
0
5
10
15
20
0.0
 0.5
 1.0
1e8
0
5
10
15
20
KI= 1e-4
KI= 1e-3
KI= 1e-2
KI= 1e-1
KI= 1
Figure 4. Top row : Constraint-violating oscillations decrease in
magnitude and period from increases in the Lagrange multiplier
learning rate, KI. At all levels, oscillations are damped by PI-
control,KP= 0.25,1.Bottom row : Returns diminish for large
KI; proportional control maintains high returns while reducing
constraint violations. Environment: D OGGO GOAL2, cost limit 50.
We examine the trade-off between reward and constraint vio-
lation by forming an overall cost ﬁgure of merit (FOM). We
use the sum of non-discounted constraint violations over the
learning iterates, CFOM=/summationtext
k(D(πθk)−d)+,D(πθ) =
Eτ∼π/bracketleftBig/summationtextT
t=0C(st,at,s′
t)/bracketrightBig
, and estimate it online from the
learning data. Figure 5compares ﬁnal returns against this
cost FOM for the same set of experiments as in Figure 4.
Each point represents a different setting of KI, averaged
over four runs. PI-control expanded the Pareto frontier of
this trade-off into a new region of high rewards at rela-
tively low cost which was inaccessible using the Lagrangian
method. These results constitute a new state of the art over
the benchmarks in Ray et al. (2019 ).
We performed similar experiments on several Safety Gym
environments in addition to D OGGO GOAL2: P OINT GOAL1,
the simplest domain with a point-like robot, C ARBUTTON 1,
for slightly more challenging locomotive control, and D OG-
GOBUTTON 1 for another challenging task (see appendix
for learning curves like Figure 4). Figure 6plots the cost
ﬁgure of merit over the same range of values for KI, and
Responsive Safety in RL by PID Lagrangian Methods
0
 1000
 2000
 3000
Cost FOM
8
10
12
14Return
KP= 0
KP= 0.25
KP= 1
Figure 5. Pareto frontier of return versus cost FOM, which im-
proves (up and to the left) with PI-control, KP= 0.25,1. Each
point is a different setting of KI(see Figure 4).
for two strengths of added proportional control, for these
environments. PI-control clearly improved the cost FOM
(lower is better) for KI<10−1, above which the fast in-
tegral control dominated. Hence robustness to the value
forKIwas signiﬁcantly improved in all the learning tasks
studied.
10−4
10−2
100
101
102
103
104
DoggoButton1
10−4
10−2
100
102
103
104
DoggoGoal2
10−4
10−2
100
KI
101
102
103
104
Cost FOM
CarButton1
KP= 0
KP= 0.25
KP= 1
10−4
10−2
100
100
101
102
103
PointGoal1
Figure 6. Learning run cost FOM versus penalty learning rate, KI,
from four environments spanning the robots in Safety Gym. Each
point is an average over four runs. In all cases, PI-control improves
performance (lower is better) over a wide and useful range of KI,
easing selection of that hyperparameter.
6.3.2. C ONTROL EFFICIENCY
We further investigated why increasing the penalty learning
rate,KI, eventually reduces reward performance, as was
seen in the robustness study. Figure 7shows learning curves
for three settings: I- and PI-control with the same, moderate
KI= 10−3, and I-control with high KI= 10−1. The high-
KIsetting achieved responsive cost performance but lower
long-term returns, which appears to result from wildly ﬂuc-
tuating control. In contrast, PI-control held relatively steady,
despite the noise, allowing the agent to do reward-learning
at every iteration. The bottom panel displays individual
control iterates, here displayed as u=λ/(1+λ), over the
ﬁrst 7M environment steps, while the others show smoothed
curves over the entire learning run, over 40M steps.
0
 1
 2
 3
 4
1e7
0
5
10Return
IKI+
PI
I
0
 1
 2
 3
 4
1e7
150
200Cost
0
 1
 2
 3
 4
1e7
0.0
0.5Control
0
 1
 2
 3
 4
 5
 6
Environment Steps 1e6
0.0
0.5
1.0Control
Figure 7. I- and PI-control with moderate KI= 10−3and I-
control with fast KI= 10−1(IKI+).TopReturns diminished for
fast-KI, but high for PI. Second Cost oscillations mostly damped
by PI, removed by fast- KI.Third Control (smoothed) varies more
rapidly under fast- KI, is relatively steady for PI. Bottom Con-
trol over ﬁrst 500 RL iterations; fast- KIslams the control to the
extremes, causing the diminished returns. Environment: D OG-
GOBUTTON 1, cost limit 200.
6.3.3. P REDICTIVE CONTROL BY DERIVATIVES
Figure 8demonstrates the predictive capabilities of deriva-
tive cost control in a noisy deep RL setting. It removed
cost overshoot from both the I- and PI-controlled baselines.
It was further able to slow the approach of the cost curve
towards the limit, a desirable behavior for online learning
systems requiring safety monitoring. Curves for other envi-
ronments are available in an appendix.
7. Reward-Scale Invariance
In the preceding sections, we showed that PID control im-
proves hyperparameter robustness in every constrained RL
environment we tested. Here we propose a complemen-
tary method to promote robustness both within and across
environments. Speciﬁcally, it addresses the sensitivity of
learning dynamics to the relative numerical scale of reward
and cost objectives.
Consider two CMDPs that are identical except that in one
the rewards are scaled by a constant factor, ρ. The opti-
mal policy parameters, θ∗remain unchanged, but clearly λ∗
must scale by ρ. To attain the same learning dynamics, all
Responsive Safety in RL by PID Lagrangian Methods
0.0
 0.6
 1.2
1e7
100
150
200CostKI= 1, KP= 0
KD= 0
KD= 4
KD= 5
0.0
 0.6
 1.2
1e7
150
200KI= 1, KP= 2
KD= 0
KD= 2
KD= 4
0.0
 0.6
 1.2
1e7
0
5Return
0.0
 0.6
 1.2
Environment Steps 1e7
0
5
Figure 8. Derivative control can prevent cost overshoot and slow
the rate of cost increase within feasible regions, which the La-
grangian method cannot do. Environment: D OGGO BUTTON 1,
cost limit 200.
controller settings, λ0,KI,KP, andKDmust therefore be
scaled by ρ. This situation might feature naturally within
a collection of related learning environments. Addition-
ally, within the course of learning an individual CMDP, the
balance between reward and cost magnitudes can change
considerably, placing burden on the controller to track the
necessary changes in the scale of λ.
One way to promote performance of a single choice of
controller settings across these cases would be to maintain
a ﬁxed meaning for the value of λin terms of the relative
inﬂuence of reward versus cost on the parameter update. To
this end, we introduce an adjustable scaling factor, βk, in
the policy gradient:
∇θL= (1−uk)∇θJ(πθk)−ukβk∇θJC(πθk)(25)
A conspicuous choice for βkis the ratio of un-scaled policy
gradients:
β∇,k=||∇θJ(πθk)||
||∇θJC(πθk)||(26)
since it balances the total gradient to have equal-magnitude
contribution from reward- and cost-objectives at λ= 1and
encourages λ∗= 1. Furthermore, β∇is easily computed
with existing algorithm components.
To test this method, we ran experiments on Safety Gym en-
vironments with their rewards scaled up or down by a factor
of 10. Figure 9shows a representative cross-section of re-
sults from the P OINT GOAL1 environment using PI-control.
The different curves within each plot correspond to different
reward scaling. Without objective-scaling (i.e. β= 1), the
dynamics under ρ= 10 are as if controller parameters were
instead divided by 10, and likewise for ρ= 0.1. Note the
near-logarithmic spacing of λ(λρ=10has not converged to
its full value). Using β∇, on the other hand, the learning
dynamics are nearly identical across two orders of magni-
tude of reward scale. λ0= 1becomes an obvious choice forinitialization, a point where previous theory provides little
guidance ( Chow et al. ,2019 ) (although here we left λ0= 0).
Experiments in other environments and controller settings
yielded similar results and are included in supplementary
materials. Other methods, such running normalization of
rewards and costs, could achieve similar effects and are
worth investigating, but our simple technique is surprisingly
effective and is not speciﬁc to RL.
0.0
 0.5
 1.0
 1.5
1e7
20
40
60Costβ= 1
0.0
 0.5
 1.0
 1.5
1e7
20
40
60β=β∇
0.0
 0.5
 1.0
 1.5
1e7
10
20Return
ρ= 0.1
ρ= 1
ρ= 10
0.0
 0.5
 1.0
 1.5
1e7
10
20
0.0
 0.5
 1.0
 1.5
Environment Steps 1e7
10−1
100
101λ
0.0
 0.5
 1.0
 1.5
1e7
10−1
100
101
Figure 9. Costs, returns, and Lagrange multiplier with rewards
scaled by ρ∈ {0.1,1,10}; PI-control with KI= 1e−3,KP=
0.1.Left column : without objective-weighting, learning dynam-
ics differ dramatically due to required scale of λ.Right column :
with objective-weighting, learning dynamics are nearly identical.
Environment: P OINT GOAL1, cost limit 25.
8. Conclusion
Starting from a novel development in classic Lagrangian
methods, we introduced a new set of constrained RL solu-
tions which are straightforward to understand and imple-
ment, and we have shown them to be effective when paired
with deep learning.
Several opportunities for further work lay ahead. Analysis
of the modiﬁed Lagrangian method and constrained RL as
a dynamical system may relax theoretical requirements for
a slowly-changing multiplier. The mature ﬁeld of control
theory (and practice) provides tools for tuning controller
parameters. Lastly, the control-afﬁne form may assist in
both analysis (see Liang-Liang Xie & Lei Guo (2000 ) and
Galbraith & Vinter (2003 ) for controllability properties for
uncertain nonlinear dynamics) and by opening to further
control techniques such as feedback linearization.
Our contributions improve perhaps the most commonly used
constrained RL algorithm, which is a workhorse baseline.
We have addressed its primary shortcoming while preserving
its simplicity and even making it easier to use—a compelling
combination to assist in a wide range of applications.
Responsive Safety in RL by PID Lagrangian Methods
Acknowledgements
Adam Stooke gratefully acknowledges previous support
from the Fannie and John Hertz Foundation and the NVIDIA
Corporation. We thank Carlos Florensa and the anonymous
reviewers for many helpful suggestions which improved the
manuscript.
References
Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained
policy optimization. CoRR , abs/1705.10528, 2017. URL
http://arxiv.org/abs/1705.10528 .
Altman, E. Constrained markov decision processes with
total cost criteria: Lagrangian approach and dual linear
program. Mathematical methods of operations research ,
48(3):387–417, 1998.
Altman, E. Constrained Markov decision processes , vol-
ume 7. CRC Press, 1999.
An, W., Wang, H., Sun, Q., Xu, J., Dai, Q., and Zhang, L.
A pid controller approach for stochastic optimization of
deep networks. 2018 IEEE/CVF Conference on Computer
Vision and Pattern Recognition , pp. 8522–8531, 2018.
˚Astr¨om, K. J. and H ¨agglund, T. Pid control. IEEE Control
Systems Magazine , 1066(033X/06), 2006.
Bertsekas, D. P. On penalty and multiplier methods for
constrained minimization. SIAM Journal on Control and
Optimization , 14(2):216–235, 1976.
Bertsekas, D. P. Constrained optimization and Lagrange
multiplier methods . Academic press, 2014.
Chow, Y ., Nachum, O., Du ´e˜nez-Guzm ´an, E. A., and
Ghavamzadeh, M. A lyapunov-based approach to safe
reinforcement learning. CoRR , abs/1805.07708, 2018.
URLhttp://arxiv.org/abs/1805.07708 .
Chow, Y ., Nachum, O., Faust, A., Ghavamzadeh, M.,
and Du ´e˜nez-Guzm ´an, E. A. Lyapunov-based safe
policy optimization for continuous control. CoRR ,
abs/1901.10031, 2019. URL http://arxiv.org/
abs/1901.10031 .
Dalal, G., Dvijotham, K., Vecer ´ık, M., Hester, T., Paduraru,
C., and Tassa, Y . Safe exploration in continuous action
spaces. CoRR , abs/1801.08757, 2018. URL http://
arxiv.org/abs/1801.08757 .
Galbraith, G. N. and Vinter, R. B. Lipschitz continuity of
optimal controls for state constrained problems. SIAM
Journal on Control and Optimization , 42(5):1727–1744,
2003. doi: 10.1137/S0363012902404711. URL https:
//doi.org/10.1137/S0363012902404711 .Geibel, P. and Wysotzki, F. Risk-sensitive reinforcement
learning applied to control under constraints. CoRR ,
abs/1109.2147, 2011. URL http://arxiv.org/
abs/1109.2147 .
Gu, S., Lillicrap, T., Sutskever, I., and Levine, S. Con-
tinuous deep q-learning with model-based acceleration.
InInternational Conference on Machine Learning , pp.
2829–2838, 2016.
Hestenes, M. R. Multiplier and gradient methods. Journal
of optimization theory and applications , 4(5):303–320,
1969.
Hu, B. and Lessard, L. Control interpretations for ﬁrst-order
optimization methods. CoRR , abs/1703.01670, 2017.
URLhttp://arxiv.org/abs/1703.01670 .
Isidori, A., Thoma, M., Sontag, E. D., Dickinson, B. W.,
Fettweis, A., Massey, J. L., and Modestino, J. W. Nonlin-
ear Control Systems . Springer-Verlag, Berlin, Heidelberg,
3rd edition, 1995. ISBN 3540199160.
Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L.,
Lever, G., Castaneda, A. G., Beattie, C., Rabinowitz,
N. C., Morcos, A. S., Ruderman, A., et al. Human-level
performance in 3d multiplayer games with population-
based reinforcement learning. Science , 364(6443):859–
865, 2019.
Lessard, L., Recht, B., and Packard, A. Analysis and de-
sign of optimization algorithms via integral quadratic
constraints, 2014.
Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-
end training of deep visuomotor policies. The Journal of
Machine Learning Research , 17(1):1334–1373, 2016.
Liang-Liang Xie and Lei Guo. How much uncertainty can be
dealt with by feedback? IEEE Transactions on Automatic
Control , 45(12):2203–2217, Dec 2000. ISSN 2334-3303.
doi: 10.1109/9.895559.
Liu, G.-H. and Theodorou, E. A. Deep learning theory
review: An optimal control and dynamical systems per-
spective, 2019.
Liu, Y ., Ding, J., and Liu, X. Ipo: Interior-point pol-
icy optimization under constraints. arXiv preprint
arXiv:1910.09615 , 2019.
Mnih, V ., Kavukcuoglu, K., Silver, D., Graves, A.,
Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing
atari with deep reinforcement learning. arXiv preprint
arXiv:1312.5602 , 2013.
Nishihara, R., Lessard, L., Recht, B., Packard, A., and
Jordan, M. I. A general analysis of the convergence of
admm, 2015.
Responsive Safety in RL by PID Lagrangian Methods
Nocedal, J. and Wright, S. Numerical optimization . Springer
Science & Business Media, 2006.
OpenAI. Openai ﬁve. https://blog.openai.com/
openai-five/ , 2018.
Paternain, S., Chamon, L., Calvo-Fullana, M., and Ribeiro,
A. Constrained reinforcement learning has zero dual-
ity gap. In Advances in Neural Information Processing
Systems , pp. 7553–7563, 2019.
Platt, J. C. and Barr, A. H. Constrained differential opti-
mization. In Neural Information Processing Systems , pp.
612–621, 1988.
Powell, M. J. A method for nonlinear constraints in mini-
mization problems. Optimization , pp. 283–298, 1969.
Ray, A., Achiam, J., and Amodei, D. Benchmarking Safe
Exploration in Deep Reinforcement Learning. 2019.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz,
P. Trust region policy optimization. In International
conference on machine learning , pp. 1889–1897, 2015.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347 , 2017.
Skelton, R. Dynamic Systems Control: Linear
Systems Analysis and Synthesis . Dynamic Sys-
tems Control. John Wiley & Sons, 1988. ISBN
9780471837794. URL https://books.google.
com/books?id=egFRAAAAMAAJ .
Song, Q. and Leland, R. P. An optimal control model of
neural networks for constrained optimization problems.
Optimal Control Applications and Methods , 19(5):371–
376, 1998. doi: 10.1002/(SICI)1099-1514(199809/10)19:
5/an}b∇acketle{t371::AID-OCA636 /an}b∇acket∇i}ht3.0.CO;2-8. URL https:
//onlinelibrary.wiley.com/doi/abs/10.
1002/%28SICI%291099-1514%28199809/10%
2919%3A5%3C371%3A%3AAID-OCA636%3E3.0.
CO%3B2-8 .
Sutton, R. S. and Barto, A. G. Introduction to Reinforcement
Learning . MIT Press, Cambridge, MA, USA, 1st edition,
1998. ISBN 0262193981.
Tessler, C., Mankowitz, D. J., and Mannor, S. Reward con-
strained policy optimization. CoRR , abs/1805.11074,
2018. URL http://arxiv.org/abs/1805.
11074 .
Todorov, E., Erez, T., and Tassa, Y . Mujoco: A physics
engine for model-based control. In 2012 IEEE/RSJ Inter-
national Conference on Intelligent Robots and Systems ,
pp. 5026–5033. IEEE, 2012.Wah, B. W., Wang, T., Shang, Y ., and Wu, Z. Improving the
performance of weighted lagrange-multiplier methods for
nonlinear constrained optimization. Information Sciences ,
124(1-4):241–272, 2000.
Yang, T.-Y ., Rosca, J., Narasimhan, K., and Ramadge, P. J.
Projection-based constrained policy optimization. In
International Conference on Learning Representations ,
2020. URL https://openreview.net/forum?
id=rke3TJrtPS .


Few-Shot Intent Inference via Meta-Inverse Reinforcement Learning

Learning a Prior over Intent
via Meta-Inverse Reinforcement Learning
Kelvin Xu1Ellis Ratner1Anca Dragan1Sergey Levine1Chelsea Finn1
Abstract
A signiﬁcant challenge for the practical appli-
cation of reinforcement learning to real world
problems is the need to specify an oracle reward
function that correctly deﬁnes a task. Inverse re-
inforcement learning (IRL) seeks to avoid this
challenge by instead inferring a reward function
from expert demonstrations. While appealing, it
can be impractically expensive to collect datasets
of demonstrations that cover the variation com-
mon in the real world (e.g. opening any type
of door). Thus in practice, IRL must commonly
be performed with only a limited set of demon-
strations where it can be exceedingly difﬁcult to
unambiguously recover a reward function. In this
work, we exploit the insight that demonstrations
from other tasks can be used to constrain the set
of possible reward functions by learning a “prior”
that is speciﬁcally optimized for the ability to
infer expressive reward functions from limited
numbers of demonstrations. We demonstrate that
our method can efﬁciently recover rewards from
images for novel tasks and provide intuition as to
how our approach is analogous to learning a prior.
1. Introduction
Reinforcement learning (RL) algorithms have the potential
to automate a wide range of decision-making and control
tasks across a variety of different domains, as demonstrated
by successful recent applications ranging from robotic con-
trol ( Kober & Peters ,2012 ;Levine et al. ,2016 ) to game
playing ( Mnih et al. ,2015 ;Silver et al. ,2016 ). A key as-
sumption of the RL problem statement is the availability of
a reward function that accurately describes the desired task.
For many real world tasks, reward functions can be chal-
1Department of Electrical Engineering and Computer Sci-
ence, University of California, Berkeley, USA. Correspondence to:
Kelvin Xu .
Proceedings of the 36thInternational Conference on Machine
Learning , Long Beach, California, PMLR 97, 2019. Copyright
2019 by the author(s).
Figure 1. A diagram of our meta-inverse RL approach. Our ap-
proach attempts to remedy over-ﬁtting in few-shot IRL by learning
a “prior” that constraints the set of possible reward functions to lie
within a few steps of gradient descent. Standard IRL attempts to
recover the reward function directly from the available demonstra-
tions. The shortcoming of this approach is that there is little reason
to expect generalization as it is analogous to training a density
model with only a few examples.
lenging to manually specify, while being crucial for good
performance ( Amodei et al. ,2016 ). Most real world tasks
are multifaceted and require reasoning over multiple factors
in a task (e.g. a robot cleaning in a house with children),
while simultaneously providing appropriate reward shaping
to make the task feasible with tractable exploration ( Ng
et al. ,1999 ). These challenges are compounded by the
inherent difﬁculty of specifying rewards for tasks with high-
dimensional observation spaces such as images.
Inverse reinforcement learning (IRL) is an approach that
aims to address this problem by instead inferring the reward
function from demonstrations of the task ( Ng & Russell ,
2000 ). This has the appealing beneﬁt of taking a data-
driven approach to reward speciﬁcation in place of hand
engineering. In practice however, rewards functions are
rarely learned as it can be prohibitively expensive to provide
demonstrations that cover the variability common in real
world tasks (e.g., collecting demonstrations of opening every
type of door knob). In addition, while learning a complex
function from high dimensional observations might make
an expressive function approximator seem like a reasonable
modelling assumption, in the “few-shot” domain it is noto-
riously difﬁcult to unambiguously recover a good reward
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
function with expressive function approximators. Many
prior approaches have thus relied on low-dimensional linear
models with handcrafted features that effectively encode a
strong prior on the relevant features of a task. This requires
engineering a set of features by hand that work well for a
speciﬁc problem. In this work, we propose an approach that
instead explicitly learns expressive features that are robust
even when learning with limited demonstrations.
Our approach relies on the key observation that related tasks
share a common structure that we can leverage when learn-
ing new tasks. To illustrate, considering a robot navigating
through a home. While the exact reward function we pro-
vide to the robot may differ depending on the task, there
is a structure amid the space of useful behaviours, such as
navigating to a series of landmarks, and there are certain
behaviors we always want to encourage or discourage, such
as avoiding obstacles or staying a reasonable distance from
humans. This notion agrees with our understanding of why
humans can easily infer the intents and goals (i.e., reward
functions) of even abstract agents from just one or a few
demonstrations ( Baker et al. ,2007 ), as humans have access
to strong priors about how other humans accomplish similar
tasks accrued over many years. Similarly, our objective is
to discover the common structure among different tasks,
and encode that structure in a way that can be used to infer
reward functions from a few demonstrations.
More speciﬁcally, in this work we assume access to a set of
tasks, along with demonstrations of the desired behaviors
for those tasks, which we refer to as the meta-training set .
From these tasks, we then learn a reward function parame-
terization that enables effective few-shot learning when used
to initialize IRL in a novel task. Our method is summarized
in Fig. 1. Our key contribution is an algorithm that enables
efﬁcient learning of new reward functions by using meta-
training to build a rich “prior” for goal inference. Using our
proposed approach, we show that we can learn deep neural
network reward functions from raw pixel observations on
two distinct domains with substantially better data efﬁciency
than existing methods and standard baselines.
2. Related Work
Inverse reinforcement learning (IRL) ( Ng & Russell ,2000 )
is the problem of inferring an expert’s reward function di-
rectly from demonstrations. Prior methods for performing
IRL range from margin based approaches ( Abbeel & Ng ,
2004 ;Ratliff et al. ,2006 ) to probabilistic approaches ( Ra-
machandran & Amir ,2007 ;Ziebart et al. ,2008 ). Although it
is possible to extend our approach to any other IRL method,
in this work we base on work on the maximum entropy
(MaxEnt) framework ( Ziebart et al. ,2008 ). In addition to
allowing for sub-optimality in the expert demonstrations,
MaxEnt-IRL can be re-framed as a maximum likelihood
estimation problem. (Sec. 3).In part to combat the under-speciﬁed nature of IRL, prior
work has often used low-dimensional linear parameteriza-
tions with handcrafted features ( Abbeel & Ng ,2004 ;Ziebart
et al.,2008 ). In order to learn from high dimensional input,
Wulfmeier et al. (2015 ) proposed applying fully convolu-
tional networks ( Shelhamer et al. ,2017 ) to the MaxEnt
IRL framework ( Ziebart et al. ,2008 ) for several naviga-
tion tasks ( Wulfmeier et al. ,2016b ;a). Other methods that
have incorporated neural network rewards include guided
cost learning (GCL) ( Finn et al. ,2017a ), which uses impor-
tance sampling and regularization for scalability to high-
dimensional spaces, and adversarial IRL ( Fu et al. ,2018 ).
Several other methods have also proposed imitation learning
approaches based on adversarial frameworks that resemble
IRL, but do not aim to directly recover a reward function ( Ho
& Ermon ,2016 ;Li et al. ,2017 ;Hausman et al. ,2017 ;Kue-
ﬂer & Kochenderfer ,2018 ). In this work, instead of improv-
ing the ability to learn reward functions on a single task,
we focus on the problem of effectively learning to use prior
demonstration data from other IRL tasks, allowing us to
learn new tasks from a limited number demonstrations even
with expressive non-linear reward functions.
Prior work has explored the problem of multi-task IRL,
where the demonstrated behavior is assumed to have origi-
nated from multiple experts achieving different goals. Some
of these approaches include those that aim to incorporate a
shared prior over reward functions through extending the
Bayesian IRL ( Ramachandran & Amir ,2007 ) framework to
the multi-task setting ( Dimitrakakis & Rothkopf ,2012 ;Choi
& Kim ,2012 ). Other approaches have clustered demon-
strations while simultaneously inferring reward functions
for each cluster ( Babe s¸-Vroman et al. ,2011 ) or introduced
regularization between rewards to a common “shared re-
ward” ( Li & Burdick ,2017 ). Our work is similar in that we
also seek to encode prior information common to the tasks.
However, a critical difference is that our method speciﬁcally
aims to distill the meta-training tasks into a prior that can
then be used to learn rewards for new tasks efﬁciently. The
goal therefore is not to acquire good reward functions that
explain the meta-training tasks, but rather to use them to
learn efﬁciently on new tasks.
Our approach builds on work on the broader problem of
meta-learning ( Schmidhuber ,1987 ;Bengio et al. ;Naik &
Mammone ,1992 ;Thrun & Pratt ,2012 ) and generative mod-
elling ( Rezende et al. ,2016 ;Reed et al. ,2018 ;Mordatch ,
2018 ). Prior work has proposed a variety of solutions for
learning to learn including memory based methods ( Duan
et al.,2016 ;Santoro et al. ,2016 ;Wang et al. ,2016 ;Mishra
et al. ,2017 ), methods that learn an optimizer and/or ini-
tialization ( Andrychowicz et al. ,2016 ;Ravi & Larochelle ,
2016 ;Finn et al. ,2017a ;Li & Malik ,2017 ), and meth-
ods that compare new datapoints in a learned metric space
(Koch ,2015 ;Wang & Hebert ,2016 ;Vinyals et al. ,2016 ;
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
Shyam et al. ,2017 ;Snell et al. ,2017 ). Our work is moti-
vated by the goal of broadening the applicability of IRL,
but in principle it is possible to adapt many of these meta-
learning approaches for our problem statement. We build
upon Finn et al. (2017a ), which has also been previously
applied to the related problems of imitation learning and
human motion prediction ( Wang et al. ,2016 ;Finn et al. ,
2017b ;Alet et al. ,2018 ). We leave it to future work to do
a comprehensive investigation of different meta-learning
approaches which could broaden the applicability of IRL.
3. Preliminaries and Overview
In this section, we introduce our notation and describe the
IRL and meta-learning problems.
3.1. Learning Rewards via Maximum Entropy Inverse
Reinforcement Learning
The standard Markov decision process (MDP) is deﬁned by
the tuple (S,A,ps,r ,)where SandAdenote the set of
possible states and actions respectively, r:S⇥A!R
is the reward function, 2[0,1]is the discount factor
andps:S⇥S⇥A ![0,1]denotes the transition
distribution over the next state st+1, given the current
state stand current action at. Typically, the goal of “for-
ward” RL is to maximize the expected discounted return
R(⌧)=PT
t=1t1r(st,at).
In IRL, we instead assume that the reward function is
unknown but that we instead have access to a set of ex-
pert demonstrations D={⌧1,...,⌧ K}, where ⌧k=
{s1,a1,..., sT,aT}.
The goal of IRL is to recover the unknown reward function
rfrom the set of demonstrations. We build on the maximum
entropy (MaxEnt) IRL framework by Ziebart et al. (2008 ),
which models trajectories as being distributed proportional
to their exponentiated return
p(⌧)=1
Zexp ( R(⌧)), (1)
where Zis the partition function, Z=R
⌧exp( R(⌧))d⌧.
This distribution can be shown to be induced by the optimal
policy in entropy regularized forward RL problem:
⇡⇤= arg max
⇡E⌧⇠⇡[R(⌧)log⇡(⌧)]. (2)
This formulation allows us to pose the reward learning prob-
lem as a maximum likelihood estimation (MLE) problem in
an energy-based model rby deﬁning the following loss:
min
E⌧⇠D[LIRL(⌧)]:=min
E⌧⇠D[logp(⌧)].(3)
Learning in general energy-based models of this form is
common in many applications such as structured predic-
tion. However, in contrast to applications where learningcan be supervised by millions of labels (e.g. semantic seg-
mentation), the learning problem in Eq. 3must typically
be performed with a relatively small number of example
demonstrations. In this work, we seek to address this issue
in IRL by providing a way to integrate information from
prior tasks to constrain the optimization in Eq. 3in the
regime of limited demonstrations.
3.2. Meta-Learning
The goal of meta-learning algorithms is to optimize for
the ability to learn efﬁciently on new tasks. Rather than
attempting to generalize to new datapoints, meta-learning
can be understood as attempting to generalize to new tasks .
It is assumed in the meta-learning setting that there are two
disjoint sets of tasks that we refer to as the meta-training
set{Ti;i=1 ..N}and meta-test set {Tj;j=1 ..M},
which are both drawn from a distribution p(T). During
meta-training time, the meta-learner attempts to learn the
structure of the tasks in the meta-training set, such that when
it is presented with a test task, it can leverage this structure
to learn efﬁciently from a limited number of examples.
To illustrate this distinction, consider the case of few-
shot learning setting. Let f✓denote the learner, and let
a task be deﬁned by learning from Ktraining examples
Xtr
T={x1...,xK},Ytr
T={y1...,yK}, and evaluat-
ing on K0test examples Xtest
T={x1...,xK0},Ytest
T=
{y1...,yK0}. One approach to meta-learning is to directly
parameterize the meta-learner with an expressive model
such as a recurrent or recursive neural network ( Duan et al. ,
2016 ;Mishra et al. ,2017 ) conditioned on the task training
data and the inputs for the test task: f✓(Y|Xtest
T,Xtr
T,Ytr
T).
Such a model is optimized using log-likelihood across all
tasks. In this approach to meta-learning, since neural net-
works are known to be universal function approximators
(Siegelmann & Sontag ,1995 ), any desired structure be-
tween tasks can be implicitly encoded.
Rather than learn a single black-box function, another ap-
proach to meta-learning is to learn components of the learn-
ing procedure such as the initialization ( Finn et al. ,2017a )
or the optimization algorithm ( Ravi & Larochelle ,2016 ;
Andrychowicz et al. ,2016 ). In this work we extend the
approach of model agnostic meta-learning (MAML) intro-
duced by Finn et al. (2017a ), which learns an initialization
that is adapted by gradient descent. Concretely, in the super-
vised learning case, given a loss function L(✓,XT,YT)(e.g.
cross-entropy), MAML performs the following optimization
min
✓X
TL(T,Xtest
T,Ytest
T)
=m i n
✓X
TL
✓↵r✓L(✓,Xtr
T,Ytr
T),Xtest
T,Ytest
T
,(4)
where the optimization is over an initial set of parameters ✓
and the loss on the held out tasks Xtest
Tbecomes the signal
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
for learning the initial parameters for gradient descent (with
step size ↵) on Xtr
T. This optimization is analogous to
adding a constraint in a multi-task setting, which we show
in later sections is analogous in our setting to learning a
prior over reward functions.
4. Learning to Learn Rewards
Our goal in meta-IRL is to learn how to learn reward func-
tions across many tasks such that the model can infer the
reward function for a new task using only one or a few expert
demonstrations. Intuitively, we can view this problem as
aiming to learn a prior over the rewards of expert demonstra-
tors, such that when given just one or a few demonstrations
of a new task, we can combine the learned prior with the
new data to effectively determine the expert’s intent. Such
a prior is helpful in inverse reinforcement learning settings,
since the space of reward functions with are relevant to par-
ticular task is much smaller than the space of all possible
rewards deﬁnable on the raw observations.
During meta-training, we have a set of tasks {Ti;i=
1..N}. Each task Tihas a set of demonstrations
DT={⌧1,...,⌧ K}from an expert policy which we par-
tition into disjoint Dtr
TandDtest
Tsets. The demonstrations
for each meta-training task are assumed to be produced by
the expert according to the maximum entropy model in Sec-
tion3.1. During meta-training, these tasks will be used to
encodes common structure so that our model can quickly
acquire rewards for new tasks from just a few demonstra-
tions.
After meta-training, our method is presented with a new
task. During this meta-test phase, the algorithm must infer
the parameters of the reward function r(st,at)for the
new task from a few demonstrations. As is standard in
meta-learning, we assume that the test task is from the same
distribution of tasks seen during meta-training, a distribution
that we denote as p(T).
4.1. Meta Reward and Intention Learning (MandRIL)
In order to meta-learn a reward function that can act as a
prior for new tasks and new environments, we ﬁrst formalize
the notion of a good reward by deﬁning a loss LT(✓)on
the reward function r✓for a particular task T. We use the
MaxEnt IRL loss LIRLdiscussed in Section 3, which, for
a given DT, leads to the following gradient ( Ziebart et al. ,
2008 ):
r✓LT(✓)=@r✓
@✓⇥
E⌧[µ⌧]µDT⇤
. (5)
where µ⌧are the state-action visitations under the optimal
maximum entropy policy under r✓, and µDare the mean
state visitations under the demonstrated trajectories.
If our end goal were to achieve a single reward function thatAlgorithm 1 Meta Reward and Intention Learning (Man-
dRIL)
1:Input: Set of meta-training tasks {T }meta-train
2:Input: hyperparameters ↵,
3:function MAXENTIRL-G RAD(r✓,T,D).Single
task update
4: #Compute state visitations of demos
5: µD=STATE -VISITATIONS -TRAJ(T,D)
6: #Compute Max-Ent state visitations
7: E⌧[µ⌧]=STATE -VISITATIONS -POLICY (r✓,T)
8: #MaxEntIRL gradient (Ziebart et al. ,2008 )
9:@L
@r✓=E⌧[µ⌧]µD
10: Return@L
@r✓
11:
12:Randomly initialize ✓
13:while not done do
14: Sample batch of tasks Ti⇠{T }meta-train
15: for all Tido
16: Sample demos Dtr={⌧1,...,⌧ K}⇠T i
17: #Inner loss computation
18:@Ltr
Ti(✓)
@r✓=MAXENTIRL-G RAD(r✓,Ti,Dtr)
19: Compute r✓Ltr
Ti(✓)from@Ltr
Ti(✓)
@r✓
20: Compute Ti=✓↵r✓Ltr
Ti(✓)
21: Sample demos Dtest={⌧0
1,...,⌧0
K0}⇠T i
22: #Outer loss computation
23:@Ltest
Ti
@r✓=MAXENTIRL-G RAD(rTi,Ti,Dtest))
24: #Compute meta-gradient
25: Compute r✓Ltest
Tifrom@Ltest
Ti
@r✓via chain rule
26: Compute update to ✓ ✓P
ir✓Ltest
Ti
works as well as possible across all tasks in {Ti;i=1..N},
then we could simply follow the mean gradient across all
tasks. However, our objective is different: instead of opti-
mizing performance on the meta-training tasks, we aim to
learn a reward function that can be quickly and efﬁciently
adapted to new tasks at meta-test time. In doing so, we aim
to encode prior information over the task distribution in this
learned reward prior.
We propose to implement such a learning algorithm by ﬁnd-
ing the parameters ✓, such that starting from ✓and taking
a small number of gradient steps on a few demonstrations
from given task leads to a reward function for which a set
oftestdemonstrations have high likelihood, with respect to
the MaxEnt IRL model. In particular, we would like to ﬁnd
a✓such that the parameters
T=✓↵r✓Ltr
T(✓) (6)
lead to a reward function rTfor task T, such that the IRL
loss (corresponding to negative log-likelihood) for a disjoint
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
set of test demonstrations, given by LT,test
IRL, is minimized.
The corresponding optimization problem for ✓can therefore
be written as follows:
min
✓NX
i=1Ltest
Ti(Ti)=NX
i=1Ltest
Ti
✓↵r✓Ltr
Ti(✓)
.(7)
Our method acquires this prior ✓over rewards in the task
distribution p(T)by optimizing this loss. This amounts to
an extension of the MAML algorithm in Section 3.2to the
inverse reinforcement learning setting. This extension is
quite challenging, because computing the MaxEnt IRL gra-
dient requires repeatedly solving for the current maximum
entropy policy and visitation frequencies, and the MAML
objective requires computing derivatives through this gradi-
ent step. Next, we describe in detail how this is done. An
overview of our method is also outlined in Alg. 1.
Meta-training. The computation of the meta-gradient for
the objective in Eq. 7can be conceptually separated into two
parts. First, we perform the update in Eq. 6by computing
theexpected state visitations µ, which is the expected num-
ber of times an agent will visit each state. We denote this
overall procedure as STATE -VISITATIONS -POLICY , and fol-
lowZiebart et al. (2008 ) by ﬁrst computing the maximum
entropy optimal policy in Eq. 2under the current r✓, and
then approximating µusing dynamic programming. Next,
we compute the state visitation distribution of the expert
using a procedure which we denote as STATE -VISITATIONS -
TRAJ. This can be done either empirically, by averaging the
state visitation of the experts demonstrations, or by using
STATE -VISITATIONS -POLICY if the true reward is available
at meta-training time. This allows us to recover the IRL
gradient according to Eq. 5, which we can then apply to
compute Taccording to Eq. 6.
Second, we need to differentiate through this update to
compute the gradient of the meta-loss in Eq. 7. Note that
the meta-loss itself is the IRL loss evaluated with a different
set of test demonstrations. We follow the same procedure as
above to evaluate the gradient of LT,test
IRL with respect to the
post-update parameters T, and then apply the chain rule
to compute the meta-gradient:
r✓Ltest
T(✓)=@
@✓(✓↵r✓Ltr
T(✓))@rT
@T@Ltest
T
@rT
=
I↵@2Ltr
T(✓)
@✓2↵@r✓
@✓D@r✓
@✓>!
@rT
@T@Ltest
T
@rT
(8)
where on the second line we differentiate through the
MaxEnt-IRL update, and we deﬁne the |S||A| -dimensional
diagonal matrix Das
D:=diag ⇢@
@r✓,i(E⌧[µ⌧])i|S||A|
i=1!
.a1 a2
s1 s2 ......
sTaT
r
dynr
dynr
dynT
Figure 2. Our approach can be understood as approximately learn-
ing a distribution over the demonstrations ⌧, in the factor graph
p(⌧)=1
ZQT
t=1r(T,st,at)dyn(st+1,st,at)(above) where
we learn a prior over T, which during meta-test is used for MAP
inference over new expert demonstrations.
A detailed derivation of this expression is provided in the
supplementary Appendix E.
Meta-testing. Once we have acquired the meta-trained
parameters ✓that encode a prior over p(T), we can lever-
age this prior to enable fast, few-shot IRL of novel tasks
in{Tj;j=1..M}. For each task, we ﬁrst compute the
state visitations from the available set of demonstrations for
that task. Next, we use these state visitations to compute
the gradient, which is the same as the inner loss gradient
computation of the meta-training loop in Alg. 1. We apply
this gradient to adapt the parameters ✓to the new task. Even
if the model was trained with only one inner gradient steps,
we found in practice that it was beneﬁcial to take substan-
tially more gradient steps during meta-testing; performance
continued to improve with up to 20 steps.
4.2. Connection to Learning a Prior over Intent
The objective in Eq. 6optimizes for parameters that enable
the reward function to generalize efﬁciently on a wide range
of tasks. Intuitively, constraining the space of reward func-
tions to lie within a few steps of gradient descent can be
interpreted as expressing a “locality” prior over reward func-
tion parameters. This intuition can be made more concrete
by the following analysis.
By viewing IRL as maximum likelihood estimation in a
particular graphical model (Fig. 2), we can take the perspec-
tive of Grant et al. (2018) who showed that for a linear
model, fast adaptation via a few steps of gradient descent in
MAML is performing MAP inference over , under a Gaus-
sian prior with the mean ✓and a covariance that depends
on the step size, number of steps and hessian of the loss.
This is based on the connection between early stopping and
regularization previously discussed in Santos (1996), which
we refer the readers to for a more detailed discussion. The
interpretation of MAML as imposing a Gaussian prior on
the parameters is exact in the case of a likelihood that is
quadratic in the parameters (such as the log-likelihood of
a Gaussian in terms of its mean). For any non-quadratic
likelihood, this is an approximation in a local neighborhood
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
around ✓(i.e. up to convex quadratic approximation). In the
case of complex parameterizations, such as deep function
approximators, this is a coarse approximation and unlikely
to be the mode of a posterior. However, we can still frame
the effect of early stopping and initialization as serving as
a prior in a similar way as prior work ( Sj¨oberg & Ljung ,
1995 ;Duvenaud et al. ,2016 ;Grant et al. ,2018 ). More
importantly, this interpretation hints at future extensions to
our approach that could beneﬁt from employing more fully
Bayesian approaches to reward and goal inference.
5. Experiments
Our evaluation seeks to answer two questions. First, we aim
to test our core hypothesis that using prior task experience
enables reward learning for new tasks with just a few demon-
strations. Second, we compare our method with alternative
approaches that make use of multi-task experience.
We test our core hypothesis by comparing learning perfor-
mance on a new tasks starting from the initialization pro-
duced by MandRIL with learning a separate model for every
task starting either from a random initialization or from an
initialization obtained by supervised pre-training. We re-
fer to these approaches as learning “FROM SCRATCH ”and
“AV E R AG E G R A D I E N T ”pretraining respectively. Our su-
pervised pre-training baseline follows the average gradient
during meta-training tasks and ﬁnetunes at meta-test time
(as discussed in Section 4). Unlike our method, supervised
pre-training does not optimize for a model that performs
well under ﬁne tuning, but does use the same prior data
to pre-train. We additionally compare to pre-training on a
single task as well as all the meta-training tasks.
To our knowledge, there is no prior work that addresses
the speciﬁc meta-inverse reinforcement learning problem
introduced in this paper. Thus, to provide a point of com-
parison and calibrate the difﬁculty of the tasks, we adapt
two alternative black-box meta-learning methods to the IRL
setting. The comparisons to both of the black-box methods
described below evaluate the importance of incorporating
the IRL gradient into the meta-learning process, rather than
learning the adaptation process entirely from scratch.
Demo conditional model: Our method implicitly condi-
tions on the demonstrations through the gradient update.
In principle, a conditional deep model with sufﬁcient ca-
pacity could implicitly implement a similar learning rule.
Thus, we consider a conditional model (often referred to as
a “contextual model” ( Finn et al. ,2017b )), which receives
the demonstration as an additional input.
Recurrent meta-learner: We additionally compare to an
RNN-based meta-learner ( Santoro et al. ,2016 ;Duan et al. ,
2017 ). Speciﬁcally, we implement a conditional model by
feeding both images and sequences of states visited by the
demonstrations to an LSTM.
Figure 3. An example task on the SpriteWorld domain. When
learning a task, the agent has access to the image (left) and demon-
strations (red arrows). To evaluate learning (right), the agent is
tested for its ability to recover the reward for the task when the
objects have been rearranged. The reward structure we wish to cap-
ture can be illustrated by considering the initial state in blue. An
policy acting optimally under a correctly inferred reward should
interpret the other objects as obstacles, and prefer a path on dirt.
We consider two environments: (1)an image-based navi-
gation task with an aerial viewpoint, (2)a ﬁrst-person nav-
igation task in a simulated home environment with object
interaction. We describe here the environments and evalua-
tion protocol and provide detailed experimental settings and
hyperparameters for both domains in Appendices AandB.
(1) SpriteWorld navigation domain. Since most prior
IRL work (and multi-task IRL work) studied settings where
linear reward function approximators sufﬁce (i.e., low-
dimensional state spaces and hand-designed features), we
design an experiment that is signiﬁcantly more challenging—
that requires learning rewards on raw pixels. We consider
a navigation problem where we must learn a convolutional
neural network that directly maps image pixels to rewards.
We introduce a family of tasks called “SpriteWorld.” Some
example tasks are shown in Fig. 3. Tasks involve navigating
to goal objects while exhibiting preference over terrain types
(e.g., the agent prefers to traverse dirt tiles over traversing
grass tiles). At meta-test time, we provide one or a few
demonstrations in a single training environment and eval-
uate the reward learned using these demonstrations in a
new, test environment that contains the same objects as the
training environment, but arranged differently. Evaluating
in a new test environment is critical to measure that, after
adapting to the training environment from a few demonstra-
tions, the reward learned the correct visual cues, rather than
simply memorizing the demonstration trajectory.
We generate unique tasks in this domain as follows. First,
we randomly choose a set of three sprites from one hundred
sprites from the original game (creating a total of 161,700
unique tasks). We randomly place these three sprites within
a randomly generated terrain tiling; we designate one of
the sprites to be the goal landmark of the navigation task.
The other two objects are treated as obstacles for which the
agent incurs a large negative reward for not avoiding. In
each task, we optimize our model on a training world and
generalization in a test world, as described below.
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
S
G
Figure 4. An example task in the SUNCG environment (top). The
agent must complete either a “NA V” task (blue line) where the
goal is to navigate from the start to the cup or a “PICK” task (blue
+ green line) where the agent must also bring the cup to the bed.
The agent’s observation is a panoramic ﬁrst-person viewpoint (see
bottom left for RGB). Following the convention in prior work ( Fu
et al. ,2019 ), we provide to the reward function the correspond-
ing semantic images (bottom right). These images are 32⇥24
containing 61 channels corresponding to each object class.
(2) SUNCG navigation domain: In addition to the
SpriteWorld domain, we evaluate our approach on a ﬁrst
person image-based navigation task in an indoor house envi-
ronment where the agent must interact with objects. We use
an environment built on top of the SUNCG dataset ( Song
et al.,2017 ) which has previously been used in the context
of IRL ( Fu et al. ,2019 ) with language instructions. We
follow a similar task setup as Fu et al. (2019 ), although we
omit the language instructions. In this domain, we consider
tasks that can be categorized into two types.
Navigation (NA V): In this task, the agent must navigate
to a location in the house that corresponds either to a target
object or location. For example, in the blue line of Fig. 4,
the agent must navigate to the “cup” object.
Pick-and-place (PICK): In this more difﬁcult task, the
agent moves an object between two locations. For example,
in Fig 4, the agent must navigate to the “cup”, perform a
pick action and then navigate to the “bedroom”.
Evaluation protocol: We evaluate on held-out tasks that
were unseen during meta-training. In the SpriteWorld do-
main, we consider two settings: (1) tasks involving new
combinations and placements of sprites, with sprites that
were present during meta-training, and (2) tasks with combi-
nations of unseen sprites which we refer to as “out of domain
objects.” For each task, we generate one environment (a set
Figure 5. An example adaptation in an UNSEEN-HOUSE (best
viewed in color). The agent starts in one room (blue square) and
is required to pick up the vase (green square) and take it to the
living room (red square). The value function (blue is high, red is
low) under the learned reward (bottom) exhibits no “PICK” task
structure pre-adaptation (bottom left-column). Post-adaptation
(bottom right-column), the reward function successfully leads the
agent to the vase (bottom ﬁgure, top-right plot) and after the pick
action (bottom ﬁgure, bottom-right plot) is performed, navigates
the agent to the goal location.
of sprite positions) along with demonstrations for adapting
the reward, and generate a second environment (with new
sprite positions) for evaluating the adapted reward.
In the SUNCG domain, we similarly evaluate on both novel
combinations of objects and locations. We follow the eval-
uation protocol of Fu et al. (2019 ) and evaluate on “TEST”
tasks which consist of tasks within the same houses as train-
ing, but with novel combinations of objects and locations.
In addition, we evaluate on environments which consists
of new houses not in the training set. We refer to these as
“UNSEEN-HOUSES.” This evaluation adds complexity by
testing the models ability to successfully infer rewards in
an entirely new scene. In total, the dataset consists of 1413
tasks (716 PICK, 697 NA V). The meta-train set is composed
of 1004 tasks, the “TEST” set contains 236 tasks, and the
“UNSEEN-HOUSES” set contains 173 tasks.
Evaluation Metrics. We measure performance using
the expected value difference, which measures the sub-
optimality of a policy learned under the learned reward;
this is a performance metric used in prior IRL work ( Levine
et al.,2011 ;Wulfmeier et al. ,2015 ). The metric is computed
by taking the difference between the value of the optimal
policy under the learned reward and the value of the optimal
policy under the true reward. On the SUNCG domain, we
follow Fu et al. (2019 ) and report the success rate of the
optimal policy under the learned reward function.
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
Figure 6. Meta-test performance on the SpriteWorld domain (lower
is better): held-out tasks performance (top) and held-out tasks with
novel sprites (bottom). The recurrent meta-learner has a value
difference >60in both test settings. In both test settings, Man-
dRIL achieves comparable performance to the training environ-
ment, while the other methods overﬁt until they receive at least 10
demonstrations (see Appendix Cfor training environment perfor-
mance). We ﬁnd that pre-training on the full set of tasks leads to
negative transfer, while pre-training on a single task is comparable
to random initialization. ManDRIL outperforms both alternative
initialization approaches, which shows that optimizing for initial
weights for ﬁne-tuning robustly improves performance. Shaded
regions show 95% conﬁdence intervals.
Results. The results for SpriteWorld are shown in Fig. 6,
which illustrate test performance with in-distribution and
out-of-distribution sprites. Our approach, MandRIL,
achieves consistently better performance in both settings.
Most signiﬁcantly, our approach performs well even with
single-digit numbers of demonstrations. By comparison, al-
ternative meta-learning methods generally overﬁt consider-
ably, attaining good training performance (see Appendix. C
for curves) but poor test performance. Learning the reward
function from scratch is in fact the most competitive base-
line – as the number of demonstrations increases, simply
training the reward function from scratch on the new task is
the only method that matches the performance of MandRIL
when provided 20or more demonstrations. With only a
few demonstrations however, MandRIL has substantially
lower value difference. It is worth noting the performanceTable 1. Success rate (%) on heldout tasks with 5 demonstra-
tions. ManDRIL achieves consistently better performance on all
task/environment types. Results are averaged over 3 random seeds.
METHODTEST UNSEEN HOUSES
PICK NAV T OTAL PICK NAV TOTAL
BEHA VIORAL CLONING 0.4 8.2 4.3 3.7 12.0 9.4
MAXENTIRL ( AVG G R A D I E N T )3 7 . 38 3 . 7 6 0 . 8 3 8 . 3 8 9 . 7 7 3 . 3
MAXENTIRL ( FROM SCRATCH )4 2 . 48 7 . 9 6 5 . 4 4 8 . 18 9 . 9 7 6 . 5
MANDRIL( OURS ) 52.3 90.7 77.3 56.3 91.0 82.6
MANDRIL (P RE-ADAPTATION )6 . 0 3 5 . 3 2 0 . 74 . 3 3 4 . 6 2 5 . 3
of MandRIL on the out of distribution test setting (Fig. 6,
bottom): although the evaluation is on new sprites, Man-
dRIL is still able to adapt via gradient descent and exceed
the performance all other methods.
In both domains, we perform a comparison to representa-
tions ﬁnetuned from a supervised pre-training phase in Fig. 6
and Table 5. We compare against an approach that follows
the mean gradient across the tasks at meta-training time and
is ﬁne-tuned at meta-test time which we ﬁnd consistently
leads to negative transfer. We conclude that ﬁne tuning
reward functions learned in this manner is not an effective
way of using prior task information. In contrast, we ﬁnd that
our approach, which explicitly optimizes for initial weights
for ﬁne-tuning, robustly improves performance on all task
types and test settings. By visualizing the value under the
learned reward function (see Fig. 5), we see that even with
a small number of gradient steps, the reward function can
be effectively adapted to an unseen home layout.
Note that the SUNCG task is substantially more challeng-
ing, requiring the reward function to interpret ﬁrst-person
images. Indeed, the pretrained MaxEntIRL algorithm that
does not use meta-learning exhibits negative transfer, as
illustrated by the lower performance of this method on all
tasks as compared to the learning “FROM SCRATCH ”ver-
sion, which learns each task entirely from random initial-
ization. Training from scratch is a strong baseline here,
because the method still sees every single ﬁrst-person image
in the house – 2257.7 images on average. This provides
sufﬁcient variety to learn effective visual features in many
cases. Nonetheless, our method (last row in Table 1) pro-
duces a substantial improvement, especially on the much
harder “PICK” task, demonstrating that meta-learning can
produce positive transfer even when pre-training does not.
6. Conclusion
In this work, we present an approach that enables few-shot
learning of reward functions. We achieve this through a
novel formulation of IRL that learns to encode common
structure across tasks. Using our meta-IRL approach, we
show that we can leverage data from previous tasks to effec-
tively learn reward functions from raw pixel observations
for new tasks, from only a handful of demonstrations. Our
work paves the way for future work that considers unknown
dynamics, or work that employs more fully probabilistic
approaches to reward and goal inference.
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
ACKNOWLEDGMENTS
We thank Frederik Ebert, Adam Gleave, Erin Grant, Sergio
Guadarrama, Rowan McAllister, Charlotte Nguyen, Sid
Reddy and Aravind Srinivas for comments on an earlier
version of this paper. This work as supported by the Open
Philanthropy Foundation, NVIDIA, NSF IIS 1651843, IIS
1700696. We also acknowledge computing support from
Amazon. CF was supported by an NSF graduate research
fellowship.
References
Abbeel, P. and Ng, A. Y . Apprenticeship learning via inverse
reinforcement learning. In Proceedings of the Twenty-ﬁrst
International Conference on Machine Learning , ICML,
New York, NY, USA, 2004.
Alet, F., Lozano-P ´erez, T., and Kaelbling, L. P. Modular
meta-learning. arXiv preprint arXiv:1806.10166 , 2018.
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schul-
man, J., and Man ´e, D. Concrete problems in ai safety.
arXiv preprint arXiv:1606.06565 , 2016.
Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W.,
Pfau, D., Schaul, T., and de Freitas, N. Learning to learn
by gradient descent by gradient descent. In Advances in
Neural Information Processing Systems , pp. 3981–3989,
2016.
Babe s¸-Vroman, M., Marivate, V., Subramanian, K., and
Littman, M. Apprenticeship learning about multiple in-
tentions. In International Conference on International
Conference on Machine Learning , ICML, USA, 2011.
Baker, C., B Tenenbaum, J., and R Saxe, R. Goal inference
as inverse planning. 01 2007.
Bengio, Y., Bengio, S., and Cloutier, J. Learning a synaptic
learning rule .
Choi, J. and Kim, K.-E. Nonparametric bayesian inverse re-
inforcement learning for multiple reward functions. In Ad-
vances in Neural Information Processing Systems , NIPS,
USA, 2012.
Dimitrakakis, C. and Rothkopf, C. A. Bayesian multitask
inverse reinforcement learning. In European Conference
on Recent Advances in Reinforcement Learning , EWRL,
2012.
Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever,
I., and Abbeel, P. Rl2: Fast reinforcement learn-
ing via slow reinforcement learning. arXiv preprint
arXiv:1611.02779 , 2016.Duan, Y., Andrychowicz, M., Stadie, B. C., Ho, J., Schnei-
der, J., Sutskever, I., Abbeel, P., and Zaremba, W. One-
shot imitation learning. In NIPS , pp. 1087–1098, 2017.
Duvenaud, D., Maclaurin, D., and Adams, R. Early stop-
ping as nonparametric variational inference. In Artiﬁcial
Intelligence and Statistics , pp. 1070–1077, 2016.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-
learning for fast adaptation of deep networks. In Interna-
tional Conference on Machine Learning , 2017a.
Finn, C., Yu, T., Zhang, T., Abbeel, P., and Levine, S. One-
shot visual imitation learning via meta-learning. arXiv
preprint arXiv:1709.04905 , 2017b.
Fu, J., Luo, K., and Levine, S. Learning robust rewards with
adverserial inverse reinforcement learning. International
Conference on Learning Representations , 2018.
Fu, J., Korattikara, A., Levine, S., and Guadarrama, S. From
language to goals: Inverse reinforcement learning for
vision-based instruction following. International Confer-
ence on Learning Representations (ICLR) , 2019.
Glorot, X. and Bengio, Y. Understanding the difﬁculty
of training deep feedforward neural networks. In Pro-
ceedings of the thirteenth international conference on
artiﬁcial intelligence and statistics , pp. 249–256, 2010.
Grant, E., Finn, C., Levine, S., Darrell, T., and Grifﬁths, T.
Recasting gradient-based meta-learning as hierarchical
bayes. International Conference on Learning Represen-
tations (ICLR) , 2018.
Hausman, K., Chebotar, Y., Schaal, S., Sukhatme, G., and
Lim, J. J. Multi-modal imitation learning from unstruc-
tured demonstrations using generative adversarial nets.
InAdvances in Neural Information Processing Systems ,
pp. 1235–1245, 2017.
Ho, J. and Ermon, S. Generative adversarial imitation learn-
ing. In Neural Information Processing Systems (NIPS) ,
2016.
Hochreiter, S. and Schmidhuber, J. Long short-term memory.
Neural computation , 9(8):1735–1780, 1997.
Kingma, D. P. and Ba, J. Adam: A method for stochastic
optimization. arXiv preprint arXiv:1412.6980 , 2014.
Kober, J. and Peters, J. Reinforcement learning in robotics:
A survey. In Reinforcement Learning , pp. 579–610.
Springer, 2012.
Koch, G. Siamese neural networks for one-shot image
recognition. 2015.
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
Kueﬂer, A. and Kochenderfer, M. J. Burn-in demonstrations
for multi-modal imitation learning. 2018.
Levine, S., Popovic, Z., and Koltun, V. Nonlinear inverse
reinforcement learning with gaussian processes. In Ad-
vances in Neural Information Processing Systems , pp.
19–27, 2011.
Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end
training of deep visuomotor policies. Journal of Machine
Learning Research (JMLR) , 17(39):1–40, 2016.
Li, K. and Burdick, J. W. Meta inverse reinforcement learn-
ing via maximum reward sharing for human motion anal-
ysis. CoRR , abs/1710.03592, 2017.
Li, K. and Malik, J. Learning to optimize neural nets. arXiv
preprint arXiv:1703.00441 , 2017.
Li, Y., Song, J., and Ermon, S. Inferring the latent structure
of human decision-making from raw visual inputs. arXiv
preprint arXiv:1703.08840 , 2017.
Mishra, N., Rohaninejad, M., Chen, X., and Abbeel, P.
Meta-learning with temporal convolutions. arXiv preprint
arXiv:1707.03141 , 2017.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness,
J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje-
land, A. K., Ostrovski, G., et al. Human-level control
through deep reinforcement learning. Nature , 518(7540):
529–533, 2015.
Mordatch, I. Concept learning with energy-based models.
InICLR Workshop , 2018.
Naik, D. K. and Mammone, R. Meta-neural networks that
learn by learning. In Neural Networks, 1992. IJCNN.,
International Joint Conference on , volume 1, pp. 437–
442. IEEE, 1992.
Ng, A. Y. and Russell, S. J. Algorithms for inverse rein-
forcement learning. In Proceedings of the Seventeenth
International Conference on Machine Learning , ICML
’00, San Francisco, CA, USA, 2000. Morgan Kaufmann
Publishers Inc.
Ng, A. Y., Harada, D., and Russell, S. Policy invariance
under reward transformations: Theory and application to
reward shaping. 1999.
Ramachandran, D. and Amir, E. Bayesian inverse rein-
forcement learning. In International Joint Conference on
Artiﬁcal Intelligence , San Francisco, CA, USA, 2007.
Ratliff, N. D., Bagnell, J. A., and Zinkevich, M. A. Maxi-
mum margin planning. In Proceedings of the 23rd inter-
national conference on Machine learning , pp. 729–736.
ACM, 2006.Ravi, S. and Larochelle, H. Optimization as a model for
few-shot learning. 2016.
Reed, S., Chen, Y., Paine, T., van den Oord, A., Vinyals, O.,
Eslami, S. A., Rezende, D., and de Freitas, N. Few-shot
autoregressive density estimation: Towards learning to
learn distributions. In ICLR , 2018.
Rezende, D. J., Mohamed, S., Danihelka, I., Gregor, K., and
Wierstra, D. One-shot generalization in deep generative
models. arXiv preprint arXiv:1603.05106 , 2016.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imita-
tion learning and structured prediction to no-regret online
learning. In Proceedings of the fourteenth international
conference on artiﬁcial intelligence and statistics , pp.
627–635, 2011.
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., and
Lillicrap, T. Meta-learning with memory-augmented neu-
ral networks. In International conference on machine
learning , 2016.
Santos, R. J. Equivalence of regularization and truncated
iteration for general ill-posed problems. Linear algebra
and its applications , 236:25–33, 1996.
Schmidhuber, J. Evolutionary principles in self-referential
learning, or on learning how to learn: the meta-meta-
... hook . PhD thesis, Technische Universit ¨at M ¨unchen,
1987.
Shelhamer, E., Long, J., and Darrell, T. Fully convolu-
tional networks for semantic segmentation. IEEE Trans-
actions on Pattern Analysis and Machine Intelligence , 39
(4), April 2017.
Shyam, P., Gupta, S., and Dukkipati, A. Attentive recurrent
comparators. arXiv preprint arXiv:1703.00767 , 2017.
Siegelmann, H. T. and Sontag, E. D. On the computational
power of neural nets. Journal of computer and system
sciences , 50(1):132–150, 1995.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L.,
Van Den Driessche, G., Schrittwieser, J., Antonoglou, I.,
Panneershelvam, V., Lanctot, M., et al. Mastering the
game of go with deep neural networks and tree search.
Nature , 529(7587):484–489, 2016.
Sj¨oberg, J. and Ljung, L. Overtraining, regularization and
searching for a minimum, with application to neural net-
works. International Journal of Control , 62(6):1391–
1407, 1995.
Snell, J., Swersky, K., and Zemel, R. Prototypical networks
for few-shot learning. In Advances in Neural Information
Processing Systems , pp. 4080–4090, 2017.
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
Song, S., Yu, F., Zeng, A., Chang, A. X., Savva, M., and
Funkhouser, T. Semantic scene completion from a single
depth image. IEEE Conference on Computer Vision and
Pattern Recognition , 2017.
Synnaeve, G., Nardelli, N., Auvolat, A., Chintala,
S., Lacroix, T., Lin, Z., Richoux, F., and Usunier,
N. Torchcraft: a library for machine learning re-
search on real-time strategy games. arXiv preprint
arXiv:1611.00625 , 2016.
Thrun, S. and Pratt, L. Learning to learn . Springer Science
& Business Media, 2012.
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.
Matching networks for one shot learning. In Advances in
Neural Information Processing Systems , pp. 3630–3638,
2016.
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H.,
Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and
Botvinick, M. Learning to reinforcement learn. arXiv
preprint arXiv:1611.05763 , 2016.
Wang, Y. and Hebert, M. Learning to learn: Model regres-
sion networks for easy small sample learning. In Euro-
pean Conference on Computer Vision (ECCV) , October
2016.
Wulfmeier, M., Ondruska, P., and Posner, I. Maximum
entropy deep inverse reinforcement learning. In Neural
Information Processing Systems Conference, Deep Rein-
forcement Learning Workshop , volume abs/1507.04888,
2015.
Wulfmeier, M., Rao, D., and Posner, I. Incorporating human
domain knowledge into large scale cost function learning.
CoRR , abs/1612.04318, 2016a.
Wulfmeier, M., Wang, D. Z., and Posner, I. Watch this: Scal-
able cost-function learning for path planning in urban en-
vironments. In 2016 IEEE/RSJ International Conference
on Intelligent Robots and Systems (IROS) , pp. 2089–2095,
Oct 2016b.
Zaremba, W., Sutskever, I., and Vinyals, O. Recurrent neural
network regularization. arXiv preprint arXiv:1409.2329 ,
2014.
Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K.
Maximum entropy inverse reinforcement learning. In
Proceedings of the 23rd National Conference on Artiﬁcial
Intelligence - Volume 3 , AAAI’08. AAAI Press, 2008.

Kelvin Xu
Ellis Ratner
Anca Dragan
Sergey Levine
Chelsea Finn