Hindsight Task Relabelling: Experience Replay for Sparse Reward Meta-RL.

Hindsight Task Relabelling:
Experience Replay for Sparse Reward Meta-RL
Charles Packer
UC BerkeleyPieter Abbeel
UC BerkeleyJoseph E. Gonzalez
UC Berkeley
Abstract
Meta-reinforcement learning (meta-RL) has proven to be a successful framework
for leveraging experience from prior tasks to rapidly learn new related tasks, how-
ever, current meta-RL approaches struggle to learn in sparse reward environments.
Although existing meta-RL algorithms can learn strategies for adapting to new
sparse reward tasks, the actual adaptation strategies are learned using hand-shaped
reward functions, or require simple environments where random exploration is
sufﬁcient to encounter sparse reward. In this paper, we present a formulation of
hindsight relabeling for meta-RL, which relabels experience during meta-training
to enable learning to learn entirely using sparse reward. We demonstrate the ef-
fectiveness of our approach on a suite of challenging sparse reward goal-reaching
environments that previously required dense reward during meta-training to solve.
Our approach solves these environments using the true sparse reward function, with
performance comparable to training with a proxy dense reward function.
1 Introduction
Reinforcement learning (RL) has seen tremendous success applied to challenging games (Mnih et al.,
2015; Silver et al., 2017) and robotic control (Lillicrap et al., 2015; Levine et al., 2016), driven by
advances in compute and the use of deep neural networks as powerful function approximators in
RL algorithms. However, agents trained using deep RL often struggle to meaningfully utilize past
experience to learn new tasks, even if the new tasks differ only slightly from tasks seen during training
(Zhang et al., 2018; Packer et al., 2018; Cobbe et al., 2019). In contrast, humans are adept at utilizing
prior experience to rapidly acquire new skills and adapt to unseen environments.
Meta-reinforcement learning (meta-RL) aims to address this limitation by extending the RL frame-
work to explicitly consider structured distributions of tasks (Schmidhuber, 1987; Bengio et al., 1990;
Thrun & Pratt, 1998). Whereas conventional RL is concerned with learning a single task, meta-RL is
concerned with learning to learn , that is, learning how to quickly learn a new task by leveraging prior
experience on related tasks. Meta-RL methods generally utilize a limited amount of experience in a
new environment to estimate a latent task embedding which conditions the policy, or to compute a
policy gradient which is used to directly update the parameters of the policy.
A major challenge in both RL and meta-RL is learning with sparse rewards. When rewards are
sparse or delayed, the adaptation stage in meta-RL becomes extremely difﬁcult: inferring the task at
hand requires receiving reward signal from the environment, which in the sparse reward setting only
happens after successfully completing the task. Due to this inherent incompatibility between meta-RL
and sparse rewards, existing meta-RL algorithms that consider the sparse reward setting either only
work in simple environments that do not require temporally-extended exploration strategies for
adaptation (Duan et al., 2016; Stadie et al., 2018), or train exclusively using dense reward functions
(Gupta et al., 2018b; Rakelly et al., 2019), which are designed to encourage an agent to learn
adaptation strategies that can be directly applied to the original sparse reward setting.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
(a) Hindsight relabeling in goal-conditioned RL
 (b) Hindsight Task Relabeling (HTR) in meta-RL
Figure 1: In goal-conditioned RL (a), an agent must navigate to a provided goal location g(ﬁlled
circle, revealed to the agent). An unsuccessful attempt for goal gprovides no sparse reward signal,
but can be relabelled as a successful attempt for goal g0, creating sparse reward that can be used to
train the agent. In meta-RL (b), the task T(i.e., goal, hollow circle) is never revealed to the agent,
and instead must be inferred using experience on prior tasks and limited experience ( 1:t1) on the
new task. In (b), there is no shared optimal task T0to relabel all attempts with. HTR relabels each
attemptunder its own hindsight task T0, and modiﬁes the underlying meta-RL training loop to
learn adaptation strategies on the relabelled tasks. Note that we include multiple trajectories in
(b) vs a single trajectory in (a) to highlight the adaptation stage in meta-RL, which does not exist in
goal-conditioned RL and requires signiﬁcantly different sampling and relabeling procedures.
In this paper, we show that the concept of hindsight relabeling (Andrychowicz et al., 2017) from
conventional RL can be applied in meta-RL to enable learning to learn in the sparse reward setting.
Our key insight is that data collected on the true training tasks can be relabelled as pseudo-expert
data for easier hindsight tasks, bootstrapping meta-training with the reward signal needed to train
the agent. We introduce a new algorithm for off-policy meta-RL, which we call Hindsight Task
Relabeling (HTR), and demonstrate its effectiveness by achieving state-of-the-art performance on a
collection of challenging sparse reward environments that previously required shaped reward to solve.
Not only is our approach able to learn adaptation strategies only using sparse reward, but it also learns
strategies with comparable performance to existing approaches that use shaped reward functions.
2 Related Work
In meta-RL, an agent learns an adaptation strategy by repeatedly adapting to training tasks in the
inner loop of meta-training , with the hope that the learned adaptation procedure will generalize to
new, unseen tasks during meta-testing . Context-based methods use recent experience (i.e., context, in
the form of trajectories or transitions) in a new task to estimate a latent task embedding, and differ
mainly in how this embedding is computed: Duan et al. (2016); Wang et al. (2016); Mishra et al.
(2018) propose aggregating context into the hidden state of the policy, whereas Rakelly et al. (2019)
propose explicitly feeding the task embedding as input to the policy. Gradient-based methods use
recent context to update differentiable hyperparameters (Xu et al., 2018b), loss functions (Sung et al.,
2017; Houthooft et al., 2018), or to directly update policy parameters (Finn et al., 2017; Stadie et al.,
2018; Xu et al., 2018a; Zintgraf et al., 2018; Gupta et al., 2018b; Rothfuss et al., 2019).
Many of the aforementioned approaches struggle to learn effective exploration strategies for tasks
with sparse or delayed rewards. Methods that directly update the policy either implicitly (e.g., with a
hidden state) or explicitly (e.g., with gradients) are generally unable to learn a policy that meaningfully
explores, since the primary source of randomness is action-space, and thus is time-invariant. Gupta
et al. (2018b) and Rakelly et al. (2019) propose using a probabilistic task variable that is sampled once
per episode, which makes the primary source of variability task-dependent and enables temporally-
extended exploration. While this approach enables effective adaptation in the sparse reward setting,
both Gupta et al. (2018b) and Rakelly et al. (2019) learn the actual adaptation strategies using shaped
(dense) reward functions during meta-training. In the environments they consider, the adaptation
strategies learned using dense rewards generalize to sparse rewards despite the difference in reward
structure; however, engineering a suitable reward function can be costly (Ng et al., 1999) and error
prone (Clark & Amodei, 2016), and thus it is highly desirable to devise a mechanism for learning to
learn in the sparse reward setting that does not require an auxiliary dense reward function.
2
Our proposed method is closely related to prior work in unsupervised meta-RL and goal generation.
Unsupervised meta-RL (Jabri et al., 2019; Gupta et al., 2018a) considers the meta-RL setting without
access to a training task distribution; training tasks are instead self-generated with the aim of learning
skills useful for the test task distribution. Our proposed method also generates a curriculum of training
tasks, but with the intent of learning adaptation strategies in the sparse reward setting, as opposed to
learning transferable skills in a setting with no rewards at all. Several methods exist for curriculum
generation in the context of goal-conditioned policies, including adversarial goal generation (Florensa
et al., 2018) and goal relabeling (Kaelbling, 1993; Andrychowicz et al., 2017; Levy et al., 2017; Nair
et al., 2018). Recent work (Li et al., 2020; Eysenbach et al., 2020) has proposed the use of inverse
RL to generalize goal relabeling to broader families of tasks. Unlike prior work on goal and task
relabeling that use goal- or task-conditioned policies, we focus on the meta-RL setting where the task
is unknown to the policy and must be inferred through deliberate and coherent exploration.
3 Background
In this section we formalize the meta-RL problem setting and brieﬂy describe two algorithms which
our approach, Hindsight Task Relabeling, builds on: Probabilistic Embeddings for Actor-critic RL
(PEARL) and Hindsight Experience Replay (HER).
3.1 Meta-Reinforcement Learning (Meta-RL)
Conventional RL assumes environments are modeled by a Markov Decision Processes (MDP) M,
deﬁned by the tuple (S;A;p;r;
; 0;T)whereSis the set of possible states, Ais the set of actions,
p:SAS! R0is the transition probability density, r:SA! Ris the reward function,
is
the discount factor, 0:S!R0is the initial state distribution at the beginning of each episode,
andTis the time horizon of an episode (Sutton & Barto, 2017).
Letstandatbe the state and action taken at time t. At the beginning of each episode, s0
0(). Under a stochastic policy mapping a sequence of states to actions, at(atjst;;s0)
andst+1p(st+1jst;at), generating a trajectory =fst;at;r(st;at)g, wheret= 0;1;.
Conventional RL algorithms, which assume the environment is ﬁxed, learn to maximize the
expected reward per episode JM() =EhPT
t=0
trti
, where rt=r(st;at). They often utilize
the concepts of a value function V
M(s), the expected reward conditional on s0=sand a state-action
value function Q
M(s;a), the expected reward conditional on s0=sanda0=a.
Meta-RL algorithms assume that there is a distribution of tasks p(T), and usually maximize the
expected reward over the distribution, E
Tp(T)[JT()]. The distribution of tasks corresponds to a
distribution of MDPs M(T), whereTcan paramaterize an MDP’s dynamics or reward. In this work
we focus on the latter case, where each Timplies an MDP with a different reward function under the
same dynamics. The agent is trained on a set of training tasks Ttrainp(T)during meta-training,
and evaluated on a held-out set of testing tasks Ttestp(T)during meta-testing.
An important distinction between meta-RL and multi-task or goal-conditioned RL is that in meta-RL,
the task is never explicitly revealed to the agent or policy through observations s. Instead,must
explore the environment to infer the task to maximize expected reward. This crucial difference is one
reason why Hindsight Experience Replay cannot be directly applied to the meta-RL setting.
3.2 Off-Policy Meta-Reinforcement Learning
We demonstrate the effectiveness of our approach by building on top of Probabilistic Embeddings
for Actor-critic RL (PEARL) by Rakelly et al. (2019), an off-policy meta-RL algorithm which itself
is built on top of soft actor-critic (SAC) by Haarnoja et al. (2018). SAC is an actor-critic algorithm
based on the maximum entropy RL objective that optimizes a stochastic policy with off-policy
data. In addition to the policy (actor) , SAC concurrently learns twin state-action value functions
(critics)Q. Transitions collected using the policy are added to the replay buffer B, andQandare
optimized using loss functions regularized by the entropy of .
Rakelly et al. (2019) extend SAC to the meta-RL setting by adding a context encoder that uses
recent history in an environment to estimate a task embedding, which is then utilized by the policy.
3
(a) Point Robot
 (b) Wheeled Locomotion
 (c) Ant Locomotion
Figure 2: Sparse reward environments for meta-RL that require temporally-extended exploration. In
each environment, the task (the top-left circle in (a), the green sphere in (b) and (c)) is not revealed
to the agent via the observation. The agent must instead infer the task through temporally-extended
exploration (illustrated by the dotted lines in (a)), since no reward signal is provided until the task is
successfully completed. Prior meta-RL methods such as PEARL (Rakelly et al., 2019) and MAESN
(Gupta et al., 2018b) are only able to (meta-)learn meaningful adaptation strategies using dense
reward functions. Our approach, Hindsight Task Relabeling (HTR), can (meta-)train with the original
sparse reward function and does not require additional dense reward functions.
Speciﬁcally, PEARL uses a network q(zjc)that takes as input recently collected data (i.e., context
c) to infer a probabilistic latent context variable Z. Samples from the latent context variable condition
the actor,(ajs;z), and critic,Q(s;a;z). Instead of a single replay buffer (as in SAC), there are
Tseparate buffers Bi=1:::T, corresponding to the number of tasks sampled for Ttrain . The separate
per-task replay buffers enable sampling off-policy (but on- task) context during the inner loop of
meta-training, which signiﬁcantly improves data efﬁciency.
During meta-training, transitions are collected for each Biby sampling zq(zjc)and acting
according to (ajs;z).zis periodically sampled during data collection such that the context
contains transitions collected using the policy conditioned on the prior, as well the same policy
conditioned on the posterior. Gradients used to optimize q,andQare computed in task-speciﬁc
batches across a random subset of training tasks Tip(T), each with an associated replay buffer Bi.
During meta-testing, for each test task Tip(T), an empty buffer Biis initialized, zis sampled
from the prior, and the policy conditioned on zis rolled out. As additional context is added to the
replay buffer, the belief over znarrows, enabling the policy to adapt to the task at hand.
3.3 Hindsight Experience Replay
Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) is a method for off-policy goal-
conditioned RL in environments with sparse reward. The key insight behind HER is that unsuccessful
attempts (i.e., attempts that received no reward signal) generated via a goal-conditioned policy can
be relabelled as successful attempts for a ‘hindsight’ goal that was actually achieved, e.g., the ﬁnal
state (see Figure 1). HER can be applied to any off-policy RL algorithm that uses goal-conditioned
policies (where the actor and/or critic receive the desired goal as part of the state, i.e., (ajs;g)
andQ(s;a;g)). When a transition fst;at;r(st;at;g)jggis sampled from the replay buffer during
training, with some probability HER rewrites the desired goal gwith an achieved goal g0and
recomputes the reward under the new goal. The modiﬁed transition fst;at;r(st;at;g0)jg0gis
used to optimize (ajs;g0)andQ(s;a;g0). Hindsight relabeling generates an implicit curriculum:
initially, sampled transitions are rewritten with relatively ‘easy’ goals achievable with random policies,
and as training progresses, relabelled goals are increasingly likely to be near the true desired goals.
4 Leveraging Hindsight in Meta-Reinforcement Learning
Our algorithm, Hindsight Task Relabeling, applies ideas from hindsight relabeling to the meta-RL
setting to enable learning adaptation strategies for tasks with sparse or delayed rewards. Similar to
how policies in HER are conditioned on a goal g, policies in context-based meta-RL are conditioned
on a latent task embedding z. A crucial difference in meta-RL is that the task is hidden from the
agent: the agent must infer relevant features of the task at hand using experience on prior tasks and a
limited amount of experience gathered in the environment. Our key insight is that an unsuccessful
experience collected in an unknown task can be relabelled as a successful experience for a known
hindsight task, i.e., a transition fst;at;r(st;at;T)ggenerated under an unknown Tcan be rewritten
under a knownT0asfst;at;r(st;at;T0)g, regardless of whether T02Ttrain .
4
Figure 3: Illustration of Hindsight Task Relabeling (HTR) in a meta-RL training loop. HTR is
agnostic to the underlying (off-policy) meta-RL algorithm; the agent architecture and/or training
speciﬁcs (e.g., the encoder , actorandQ-function neural networks shown in blue) can be modiﬁed
independently of the relabeling scheme. HTR can also be performed in an ‘eager’ fashion at the data
collection stage (as opposed to ‘lazy’ relabeling in the data sampling stage), see Section 3 for details.
Relabeling transitions under hindsight tasks serves as an additional source of supervision in learning
the latent task embedding z(f(c)by enabling gradient-based optimization of fin the absence of
meaningful reward signal (in PEARL, this embedding is parameterized by the context encoder q).
Similar to how HER generates an implicit curriculum of goals, HTR generates an implicit curriculum
of tasks that gradually shifts from easier hindsight tasks towards the true training distribution
Ttrainp(T): initially, the agent is unable to recover reward signal on the true training tasks, but it
can bootstrap the learning of an adaptation strategy by learning on easier hindsight tasks.
We outline our method for meta-training with Hindsight Task Relabeling in Algorithm 1 (task
relabeling is only used during meta-training; meta-testing is unchanged). We evaluate our approach
using PEARL as the off-policy meta-RL algorithm A, though the approach we describe is general to
any off-policy meta-RL algorithm, and could potentially be integrated with on-policy context-based
meta-RL algorithms (such as RL2) via hindsight policy gradients (Rauber et al., 2019).
4.1 Algorithm Design
Two important considerations in adapting hindsight relabeling to meta-RL are (i) how to choose the
hindsight task, and (ii) how to sample the transitions to be relabelled. Together these choices form
the task relabeling strategy Sin Algorithm 1. In HER, transitions are relabelled with goals that are
optimal at the trajectory-level (see Figure 1). The key difference between our method and HER is that
our meta-RL relabeling scheme needs to construct batches of data that share the same pseudo-task
(or real task), which is fundamentally different from standard HER. HER does not consider the
setting where batches of trajectories are collected in different MDPs; in HER, transitions are relabeled
independently with locally-optimal pseudo-goals, and batches of training data for the agent contain
many different pseudo-goals (and real goals). In the meta-RL setting, the agent must estimate the
task from a recent context of transitions with the same underlying task (unknown to the agent), and
therefore relabeling should take into consideration more than a single trajectory; the agent explores
an environment which has a ﬁxed task T p(T), and as it accumulates context it uses the context to
identify the singular task and update its policy accordingly.
A naive application of hindsight relabeling to meta-RL would assign each transition in the batch its
own hindsight task (e.g., give each in Figure 1b its own T0). This is a simple and straightforward
way to apply HER to meta-RL that may seem correct at the surface-level, however, this is bound to
fail if the meta-RL training process is left unchanged, since the context encoder is trained to estimate
the task from a collection of transitions generated in that task (not a batch of transitions generated
5
Algorithm 1: Hindsight Task Relabelling for Off-Policy Meta-Reinforcement Learning
Require: Off-policy meta-RL algorithm A, training tasksT1:Tp(T), context encoder ,
actor, criticQ, relabelling function R, task relabelling strategy S, and task
relabelling probability K
1Initialize a replay buffer Bifor each training task Ti
2while meta-training do
3 foreachTido
4 Collect data onTiaccording to A(e.g., by rolling out conditioned on z)
5 Add data to task replay buffer Bi
6 foreachTido
7phU(0;1)
8 ifphKthen
9 Select a minibatch of transitions biand hindsight task Thusing strategy S
10 (e.g., sample transitions from trajectory t, and select a state reached in tforTh)
11 Relabel transitions in biaccording toTh, i.e.,R(bi;Th)
12 else
13 Sample minibatch of transitions biBi
14 Compute gradients and update ,, andQusingbiaccording to A
across several distinct tasks). A similarly ﬂawed approach is to generate a single hindsight task
from a random transition in the batch, and then to relabel the entire batch of transitions with the
same hindsight task (e.g., choose a T0Figure 1b using a random to relabel all s with). The issue
with this approach is that the hindsight task is only guaranteed to be optimal for a single trajectory,
and may in fact be highly sub-optimal for other trajectories in the batch. We present two relabeling
strategies, Single Episode Relabeling (SER) andEpisode Clustering (EC) , that are inspired by the
relabeling strategy used in Andrychowicz et al. (2017) but are speciﬁcally designed to work in the
meta-RL setting.
4.2 Single Episode Relabeling (SER) strategy
In SER, a single episode (i.e., trajectory) is randomly selected to generate the hindsight task and
sample transitions from (i.e., sample Ntransitions from 1 trajectory, each relabelled under 1 new
task). Despite the fact that this reduces the pool of context from the entire task buffer to a single
episode, this setting closely resembles meta-testing (where context is drawn purely online), and we
found this resampling strategy to work well in practice since it results in more relabelled transitions
with non-zero reward. HTR using SER is outlined in Algorithm 1.
4.3 Episode Clustering (EC) strategy
An alternative strategy to Single Episode Resampling is to cluster trajectories that satisﬁed similar
hindsight tasks into the same task buffers Bh, essentially creating additional task buffers for hindsight
tasks that can be used for training in-place of the real task buffers Bi(which are also sampled with
some probability K). For example, in goal-reaching tasks, trajectories can easily be mapped to
hindsight buffers with trajectories from separate trials by discretizing the state space and creating a
buffer for each partition. In more complex tasks, alternative clustering approaches (e.g., learning-
based) can be used to group together trajectories that can be relabeled with high reward conditioned
on the same tasks. Episode clustering relabels a similar number of transitions with non-zero reward
to the resampling strategy, but with less duplicate transitions per-batch. In practice, we found the
SER strategy to be far less complex and similarly effective to the EC approach. See Section 5 for an
empirical comparison between SER and EC, and the supplement for a more detailed comparison.
4.4 Comparison of HTR and HER
Both SER and EC are similar to the relabeling strategy used in HER in some respects, but differ
in others: HER samples Ntransitions from the full replay buffer, and relabels each transition
independently ( Ndifferent goals). HTR with SER samples 1 trajectory, and samples Ntransitions
from that trajectory all relabelled with 1 new task. HTR with EC samples Ntransitions from 1
6
hindsight replay buffer, where each transition in that buffer has been relabeled with 1 new task. In
HER, the hindsight goal for a given transition is chosen by selecting a random transition that occurs
after the sampled transition in the same trajectory. In HTR, hindsight tasks are assigned in the same
way as HER, but a single hindsight task is applied to an entire batch of transitions.
An important distinction between HER and HTR is that in HER, bootstrapping training with hindsight
goals does not change the optimal policy, since a goal-conditioned policy with sufﬁcient capacity
should (in theory) be able to represent an optimal policy for every goal (including both original
goals and hindsight goals). In contrast, the optimal exploration strategy for a particular meta-RL
environment depends on its true task distribution, and bootstrapping training with hindsight tasks
may expand the training task distribution in a manner that changes the optimal exploration strategy.
One way to beneﬁt from HTR’s bootstrapping while ensuring the ﬁnal learned strategy is optimal for
the original task distribution is to only relabel data from tasks on which the agent has never received
(non-zero) reward. Similarly, another solution is to anneal the relabelling probability Kto zero during
meta-training. Neither method was required to achieve the results shown in this paper, however, we
expect they may be useful if applying HTR to other meta-RL environments.
4.5 Limitations
The key limitation of our method is that it assumes trajectories can be relabeled under new task
(i.e., a mapping s!T or!T exists), which is a reasonable assumption for goal-reaching
environments (as studied in Rakelly et al. (2019); Gupta et al. (2018b); Andrychowicz et al. (2017)),
but may be signiﬁcantly more challenging in other environments. Imagine a complex robotic
manipulation environment (e.g., retrieving an object from a drawer) where reward is sparse and only
received upon successful completion of the entire task; in this scenario, relabeling the zero-reward
transitions generated by an initial random policy into useful signal for training an optimal policy is
less straightforward than the goal-reaching setting, and may require explicit sub-task speciﬁcation
using domain expertise.
Similar to how HER is not necessarily only for goal-reaching goal-conditioned RL, HTR is not
necessarily only for goal-reaching meta -RL, and if a good relabeling function exists then either
approach can be successfully applied in their respective setting (HER for goal-conditioned RL, HTR
for meta-RL). However, as in the original HER work, we focus our experiments on goal-reaching
tasks, where the reward function is a simple function of the state and can be easily repurposed for
relabeling. It may be possible to relax this assumption and extend HTR to more families of tasks
by employing more complex task relabeling schemes, e.g., by extending work on inverse RL for
hindsight relabeling (Li et al., 2020; Eysenbach et al., 2020) to the meta-RL setting, or by carefully
engineering reward functions speciﬁc to each environment, however we leave this to future work.
Additionally, our method adds a new hyperparameter Kto the underlying off-policy meta-RL
algorithm, which may need to be tuned for optimal results. The speciﬁcation and tuning of a
relabeling probability Kis also required in standard HER (see Andrychowicz et al. (2017)).
5 Experiments
In evaluating our proposed method, we aim to answer the following questions. (i) Can Hindsight Task
Relabeling enable (meta-)learning of adaptation strategies in challenging sparse reward environments,
where existing meta-RL methods fail without shaped reward? (ii) How do the adaptation strategies
learned using Hindsight Task Relabeling compare to those learned using shaped reward functions?
(iii) How do key implementation choices (such as relabeling probability K) affect its performance?
5.1 Environments
We evaluate our method on a suite of sparse reward environments based those proposed by Gupta
et al. (2018b) and Rakelly et al. (2019) (see Figure 2). In prior work, each environment exposes two
reward functions: a dense reward function used during meta-training, and a sparse reward function
used during meta-testing. The key difference in our experimental setup is that we consider the setting
where sparse reward is used both during meta-testing and meta-training . In each environment, a
set of 100tasks is sampled for meta-training, and a set of 100tasks is sampled from the same task
distribution for meta-testing. The environments were each modiﬁed to increase their difﬁculty, such
7
0 200 400 600 800 1000
Meta-training iterations0.00.51.01.52.02.53.0Average returnPoint Robot
shaped (oracle)
sparse + HTR
sparse
0 250 500 750 1000 1250 1500 1750 2000
Meta-training iterations410
400
390
380
370
360
350
340
Average returnWheeled Locomotion
shaped (oracle)
sparse + HTR
sparse
0 250 500 750 1000 1250 1500 1750 2000
Meta-training iterations300325350375400425450475500Average returnAnt Locomotion
shaped (oracle)
sparse + HTR
sparseFigure 4: Evaluating adaptation to train tasks progressively during meta-training. Y-axis measures
average sparse return during adaptation throughout meta-training (shaded std dev), though the oracle
is still trained using dense reward. Conventional meta-RL methods struggle to learn using sparse
reward. Hindsight Task Relabeling (HTR) is comparable to dense reward meta-training performance.
2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0
Episodes of adaptation012345Average returnPoint Robot
shaped (oracle)
sparse + HTR
sparse
2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0
Episodes of adaptation450
400
350
300
250
Average returnWheeled Locomotion
shaped (oracle)
sparse + HTR
sparse
2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0
Episodes of adaptation200300400500600700Average returnAnt Locomotion
shaped (oracle)
sparse + HTR
sparse
Figure 5: Evaluating adaptation to test tasks after meta-training. Y-axis measures average (sparse)
return during adaptation using context collected online, using sparse reward only. Adaptation
strategies learned with Hindsight Task Relabeling (HTR) generalize to held-out tasks as well as the
oracle which is learned using shaped reward functions. Without HTR or access to a shaped reward
during meta-training, the agent is unable to learn a reasonable strategy.
that random exploration is far less likely to encounter sparse reward. Refer to the supplement for
further details on the experimental setup.
•Point Robot . A point robot must navigate a 2D plane to different goals located along the
perimeter of a half-semicircle. The state includes the robot’s coordinates but does not include
the goal, therefore the agent must explore the environment to discover the goal. In the dense
reward variant of the environment, the reward is the negative L2distance to the goal. In the
sparse reward variant, reward is given only when the agent is within a short distance to the
goal. An example optimal exploration strategy in the sparse reward variant is to efﬁciently
traverse the perimeter of the semicircle until the goal is found.
•Wheeled Locomotion . To test if our approach generalizes to more complex state and
action spaces, we use several continuous control environments. In the wheeled locomotion
environment, the agent must navigate to the goal distribution by controlling two wheels
independent to turn. Similar to the point robot, the dense reward function is the negative
distance to the goal, while the sparse reward function only provides reward signal when the
agent is nearby the goal.
•Ant (Quadruped) Locomotion . This environment requires controlling a quadruped ant with a
high-dimensional state and action space. The task distribution and reward functions are the
same as the point robot and wheeled locomotion environment, however, exploration requires
coordinating movement with four legs to navigate to speciﬁc locations.
5.2 HTR enables meta-training using only sparse reward
To answer question (i), we compare our hindsight task relabeling approach (using PEARL as the
base meta-RL algorithm) to standard PEARL, as well as an oracle PEARL which uses the dense
reward function during training. Our oracle baseline is equivalent to the standard meta-RL setup in
the ‘sparse reward’ experiments in Gupta et al. (2018b) and Rakelly et al. (2019), despite it never
training on the sparse reward function. Note that the term ‘oracle’ is somewhat of a misnomer, since
training on a proxy dense reward function designed to aid the learning of an RL agent (to be evaluated
8
Figure 6: Visualizing exploration behavior learned during meta-training using 300 pre-adaptation
trajectories (i.e., sampled from the latent task prior). In the sparse reward setting, without HTR
(middle row) the agent is unable to learn a meaningful exploration strategy and appears to explore
randomly near the origin. With HTR (bottom row), the agent learns to explore near the true task
distribution (grey circles), similar to an agent trained with a shaped dense reward function (top row).
on the sparse reward function) is inherently optimizing a different objective. Optimizing an RL agent
for a proxy objective has been shown to lead to inadvertently bad performance on the true objective
(Clark & Amodei, 2016), and therefore it is often better to directly optimize for the true objective
(e.g., the original sparse reward function) when possible.
As discussed in prior work, meta-learning on sparse reward environments proves extremely challeng-
ing: in Figures 4 and 6, we see that PEARL is unable to make learn a reasonable policy just using
sparse rewards in the same amount of time it takes the oracle (PEARL trained on dense reward) and
HTR (PEARL trained on sparse reward using Hindsight Task Relabelling) to converge. In all of our
tested environments, HTR is not only comparable to using shaped reward during meta-training, but
it also can learn adaptation strategies that perform as well as the oracle during meta-testing on the
sparse reward environment. In Figure 5, we see that the adaptation strategies learned with Hindsight
Task Relabeling are similarly effective to those learned by the oracle, despite the oracle having access
to a dense reward function during meta-training.
Given an indeﬁnite amount of meta-training time, or an easier environment conﬁguration (e.g., a
shorter goal distance), PEARL should be able to learn a similarly optimal strategy to the oracle and
HTR only using sparse reward. If PEARL isable to successfully learn only using sparse rewards,
HTR can be used to improve sample efﬁciency during meta-training. In the case that the amount
of time or computational resources needed to train purely on sparse rewards is intractable, HTR
is extremely effective at learning adaptation strategies comparable to using a hand-designed dense
reward function.
Additionally, as mentioned in Section 2, in many sparse reward settings designing a dense reward
function to aid training is infeasible or undesirable, in which case HTR is a strong alternative to
reward engineering. Because HTR does not train on a proxy reward, there is no opportunity for a
mismatch between reward functions at meta-training and meta-testing: HTR optimizes for the true
reward function on a superset of tasks Thindsight[Ttrainp(T), whereas using a shaped reward
optimizes for a proxy reward function on the true set of training tasks Ttrainp(T).
9
0 50 100 150 200 250 300
Meta-training iterations0.00.51.01.52.02.5Average returnshaped (oracle)
sparse + HTR SER
sparse + HTR EC
sparseFigure 7: Comparing HTR with
SER vs EC on Point Robot.
0 50 100 150 200 250 300
Meta-training iterations0.00.51.01.52.02.5Average returnK = 0.1
K = 0.3
K = 0.5
K = 0.8Figure 8: Average return when
varyingKon Point Robot.
0 50 100 150 200 250 300
Meta-training iterations0.000.250.500.751.001.251.501.752.00Relabelled
task distanceFigure 9: Average task distance
when varying Kon Point Robot.
5.3 Varying key hyperparameters
0 200 400 600 800 1000
Meta-training iterations025050075010001250150017502000Num. transitions
with rewardGround truth
Relabelled
Figure 10: Relative reward sig-
nal from hindsight vs ground
truth tasks using Point Robot.In our experiments we found a relatively low relabeling probability
(e.g.,K= 0:1and0:3) was often most effective for HTR (see
Figure 8).K= 0:1means that relabelled data from a hindsight
task is roughly as likely to be used for training as data from any
of the 100 tasks in Ttrain :K= 0:1corresponds to using far less
relabelled data for training than in conventional HER: in contrast,
Andrychowicz et al. (2017) relabel over 80% of sampled transi-
tions. We hypothesize that a low Kworks best for HTR since it
provides enough reward signal to bootstrap learning, without bias-
ing the task distribution too far from the ground truth distribution;
in Figure 9, we can see that both K= 0:5and0:8converge on
a mean relabelled task distance of 0:5, whereasK= 0:1and0:3
are closer to the ground truth task distance of 2:0.
In both HER and HTR, the relabeling probability is an important hyperparameter that should be
tuned for optimal performance. In the original HER paper, Andrychowicz et al. (2017) reported that a
relabelling probability of 80% or 90% worked best on their three goal-conditioned RL environments.
For both algorithms, there exists a range of relabelling probabilities (alternatively a range of ratios of
original-to-relabelled data) where the relabeling algorithm works well, and a range of values where
the algorithm performs poorly (the HER paper did not include results for under 50% relabelled data,
which may include more negative results).
In Figure 9, we see that the average distance reached by the policy during meta-training gradually in-
creases over time, increasing the average relabelled task distance. Hindsight tasks used for relabelling
initially correspond to easily achievable tasks, and shift towards the ground truth task distribution.
Initially, the only reward signal available for training comes from the relabelled transitions, however,
as training progresses the agent is able to recover true reward signal from the environment (see Figure
10) and the hindsight tasks are more likely to resemble the ground truth tasks.
As described in Section 4, we found that EC generally performed similarly to SER (see Figure 7),
despite being signiﬁcantly more complex to implement and requiring additional tuned hyperparam-
eters. We expect that a more signiﬁcant performance difference between the two approaches may
become apparent on different sparse reward meta-RL tasks, or through experimenting with more
general episode clustering approaches than the state space discretization used in this work.
6 Conclusion
In this paper, we propose a novel approach for meta-reinforcement learning in sparse reward environ-
ments that can be incorporated into any off-policy meta-RL algorithm. The sparse reward meta-RL
setting is extremely challenging, and existing meta-RL algorithms that learn adaptation strategies for
sparse reward environments often require dense or shaped reward functions during meta-training.
Our approach, Hindsight Task Relabeling (HTR), enables learning adaptation strategies in challenging
sparse reward environments without engineering proxy reward functions. HTR relabels data collected
on ground truth meta-training tasks as data for achievable hindsight tasks, which enables meta-training
on the original sparse reward objective. Not only does HTR allow for learning adaptation strategies in
challenging sparse reward environments without reward engineering, but it also generates adaptation
strategies comparable to prior approaches that use shaped reward functions.
10
Acknowledgements
We thank Katelyn Gao for providing valuable feedback and insights on our work. This project is in part based
upon work supported by the National Science Foundation under NSF CISE Expeditions Award CCF-1730628,
Berkeley Deep Drive, and gifts from Amazon Web Services, Ant Group, Ericsson, Facebook, Futurewei, Google,
Intel, Microsoft, Scotiabank, and VMware. Any opinions, ﬁndings, and conclusions or recommendations
expressed in this paper are those of the authors and do not necessarily reﬂect the views of the National Science
Foundation, Berkeley Deep Drive, and other sponsors.
References
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P.,
and Zaremba, W. Hindsight experience replay. In Neural Information Processing Systems (NeurIPS) , pp.
5048–5058, 2017.
Bengio, Y ., Bengio, S., and Cloutier, J. Learning a synaptic learning rule . Citeseer, 1990.
Clark, J. and Amodei, D. Faulty reward functions in the wild. https://blog.openai.com/faulty-reward-functions ,
2016.
Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J. Quantifying generalization in reinforcement
learning. In International Conference on Machine Learning (ICML) , 2019.
Duan, Y ., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., and Abbeel, P. RL2: Fast reinforcement learning
via slow reinforcement learning. arXiv:1611.02779 , 2016.
Eysenbach, B., Geng, X., Levine, S., and Salakhutdinov, R. Rewriting history with inverse rl: Hindsight
inference for policy improvement. arXiv preprint arXiv:2002.11089 , 2020.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In
International Conference on Machine Learning (ICML) , 2017.
Florensa, C., Held, D., Geng, X., and Abbeel, P. Automatic goal generation for reinforcement learning agents.
InInternational Conference on Machine Learning (ICML) , 2018.
Gupta, A., Eysenbach, B., Finn, C., and Levine, S. Unsupervised meta-learning for reinforcement learning.
arXiv preprint arXiv:1806.04640 , 2018a.
Gupta, A., Mendonca, R., Liu, Y ., Abbeel, P., and Levine, S. Meta-reinforcement learning of structured
exploration strategies. In Neural Information Processing Systems (NeurIPS) , pp. 5302–5311, 2018b.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep
reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML) ,
2018.
Houthooft, R., Chen, Y ., Isola, P., Stadie, B., Wolski, F., Ho, O. J., and Abbeel, P. Evolved policy gradients. In
Neural Information Processing Systems (NeurIPS) , pp. 5400–5409, 2018.
Jabri, A., Hsu, K., Gupta, A., Eysenbach, B., Levine, S., and Finn, C. Unsupervised curricula for visual
meta-reinforcement learning. In Neural Information Processing Systems (NeurIPS) , pp. 10519–10530, 2019.
Kaelbling, L. P. Learning to achieve goals. In International Joint Conferences on Artiﬁcial Intelligence (IJCAI) ,
pp. 1094–1099. Citeseer, 1993.
Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. Journal of
Machine Learning Research (JMLR) , 17(1):1334–1373, 2016.
Levy, A., Platt, R., and Saenko, K. Hierarchical actor-critic. arXiv preprint arXiv:1712.00948 , 2017.
Li, A. C., Pinto, L., and Abbeel, P. Generalized hindsight for reinforcement learning. arXiv preprint
arXiv:2002.11708 , 2020.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y ., Silver, D., and Wierstra, D. Continuous
control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.
Mishra, N., Rohaninejad, M., Chen, X. P., and Abbeel, P. A simple neural attentive meta-learner. In International
Conference on Learning Representations (ICLR) , 2018.
11
Mnih, V ., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M.,
Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., et al. Human-level control through deep
reinforcement learning. Nature , 518(7540), 2015.
Nair, A. V ., Pong, V ., Dalal, M., Bahl, S., Lin, S., and Levine, S. Visual reinforcement learning with imagined
goals. In Neural Information Processing Systems (NeurIPS) , pp. 9191–9200, 2018.
Ng, A. Y ., Harada, D., and Russell, S. Policy invariance under reward transformations: Theory and application
to reward shaping. In International Conference on Machine Learning (ICML) , volume 99, pp. 278–287, 1999.
Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V ., and Song, D. Assessing generalization in deep
reinforcement learning. arXiv:1810.12282 , 2018.
Rakelly, K., Zhou, A., Quillen, D., Finn, C., and Levine, S. Efﬁcient off-policy meta-reinforcement learning via
probabilistic context variables. In International Conference on Machine Learning (ICML) , 2019.
Rauber, P., Ummadisingu, A., Mutz, F., and Schmidhuber, J. Hindsight policy gradients. In International
Conference on Learning Representations (ICLR) , 2019.
Rothfuss, J., Lee, D., Clavera, I., Asfour, T., and Abbeel, P. Promp: Proximal meta-policy search. In International
Conference on Learning Representations (ICLR) , 2019.
Schmidhuber, J. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-
... hook . PhD thesis, Technische Universität München, 1987.
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D.,
Graepel, T., et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm.
arXiv preprint arXiv:1712.01815 , 2017.
Stadie, B. C., Yang, G., Houthooft, R., Chen, X., Duan, Y ., Wu, Y ., Abbeel, P., and Sutskever, I. Some
considerations on learning to explore via meta-reinforcement learning. arXiv preprint arXiv:1803.01118 ,
2018.
Sung, F., Zhang, L., Xiang, T., Hospedales, T., and Yang, Y . Learning to learn: Meta-critic networks for sample
efﬁcient learning. arXiv:1706.09529 , 2017.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction . MIT Press, 2nd edition, 2017.
Thrun, S. and Pratt, L. Learning to learn: Introduction and overview. In Learning to learn , pp. 3–17. Springer,
1998.
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and
Botvinick, M. Learning to reinforcement learn. arXiv:1611.05763 , 2016.
Xu, T., Liu, Q., Zhao, L., and Peng, J. Learning to explore via meta-policy gradient. In International Conference
on Machine Learning (ICML) , pp. 5463–5472, 2018a.
Xu, Z., van Hasselt, H. P., and Silver, D. Meta-gradient reinforcement learning. In Neural Information Processing
Systems (NeurIPS) , pp. 2396–2407, 2018b.
Zhang, A., Ballas, N., and Pineau, J. A dissection of overﬁtting and generalization in continuous reinforcement
learning. arXiv:1806.07937 , 2018.
Zintgraf, L. M., Shiarlis, K., Kurin, V ., Hofmann, K., and Whiteson, S. Fast context adaptation via meta-learning.
arXiv preprint arXiv:1810.03642 , 2018.
12

Charles Packer
Pieter Abbeel
Joseph E
Gonzalez