Quantifying Differences in Reward Functions.

Published as a conference paper at ICLR 2021
QUANTIFYING DIFFERENCES IN REWARD FUNCTIONS
Adam Gleave1,2Michael Dennis1Shane Legg2Stuart Russell1Jan Leike3
1UC Berkeley2DeepMind3OpenAI
gleave@berkeley.edu
ABSTRACT
For many tasks, the reward function is inaccessible to introspection or too complex
to be speciﬁed procedurally, and must instead be learned from user data. Prior
work has evaluated learned reward functions by evaluating policies optimized for
the learned reward. However, this method cannot distinguish between the learned
reward function failing to reﬂect user preferences and the policy optimization pro-
cess failing to optimize the learned reward. Moreover, this method can only tell us
about behavior in the evaluation environment, but the reward may incentivize very
different behavior in even a slightly different deployment environment. To address
these problems, we introduce the Equivalent-Policy Invariant Comparison (EPIC)
distance to quantify the difference between two reward functions directly, without
a policy optimization step. We prove EPIC is invariant on an equivalence class of
reward functions that always induce the same optimal policy. Furthermore, we ﬁnd
EPIC can be efﬁciently approximated and is more robust than baselines to the choice
of coverage distribution. Finally, we show that EPIC distance bounds the regret of
optimal policies even under different transition dynamics, and we conﬁrm empiri-
cally that it predicts policy training success. Our source code is available at https:
//github.com/HumanCompatibleAI/evaluating-rewards .
1 I NTRODUCTION
Reinforcement learning (RL) has reached or surpassed human performance in many domains with
clearly deﬁned reward functions, such as games [ 20;15;23] and narrowly scoped robotic manipulation
tasks [ 16]. Unfortunately, the reward functions for most real-world tasks are difﬁcult or impossible
to specify procedurally. Even a task as simple as peg insertion from pixels has a non-trivial reward
function that must usually be learned [ 22, IV .A]. Tasks involving human interaction can have far
more complex reward functions that users may not even be able to introspect on. These challenges
have inspired work on learning a reward function, whether from demonstrations [ 13;17;26;8;3],
preferences [1; 25; 6; 18; 27] or both [10; 4].
Prior work has usually evaluated the learned reward function ^Rusing the “rollout method”: training a
policy^Rto optimize ^Rand then examining rollouts from ^R. Unfortunately, using RL to compute
^Ris often computationally expensive. Furthermore, the method produces false negatives when the
reward ^Rmatches user preferences but the RL algorithm fails to optimize with respect to ^R.
The rollout method also produces false positives . Of the many reward functions that induce the
desired rollout in a given environment, only a small subset align with the user’s preferences. For
example, suppose the agent can reach states fA;B;Cg. If the user prefers A>B >C , but the agent
instead learns A>C >B , the agent will still go to the correct state A. However, if the initial state
distribution or transition dynamics change, misaligned rewards may induce undesirable policies. For
example, ifAis no longer reachable at deployment, the previously reliable agent would misbehave
by going to the least-favoured state C.
We propose instead to evaluate learned rewards via their distance from other reward functions, and
summarize our desiderata for reward function distances in Table 1. For benchmarks, it is usually
possible to directly compare a learned reward ^Rto the true reward function R. Alternatively,
benchmark creators can train a “proxy” reward function from a large human data set. This proxy can
then be used as a stand-in for the true reward Rwhen evaluating algorithms trained on a different or
smaller data set.
Work partially conducted while at DeepMind.
1arXiv:2006.13900v3  [cs.LG]  17 Mar 2021
Published as a conference paper at ICLR 2021
Table 1: Summary of the desiderata satisﬁed by each reward function distance. Key – the distance is:
apseudometric (section 3); invariant to potential shaping [ 14] and positive rescaling (section 3); a
computationally efﬁcient approximation achieving low error (section 6.1); robust to the choice of
coverage distribution (section 6.2); and predictive of the similarity of the trained policies (section 6.3).
Distance Pseudometric Invariant Efﬁcient Robust Predictive
EPIC 3 3 3 3 3
NPEC 7 3 7 7 3
ERC 3 7 3 7 3
Comparison with a ground-truth reward function is rarely possible outside of benchmarks. However,
even in this challenging case, comparisons can at least be used to cluster reward models trained
using different techniques or data. Larger clusters are more likely to be correct, since multiple
methods arrived at a similar result. Moreover, our regret bound (Theorem 4.9) suggests we could use
interpretability methods [12] on one model and get some guarantees for models in the same cluster.
We introduce the Equivalent-Policy Invariant Comparison (EPIC) distance that meets all the criteria
in Table 1. We believe EPIC is the ﬁrst method to quantitatively evaluate reward functions without
training a policy. EPIC (section 4) canonicalizes the reward functions’ potential-based shaping [ 14],
then takes the correlation between the canonical rewards over a coverage distribution Dof transitions.
We also introduce baselines NPEC andERC (section 5) which partially satisfy the criteria.
EPIC works best when Dhas support on all realistic transitions. We achieve this in our experiments
by using uninformative priors, such as rollouts of a policy taking random actions. Moreover, we
ﬁnd EPIC is robust to the exact choice of distribution D, producing similar results across a range of
distributions, whereas ERC and especially NPEC are highly sensitive to the choice of D(section 6.2).
Moreover, low EPIC distance between a learned reward ^Rand the true reward Rpredicts low regret.
That is, the policies ^RandRoptimized for ^RandRobtain similar returns under R. Theorem 4.9
bounds the regret even in unseen environments; by contrast, the rollout method can only determine
regret in the evaluation environment. We also conﬁrm this result empirically (section 6.3).
2 R ELATED WORK
There exist a variety of methods to learn reward functions. Inverse reinforcement learning (IRL) [ 13] is
a common approach that works by inferring a reward function from demonstrations. The IRL problem
is inherently underconstrained: many different reward functions lead to the same demonstrations.
Bayesian IRL [ 17] handles this ambiguity by inferring a posterior over reward functions. By
contrast, Maximum Entropy IRL [ 26] selects the highest entropy reward function consistent with the
demonstrations; this method has scaled to high-dimensional environments [7; 8].
An alternative approach is to learn from preference comparisons between two trajectories [ 1;25;6;18].
T-REX [4] is a hybrid approach, learning from a ranked set of demonstrations. More directly, Cabi
et al. [5] learn from “sketches” of cumulative reward over an episode.
To the best of our knowledge, there is no prior work that focuses on evaluating reward functions
directly. The most closely related work is Ng et al. [14], identifying reward transformations guaranteed
to not change the optimal policy. However, a variety of ad-hoc methods have been developed to
evaluate reward functions. The rollout method – evaluating rollouts of a policy trained on the learned
reward – is evident in the earliest work on IRL [ 13]. Fu et al. [8]reﬁned the rollout method by testing
on a transfer environment, inspiring our experiment in section 6.3. Recent work has compared reward
functions by scatterplotting returns [10; 4], inspiring our ERC baseline (section 5.1).
3 B ACKGROUND
This section introduces material needed for the distances deﬁned in subsequent sections. We start by
introducing the Markov Decision Process (MDP) formalism, then describe when reward functions
induce the same optimal policies in an MDP, and ﬁnally deﬁne the notion of a distance metric .
2
Published as a conference paper at ICLR 2021
Deﬁnition 3.1. AMarkov Decision Process (MDP) M= (S;A;
;d 0;T;R)consists of a set of
statesSand a set of actions A; a discount factor
2[0;1]; an initial state distribution d0(s); a
transition distribution T(s0js;a)specifying the probability of transitioning to s0fromsafter taking
actiona; and a reward function R(s;a;s0)specifying the reward upon taking action ain statesand
transitioning to state s0.
A trajectory = (s0;a0;s1;a1;)consists of a sequence of states si2Sand actionsai2A. The
return on a trajectory is deﬁned as the sum of discounted rewards, g(;R) =Pjj
t=0
tR(st;at;st+1),
where the length of the trajectory jjmay be inﬁnite.
In the following, we assume a discounted (
 <1) inﬁnite-horizon MDP. The results can be generalized
to undiscounted (
= 1) MDPs subject to regularity conditions needed for convergence.
Astochastic policy (ajs)assigns probabilities to taking action a2A in states2S. The objective
of an MDP is to ﬁnd a policy that maximizes the expected return G() =E()[g(;R)], where
()is a trajectory generated by sampling the initial state s0fromd0, each action atfrom the policy
(atjst)and successor states st+1from the transition distribution T(st+1jst;at). An MDPM
has a set of optimal policies (M)that maximize the expected return, (M) = arg maxG().
In this paper, we consider the case where we only have access to an MDP\R, M= (S;A;
;d 0;T).
The unknown reward function Rmust be learned from human data. Typically, only the state space
S, action spaceAand discount factor
are known exactly, with the initial state distribution d0
and transition dynamics Tonly observable from interacting with the environment M. Below, we
describe an equivalence class whose members are guaranteed to have the same optimal policy set in
anyMDP\RMwith ﬁxedS,Aand
(allowing the unknown Tandd0to take arbitrary values).
Deﬁnition 3.2. Let
2[0;1]be the discount factor, and  :S!Ra real-valued function. Then
R(s;a;s0) =
(s0)(s)is apotential shaping reward, with potential [14].
Deﬁnition 3.3 (Reward Equivalence) .We deﬁne two bounded reward functions RAandRBto be
equivalent ,RARB, for a ﬁxed (S;A;
)if and only if there exists a constant >0and a bounded
potential function  :S!Rsuch that for all s;s02S anda2A:
RB(s;a;s0) =RA(s;a;s0) +
(s0)(s): (1)
Proposition 3.4. The binary relation is an equivalence relation. Let RA;RB;RC:SAS! R
be bounded reward functions. Then is reﬂexive, RARA; symmetric, RARBimplies
RBRA; and transitive, (RARB)^(RBRC)impliesRARC.
Proof. See section A.3.1 in supplementary material.
The expected return of potential shaping
(s0)(s)on a trajectory segment (s0;;sT)is

T(sT)(s0). The ﬁrst term
T(sT)!0asT!1 , while the second term (s0)only
depends on the initial state, and so potential shaping does not change the set of optimal policies.
Moreover, any additive transformation that is not potential shaping will, for some reward Rand
transition distribution T, produce a set of optimal policies that is disjoint from the original [14].
The set of optimal policies is invariant to constant shifts c2Rin the reward, however this can already
be obtained by shifting byc

1.*Scaling a reward function by a positive factor >0scales the
expected return of all trajectories by , also leaving the set of optimal policies unchanged.
IfRARBfor some ﬁxed (S;A;
), then for any MDP\R M= (S;A;
;d 0;T)we have
((M;RA)) =((M;RB)), where (M;R)denotes the MDP speciﬁed by Mwith re-
ward function R. In other words, RAandRBinduce the same optimal policies for all initial state
distributions d0and transition dynamics T.
Deﬁnition 3.5. LetXbe a set and d:XX![0;1)a function.dis apremetric ifd(x;x) = 0
for allx2X.dis apseudometric if, furthermore, it is symmetric, d(x;y) =d(y;x)for allx;y2X;
and satisﬁes the triangle inequality, d(x;z)d(x;y) +d(y;z)for allx;y;z2X.dis ametric if,
furthermore, for all x;y2X,d(x;y) = 0 =)x=y.
We wish for d(RA;RB) = 0 whenever the rewards are equivalent, RARB, even if they are not
identical,RA6=RB. This is forbidden in a metric but permitted in a pseudometric, while retaining
*Note constant shifts in the reward of an undiscounted MDP would cause the value function to diverge.
Fortunately, the shaping
(s0)(s)is unchanged by constant shifts to when
= 1.
3
Published as a conference paper at ICLR 2021
other guarantees such as symmetry and triangle inequality that a metric provides. Accordingly, a
pseudometric is usually the best choice for a distance dover reward functions.
4 C OMPARING REWARD FUNCTIONS WITH EPIC
In this section we introduce the Equivalent-Policy Invariant Comparison (EPIC) pseudometric.
This novel distance canonicalizes the reward functions’ potential-based shaping, then compares
the canonical representatives using Pearson correlation, which is invariant to scale. Together, this
construction makes EPIC invariant on reward equivalence classes. See section A.3.2 for proofs.
We deﬁne the canonically shaped reward CDS;DA(R)as an expectation over some arbitrary distribu-
tionsDSandDAover statesSand actionsArespectively. This construction means that CDS;DA(R)
does not depend on the MDP’s initial state distribution d0or transition dynamics T. In particular,
we may evaluate Ron transitions that are impossible in the training environment, since these may
become possible in a deployment environment with a different d0orT.
Deﬁnition 4.1 (Canonically Shaped Reward) .LetR:SAS! Rbe a reward function. Given
distributionsDS2(S)andDA2(A)over states and actions, let SandS0be random variables
independently sampled from DSandAsampled fromDA. We deﬁne the canonically shaped Rto be:
CDS;DA(R) (s;a;s0) =R(s;a;s0) +E[
R(s0;A;S0)R(s;A;S0)
R(S;A;S0)]: (2)
Informally, if R0is shaped by potential , then increasing (s)decreasesR0(s;a;s0)but in-
creases E[R0(s;A;S0)], canceling. Similarly, increasing (s0)increasesR0(s;a;s0)but decreases
E[
R0(s0;A;S0)]. Finally, E[
R(S;A;S0)]centers the reward, canceling constant shift.
Proposition 4.2 (The Canonically Shaped Reward is Invariant to Shaping) .LetR:SAS! R
be a reward function and  :S!Ra potential function. Let
2[0;1]be a discount rate, and
DS2(S)andDA2(A)be distributions over states and actions. Let R0denoteRshaped by
:R0(s;a;s0) =R(s;a;s0) +
(s0)(s). Then the canonically shaped R0andRare equal:
CDS;DA(R0) =CDS;DA(R).
Proposition 4.2 holds for arbitrary distributions DSandDA. However, in the following Proposition
we show that the potential shaping applied by the canonicalization CDS;DA(R)is more inﬂuenced by
perturbations to Rof transitions (s;a;s0)with high joint probability. This suggests choosing DSand
DAto have broad support, making CDS;DA(R)more robust to perturbations of any given transition.
Proposition 4.3. LetSandAbe ﬁnite, withjSj 2. LetDS2(S)andDA2(A). Let
R;  :SAS! Rbe reward functions, with (s;a;s0) =I[(s;a;s0) = (x;u;x0)],2R,
x;x02S andu2A. Let DS;DA(R)(s;a;s0) =CDS;DA(R) (s;a;s0)R(s;a;s0). Then:
kDS;DA(R+)DS;DA(R)k1=(1 +
DS(x))DA(u)DS(x0): (3)
We have canonicalized potential shaping; next, we compare the rewards in a scale-invariant manner.
Deﬁnition 4.4. ThePearson distance between random variables XandYis deﬁned by the expression
D(X;Y ) =p
1(X;Y )=p
2, where(X;Y )is the Pearson correlation between XandY.
Lemma 4.5. The Pearson distance Dis a pseudometric. Moreover, let a;b2(0;1),c;d2Rand
X;Y be random variables. Then it follows that 0D(aX+c;bY +d) =D(X;Y )1.
We can now deﬁne EPIC in terms of the Pearson distance between canonically shaped rewards.
Deﬁnition 4.6 (Equivalent-Policy Invariant Comparison (EPIC) pseudometric) .LetDbe some
coverage distribution over transitions sa!s0. LetS;A;S0be random variables jointly sampled from
D. LetDSandDAbe some distributions over states SandArespectively. The Equivalent-Policy
Invariant Comparison (EPIC) distance between reward functions RAandRBis:
DEPIC(RA;RB) =D(CDS;DA(RA) (S;A;S0);CDS;DA(RB) (S;A;S0)): (4)
Theorem 4.7. The Equivalent-Policy Invariant Comparison distance is a pseudometric.
Since EPIC is a pseudometric, it satisﬁes the triangle inequality. To see why this is useful, consider an
environment with an expensive to evaluate ground-truth reward R. Directly comparing many learned
4
Published as a conference paper at ICLR 2021
rewards ^RtoRmight be prohibitively expensive. We can instead pay a one-off cost: query Ra ﬁnite
number of times and infer a proxy reward RPwithDEPIC(R;RP). The triangle inequality
allows us to evaluate ^Rvia comparison to RP, sinceDEPIC(^R;R)DEPIC(^R;RP) +. This is
particularly useful for benchmarks, which can be expensive to build but should be cheap to use.
Theorem 4.8. LetRA,R0
A,RB;R0
B:SAS! Rbe reward functions such that R0
ARA
andR0
BRB. Then 0DEPIC(R0
A;R0
B) =DEPIC(RA;RB)1.
The following is our main theoretical result, showing that DEPIC(RA;RB)distance gives an upper
bound on the difference in returns under eitherRAorRBbetween optimal policies
RAand
RB. In
other words, EPIC bounds the regret under RAof using
RBinstead of
RA. Moreover, by symmetry
DEPIC(RA;RB)also bounds the regret under RBof using
RAinstead of
RB.
Theorem 4.9. LetMbe a
-discounted MDP\R with ﬁnite state and action spaces SandA.
LetRA;RB:SAS ! Rbe rewards, and
A;
Bbe respective optimal policies. Let
D(t;st;at;st+1)denote the distribution over transitions SAS induced by policy at timet,
andD(s;a;s0)be the coverage distribution used to compute DEPIC . Suppose there exists K > 0such
thatKD(st;at;st+1)D(t;st;at;st+1)for all times t2N, triples (st;at;st+1)2SAS
and policies 2f
A;
Bg. Then the regret under RAfrom executing
Binstead of
Ais at most
GRA(
A)GRA(
B)16KkRAk2(1
)1DEPIC(RA;RB);
whereGR()is the return of policy under reward R.
We generalize the regret bound to continuous spaces in theorem A.16 via a Lipschitz assump-
tion, with Wasserstein distance replacing K. Importantly, the returns of
Aand
Bconverge as
DEPIC(RA;RB)!0in both cases, no matter which reward function you evaluate on.
The key assumption is that the coverage distribution Dhas adequate support for transitions occurring
in rollouts of
Aand
B. The bound is tightest when Dis similar toD
AandD
B. However,
computing
Aand
Bis often intractable. The MDP Mmay be unknown, such as when making
predictions about an unseen deployment environment. Even when Mis known, RL is computationally
expensive and may fail to converge in non-trivial environments.
In ﬁnite cases, a uniform Dsatisﬁes the requirements with KjSj2jAj. In general, it is best to
chooseDto have broad coverage over plausible transitions. Broad coverage ensures adequate support
forD
AandD
B. But excluding transitions that are unlikely or impossible to occur leads to tighter
regret bounds due to a smaller K(ﬁnite case) or Wasserstein distance (continuous case).
While EPIC upper bounds policy regret, it does not lower bound it. In fact, no reward distance can
lower bound regret in arbitrary environments. For example, suppose the deployment environment
transitions to a randomly chosen state independent of the action taken. In this case, all policies obtain
the same expected return, so the policy regret is always zero, regardless of the reward functions.
To demonstrate EPIC’s properties, we compare the gridworld reward functions from Figure 1,
reporting the distances between all reward pairs in Figure A.2. Dense is a rescaled and shaped
version of Sparse , despite looking dissimilar at ﬁrst glance, so DEPIC (Sparse;Dense ) = 0 . By
contrast,DEPIC (Path;Cliff ) = 0:27. Indeterministic gridworlds, Path andCliff have the
same optimal policy, so the rollout method could wrongly conclude they are equivalent. But in fact
the rewards are fundamentally different: when there is a signiﬁcant risk of “slipping” in the wrong
direction the optimal policy for Cliff walks along the top instead of the middle row, incurring a 1
penalty to avoid the risk of falling into the 4“cliff”.
For this example, we used state and action distributions DSandDAuniform overSandA, and
coverage distribution Duniform over state-action pairs (s;a), withs0deterministically computed. It
is important these distributions have adequate support. As an extreme example, if DSandDhave no
support for a particular state then the reward of that state has no effect on the distance. We can compute
EPIC exactly in a tabular setting, but in general use a sample-based approximation (section A.1.1).
5 B ASELINE APPROACHES FOR COMPARING REWARD FUNCTIONS
Given the lack of established methods, we develop two alternatives as baselines: Episode Return
Correlation (ERC) and Nearest Point in Equivalence Class (NPEC).
5
Published as a conference paper at ICLR 2021
0 1 2012
000
000
100
000
100
00
00
10
000
000
00
00
00Sparse Reward
0 1 2-1-1-1
-1-1-1
3-1-1
-4-4-4
0-4-4
-4-4
-4-4
0-4
222
222
22
22
22Dense Reward
0 1 2-100
-10-1
40-1
-10-1
40-1
-10
-10
40
-100
-10-1
00
0-1
0-1Path Reward
0 1 2-400
-40-1
40-1
-40-1
40-1
-40
-40
40
-400
-40-1
00
0-1
0-1Cliff Reward
-4-2024
Figure 1: Heatmaps of four reward functions for a 33gridworld. Sparse andDense look
different but are actually equivalent with DEPIC (Sparse;Dense ) = 0 . By contrast, the optimal
policies for Path andCliff are the same if the gridworld is deterministic but different if it is
“slippery”. EPIC recognizes this difference with DEPIC (Path;Cliff ) = 0:27.Key: Reward
R(s;s0)for moving from stos0is given by the triangular wedge in cellsthat is adjacent to cell
s0.R(s;s)is given by the central circle in cells. Optimal action(s) (deterministic, inﬁnite horizon,
discount
= 0:99) have bold labels. See Figure A.2 for the distances between all reward pairs.
5.1 E PISODE RETURN CORRELATION (ERC)
The goal of an MDP is to maximize expected episode return, so it is natural to compare reward
functions by the returns they induce. If the return of a reward function RAis a positive afﬁne
transformation of another reward RB, thenRAandRBhave the same set of optimal policies. This
suggests using Pearson distance, which is invariant to positive afﬁne transformations.
Deﬁnition 5.1 (Episode Return Correlation (ERC) pseudometric) .LetDbe some distribution over
trajectories. Let Ebe a random variable sampled from D. The Episode Return Correlation distance
between reward functions RAandRBis the Pearson distance between their episode returns on D,
DERC(RA;RB) =D(g(E;RA);g(E;RB)).
Prior work has produced scatter plots of the return of RAagainstRBover episodes [ 4, Figure 3]
and ﬁxed-length segments [ 10, section D]. ERC is the Pearson distance of such plots, so is a natural
baseline. We approximate ERC by the correlation of episode returns on a ﬁnite collection of rollouts.
ERC is invariant to shaping when the initial state s0and terminal state sTare ﬁxed. Let Rbe a
reward function and a potential function, and deﬁne the shaped reward R0(s;a;s0) =R(s;a;s0) +

(s0)(s). The return under the shaped reward on a trajectory = (s0;a0;;sT)is
g(;R0) =g(;R) +
T(sT)(s0). Sinces0andsTare ﬁxed,
T(sT)(s0)is constant.
It follows that ERC is invariant to shaping, as Pearson distance is invariant to constant shifts. In fact,
for inﬁnite-horizon discounted MDPs only s0needs to be ﬁxed, since
T(sT)!0asT!1 .
However, if the initial state s0is stochastic, then the ERC distance can take on arbitrary values
under shaping. Let RAandRBbe two arbitrary reward functions. Suppose that there are at least
two distinct initial states, sXandsY, with non-zero measure in D. Choose potential (s) = 0
everywhere except (sX) = (sY) =c, and letR0
AandR0
BdenoteRAandRBshaped by . As
c!1 , the correlation (g(E;R0
A);g(E;R0
B))!1. This is since the relative difference tends to
zero, even though g(E;R0
A)andg(E;R0
B)continue to have the same absolute difference as cvaries.
Consequently, the ERC pseudometric DERC(R0
A;R0
B)!0asc!1 . By an analogous argument,
setting (sX) =cand(sY) =cgivesDERC(R0
A;R0
B)!1asc!1 .
5.2 N EAREST POINT IN EQUIVALENCE CLASS (NPEC)
NPEC takes the minimum Lpdistance between equivalence classes. See section A.3.3 for proofs.
Deﬁnition 5.2 (Lpdistance) .LetDbe a coverage distribution over transitions sa!s0and letp1
be a power. The Lpdistance between reward functions RAandRBis theLpnorm of their difference:
DLp;D(RA;RB) =
Es;a;s0D
jRA(s;a;s0)RB(s;a;s0)jp1=p:
6
Published as a conference paper at ICLR 2021
S
S
D
D
M
RBS
S
D
D
M
RA0.00 0.51 0.00 0.51 0.63
0.51 0.00 0.51 0.00 0.55
0.00 0.51 0.00 0.51 0.63
0.51 0.00 0.51 0.00 0.55
0.63 0.55 0.63 0.55 0.00
(a) EPIC
S
S
D
D
M
RB0.00 0.88 0.22 0.86 0.96
0.92 0.00 0.94 0.04 0.74
0.58 0.95 0.00 0.81 0.97
0.88 0.04 0.84 0.00 0.58
0.93 0.79 0.80 0.62 0.00 (b) NPEC
S
S
D
D
M
RB0.00 0.41 0.56 0.56 0.57
0.41 0.00 0.61 0.55 0.62
0.56 0.61 0.00 0.11 0.64
0.56 0.55 0.11 0.00 0.64
0.57 0.62 0.64 0.64 0.00
0.00.51.0
¯D(RA,RB) (c) ERC
Figure 2: Approximate distances between hand-designed reward functions in PointMass , where the
agent moves on a line trying to reach the origin. EPIC correctly assigns 0distance between equivalent
rewards such as (D
;S
)whileDNPEC (D
;S
) = 0:58andDERC(D
;S
) = 0:56. The
coverage distribution Dis sampled from rollouts of a policy unitaking actions uniformly at random.
Key: The agent has position x2R, velocity _x2Rand can accelerate x2R, producing future
positionx02R.
quadratic penalty on control x2,
no control penalty. SisSparse(x) = 1[jxj<
0:05],Dis shaped Dense(x;x0) = Sparse( x) +jx0jjxj, while MisMagnitude( x) =jxj.
TheLpdistance is affected by potential shaping and positive rescaling that do not change the optimal
policy. A natural solution is to take the distance from the nearest point in the equivalence class:
DU
NPEC (RA;RB) = infR0
ARADLp;D(R0
A;RB). Unfortunately, DU
NPEC is sensitive to RB’s scale.
It is tempting to instead take the inﬁmum over both arguments of DLp;D. However,
infR0
ARA;R0
BRBDLp;D(R0
A;RB) = 0 since all equivalence classes come arbitrarily close to
the origin in Lpspace. Instead, we ﬁx this by normalizing DU
NPEC .
Deﬁnition 5.3. NPEC is deﬁned by DNPEC (RA;RB) =DU
NPEC (RA;RB)=DU
NPEC (Zero;RB)
whenDU
NPEC (Zero;RB)6= 0, and is otherwise given by DNPEC (RA;RB) = 0 .
IfDU
NPEC (Zero;RB) = 0 thenDU
NPEC (RA;RB) = 0 sinceRAcan be scaled arbitrarily close to
Zero . Since all policies are optimal for RZero , we choose DNPEC (RA;RB) = 0 in this case.
Theorem 5.4. DNPEC is a premetric on the space of bounded reward functions. Moreover, let
RA;RA0;RB;RB0:SAS ! Rbe bounded reward functions such that RARA0and
RBRB0. Then 0DNPEC (RA0;RB0) =DNPEC (RA;RB)1.
Note thatDNPEC may not be symmetric and so is not, in general, a pseudometric: see proposition A.3.
The inﬁmum in DU
NPEC can be computed exactly in a tabular setting, but in general we must
approximate it using gradient descent. This gives an upper bound for DU
NPEC , but the quotient of
upper bounds DNPEC may be too low or too high. See section A.1.2 for details of the approximation.
6 E XPERIMENTS
We evaluate EPIC and the baselines ERC and NPEC in a variety of continuous control tasks. In
section 6.1, we compute the distance between hand-designed reward functions, ﬁnding EPIC to be the
most reliable. NPEC has substantial approximation error, and ERC sometimes erroneously assigns
high distance to equivalent rewards. Next, in section 6.2 we show EPIC is robust to the exact choice
of coverage distribution D, whereas ERC and especially NPEC are highly sensitive to the choice
ofD. Finally, in section 6.3 we ﬁnd that the distance of learned reward functions to a ground-truth
reward predicts the return obtained by policy training, even in an unseen test environment.
6.1 C OMPARING HAND -DESIGNED REWARD FUNCTIONS
We compare procedurally speciﬁed reward functions in four tasks, ﬁnding that EPIC is more reliable
than the baselines NPEC and ERC, and more computationally efﬁcient than NPEC. Figure 2 presents
results in the proof-of-concept PointMass task. The results for Gridworld ,HalfCheetah
andHopper , in section A.2.4, are qualitatively similar.
7
Published as a conference paper at ICLR 2021
Table 2: Low reward distance from the ground-truth ( GT) inPointMaze-Train predicts high pol-
icy return even in unseen task PointMaze-Test . EPIC distance is robust to the choice of coverage
distributionD, with similar values across columns, while ERC and especially NPEC are sensitive
toD.Center : approximate distances ( 1000scale) of reward functions from GT. The coverage
distributionDis computed from rollouts in PointMaze-Train of: a uniform random policy uni,
an expertand a Mixture of these policies. DSandDAare computed by marginalizing D.Right :
mean GTreturn over 9seeds of RL training on the reward in PointMaze-{Train,Test} , and
returns for AIRL’s generator policy. Conﬁdence Intervals : see Table A.7.
Reward 1000DEPIC 1000DNPEC 1000DERC Episode Return
Function uniMixuniMixuniMix Gen. Train Test
GT 0:06 0:05 0:04 0:04 3:17 0:01 0:00 0:00 0:00 —5:196:59
Regress 35:8 33:7 26:1 1:42 38:9 0:35 9:99 90:7 2:43 —5:476:30
Pref 68:7 100 56:8 8:51 1333 9 :74 24:9 360 19 :6 —5:575:04
AIRL SO 572 520 404 817 2706 488 549 523 240 5:4327:322:7
AIRL SA 776 930 894 1067 2040 1039 803 722 964 5:0530:729:0
Mirage 17:0 0:05 397 0 :68 6:30 597 35 :3<0.01 166 —30:429:1
InPointMass the agent can accelerate xleft or right on a line. The reward functions include (
 )
or exclude (
 ) a quadratic penalty x2. The sparse reward ( S) gives a reward of 1in the region0:05
from the origin. The dense reward ( D) is a shaped version of the sparse reward. The magnitude
reward ( M) is the negative distance of the agent from the origin.
We ﬁnd that EPIC correctly identiﬁes the equivalent reward pairs ( S
-D
 andS
-D
)
with estimated distance <1103. By contrast, NPEC has substantial approximation error:
DNPEC (D
;S
) = 0:58. Similarly,DERC(D
;S
) = 0:56due to ERC’s erroneous handling of
stochastic initial states. Moreover, NPEC is computationally inefﬁcient: Figure 2(b) took 31 hours to
compute. By contrast, the ﬁgures for EPIC and ERC were generated in less than two hours, and a
lower precision approximation of EPIC ﬁnishes in just 17seconds (see section A.2.6).
6.2 S ENSITIVITY OF REWARD DISTANCE TO COVERAGE DISTRIBUTION
Reward distances should be robust to the choice of coverage distribution D. In Table 2 (center),
we report distances from the ground-truth reward ( GT) to reward functions (rows) across coverage
distributionsD2funi;;Mixg(columns). We ﬁnd EPIC is fairly robust to the choice of D
with a similar ratio between rows in each column D. By contrast, ERC and especially NPEC are
substantially more sensitive to the choice of D.
We evaluate in the PointMaze MuJoCo task from Fu et al. [8], where a point mass agent must
navigate around a wall to reach a goal. The coverage distributions Dare induced by rollouts from
three different policies: unitakes actions uniformly at random, producing broad support over
transitions;is an expert policy, yielding a distribution concentrated around the goal; and Mix is a
mixture of the two. In EPIC, DSandDAare marginalized from Dand so also vary with D.
We evaluate four reward learning algorithms: Regress ion onto reward labels [ target method from 6,
section 3.3], Pref erence comparisons on trajectories [ 6], and adversarial IRL with a state-only ( AIRL
SO) and state-action ( AIRL SA ) reward model [ 8]. All models are trained using synthetic data from
an oracle with access to the ground-truth; see section A.2.2 for details.
We ﬁnd EPIC is robust to varying Dwhen comparing the learned reward models: the distance varies
by less than 2, and the ranking between the reward models is the same across coverage distributions.
By contrast, NPEC is highly sensitive to D: the ratio of AIRL SO (817) toPref (8:51) is96 : 1
underunibut only 2 : 1 (2706 : 1333 ) under. ERC lies somewhere in the middle: the ratio is
22 : 1 (549 : 24:9) underuniand3 : 2 (523 : 360 ) under.
We evaluate the effect of pathological choices of coverage distribution Din Table A.8. For example,
Indindependently samples states and next states, giving physically impossible transitions, while Jail
constrains rollouts to a tiny region excluding the goal. We ﬁnd that the ranking of EPIC changes in
only one distribution, whilst the ranking of NPEC changes in two cases and ERC changes in all cases.
8
Published as a conference paper at ICLR 2021
However, we do ﬁnd that EPIC is sensitive to DonMirage , a reward function we explicitly designed
to break these methods. Mirage assigns a larger reward when close to a “mirage” state than when at
the true goal, but is identical to GTat all other points. The “mirage” state is rarely visited by random
exploration unias it is far away and on the opposite side of the wall from the agent. The expert
policyis even less likely to visit it, as it is not on or close to the optimal path to the goal. As a
result, the EPIC distance from Mirage toGT(Table 2, bottom row) is small under uniand.
In general, any black-box method for assessing reward models – including the rollout method – only
has predictive power on transitions visited during testing. Fortunately, we can achieve a broad support
over states with Mix: it often navigates around the wall due to , but strays from the goal thanks to
uni. As a result, EPIC under Mix correctly infers that Mirage is far from the ground-truth GT.
These empirical results support our theoretically inspired recommendation from section 4: “in general,
it is best to choose Dto have broad coverage over plausible transitions.” Distributions such as
are too narrow, assigning coverage only on a direct path from the initial state to the goal. Very
broad distributions such as Indwaste probability mass on impossible transitions like teleporting.
Distributions like Mix strike the right balance between these extremes.
6.3 P REDICTING POLICY PERFORMANCE FROM REWARD DISTANCE
We ﬁnd that low distance from the ground-truth reward GT(Table 2, center) predicts high GTreturn
(Table 2, right) of policies optimized for that reward. Moreover, the distance is predictive of return not
just in PointMaze-Train where the reward functions were trained and evaluated in, but also in the
unseen variant PointMaze-Test . This is despite the two variants differing in the position of the
wall, such that policies for PointMaze-Train run directly into the wall in PointMaze-Test .
BothRegress andPref achieve very low distances at convergence, producing near-expert policy
performance. The AIRL SO andAIRL SA models have reward distances an order of magnitude
higher and poor policy performance. Yet intriguingly, the generator policies for AIRL SO andAIRL
SA– trained simultaneously with the reward – perform reasonably in PointMaze-Train . This
suggests the learned rewards are reasonable on the subset of transitions taken by the generator policy,
yet fail to transfer to the different transitions taken by a policy being trained from scratch.
Figure A.6 shows reward distance and policy regret during reward model training. The lines all
closely track each other, showing that the distance to GTis highly correlated with policy regret for
intermediate reward checkpoints as well as at convergence. Regress andPref converge quickly
to low distance and low regret, while AIRL SO andAIRL SA are slower and more unstable.
7 C ONCLUSION
Our novel EPIC distance compares reward functions directly, without training a policy. We have
proved it is a pseudometric, is bounded and invariant to equivalent rewards, and bounds the regret of
optimal policies (Theorems 4.7, 4.8 and 4.9). Empirically, we ﬁnd EPIC correctly infers zero distance
between equivalent reward functions that the NPEC and ERC baselines wrongly consider dissimilar.
Furthermore, we ﬁnd the distance of learned reward functions to the ground-truth reward predicts the
return of policies optimized for the learned reward, even in unseen environments.
Standardized metrics are an important driver of progress in machine learning. Unfortunately, tradi-
tional policy-based metrics do not provide any guarantees as to the ﬁdelity of the learned reward
function. We believe the EPIC distance will be a highly informative addition to the evaluation toolbox,
and would encourage researchers to report EPIC distance in addition to policy-based metrics. Our
implementation of EPIC and our baselines, including a tutorial and documentation, are available at
https://github.com/HumanCompatibleAI/evaluating-rewards .
ACKNOWLEDGEMENTS
Thanks to Sam Toyer, Rohin Shah, Eric Langlois, Siddharth Reddy and Stuart Armstrong for helpful
discussions; to Miljan Martic for code-review; and to David Krueger, Matthew Rahtz, Rachel
Freedman, Cody Wild, Alyssa Dayan, Adria Garriga, Jon Uesato, Zac Kenton and Alden Hung for
feedback on drafts. This work was supported by Open Philanthropy and the Leverhulme Trust.
9
Published as a conference paper at ICLR 2021
REFERENCES
[1]Riad Akrour, Marc Schoenauer, and Michele Sebag. Preference-based policy learning. In
Machine Learning and Knowledge Discovery in Databases , 2011.
[2]Dario Amodei, Paul Christiano, and Alex Ray. Learning from hu-
man preferences, June 2017. URL https://openai.com/blog/
deep-reinforcement-learning-from-human-preferences/ .
[3]Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli,
and Edward Grefenstette. Learning to understand goal speciﬁcations by modelling reward. In
ICLR , 2019.
[4]Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond
suboptimal demonstrations via inverse reinforcement learning from observations. In ICML ,
2019.
[5]Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott
Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, Oleg Sushkov, David
Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, and Ziyu Wang. Scaling data-driven
robotics with reward sketching and batch reinforcement learning. arXiv: 1909.12200v2 [cs.RO],
2019.
[6]Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep
reinforcement learning from human preferences. In NIPS , pp. 4299–4307, 2017.
[7]Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal
control via policy optimization. In ICML , 2016.
[8]Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse
reinforcement learning. In ICLR , 2018.
[9]Ashley Hill, Antonin Rafﬁn, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene
Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert,
Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable Baselines. https:
//github.com/hill-a/stable-baselines , 2018.
[10] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward
learning from human preferences and demonstrations in Atari. In NeurIPS , pp. 8011–8023,
2018.
[11] Charles L. Lawson and Richard J. Hanson. Solving Least Squares Problems . SIAM, 1995.
[12] Eric J. Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions. In
Proceedings of the Workshop on Deep Reinforcement Learning at NeurIPS , 2020.
[13] Andrew Y . Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In ICML ,
2000.
[14] Andrew Y . Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transforma-
tions: theory and application to reward shaping. In NIPS , 1999.
[15] OpenAI. OpenAI Five. https://blog.openai.com/openai-five/ , 2018.
[16] OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew,
Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider,
Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba,
and Lei Zhang. Solving Rubik’s Cube with a robot hand. arXiv: 1910.07113v1 [cs.LG], 2019.
[17] Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI ,
2007.
[18] Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry, and Sanjit A. Seshia. Active preference-based
learning of reward functions. In RSS, July 2017.
10
Published as a conference paper at ICLR 2021
[19] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal
policy optimization algorithms. arXiv:1707.06347v2 [cs.LG], 2017.
[20] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess-
che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander
Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap,
Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the
game of Go with deep neural networks and tree search. Nature , 529(7587):484–489, 2016.
[21] Satinder P. Singh and Richard C. Yee. An upper bound on the loss from approximate optimal-
value functions. Machine Learning , 16(3):227–233, September 1994.
[22] Mel Vecerik, Oleg Sushkov, David Barker, Thomas Rothörl, Todd Hester, and Jon Scholz. A
practical approach to insertion with variable socket position using deep reinforcement learning.
InICRA , 2019.
[23] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik,
Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh,
Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P.
Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin
Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu
Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKin-
ney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris
Apps, and David Silver. Grandmaster level in StarCraft II using multi-agent reinforcement
learning. Nature , 575(7782):350–354, 2019.
[24] Steven Wang, Adam Gleave, and Sam Toyer. imitation: implementations of inverse
reinforcement learning and imitation learning algorithms. https://github.com/
humancompatibleai/imitation , 2020.
[25] Aaron Wilson, Alan Fern, and Prasad Tadepalli. A Bayesian approach for policy learning from
trajectory preference queries. In NIPS , 2012.
[26] Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy
inverse reinforcement learning. In AAAI , 2008.
[27] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei,
Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.
arXiv: 1909.08593v2 [cs.CL], 2019.
11
Published as a conference paper at ICLR 2021
A S UPPLEMENTARY MATERIAL
A.1 A PPROXIMATION PROCEDURES
A.1.1 S AMPLE -BASED APPROXIMATION FOR EPIC DISTANCE
We approximate EPIC distance (deﬁnition 4.6) by estimating Pearson distance on a set of samples,
canonicalizing the reward on-demand. Speciﬁcally, we sample a batch BVofNVsamples from the
coverage distribution D, and a batch BMofNMsamples from the joint state and action distributions
DSDA. For each (s;a;s0)2BV, we approximate the canonically shaped rewards (deﬁnition 4.1)
by taking the mean over BM:
CDS;DA(R) (s;a;s0) =R(s;a;s0) +E[
R(s0;A;S0)R(s;A;S0)
R(S;A;S0)] (5)
R(s;a;s0) +
NMX
(x;u)2BMR(s0;u;x) (6)
1
NMX
(x;u)2BMR(s;u;x )c: (7)
We drop the constant cfrom the approximation since it does not affect the Pearson distance; it can also
be estimated in O(N2
M)time byc=
N2
MP
(x;)2BMP
(x0;u)2BMR(x;u;x0). Finally, we compute
the Pearson distance between the approximate canonically shaped rewards on the batch of samples
BV, yielding an O(NVNM)time algorithm.
A.1.2 O PTIMIZATION -BASED APPROXIMATION FOR NPEC DISTANCE
DNPEC (RA;RB)(section 5.2) is deﬁned as the inﬁmum of Lpdistance over an inﬁnite set of
equivalent reward functions RRA. We approximate this using gradient descent on the reward
model
R;c;w(s;a;s0) = exp()RA(s;a;s0) +c+
w(s0)w(s); (8)
where;c2Rare scalar weights and wis a vector of weights parameterizing a deep neural network
w. The constant c2Ris unnecessary if whas a bias term, but its inclusion simpliﬁes the
optimization problem.
We optimize ;c;w to minimize the mean of the cost
J(;c;w )(s;a;s0) =kR;c;w(s;a;s0);RB(s;a;s0)kp(9)
on samples (s;a;s0)from a coverage distribution D. Note
E(S;A;S0)D[J(;c;w )(S;A;S0)]1=p=DLp;D(R;c;w;RB) (10)
upper bounds the true NPEC distance since R;c;wRA.
We found empirically that andcneed to be initialized close to their optimal values for gradient
descent to reliably converge. To resolve this problem, we initialize the afﬁne parameters to  log
andcfound by:
arg min
0;c2RE
s;a;s0D(RA(s;a;s0) +cRB(s;a;s0))2: (11)
We use the active set method of Lawson & Hanson [11] to solve this constrained least-squares problem.
These initial afﬁne parameters minimize the LpdistanceDLp;D(R;c;0(s;a;s0);RB(s;a;s0))when
p= 2with the potential ﬁxed at 0(s) = 0 .
A.1.3 C ONFIDENCE INTERVALS
We report conﬁdence intervals to help measure the degree of error introduced by the approximations.
Since approximate distances may not be normally distributed, we use bootstrapping to produce a
distribution-free conﬁdence interval. For EPIC, NPEC and Episode Return (sometimes reported as
regret rather than return), we compute independent approximate distances or returns over different
12
Published as a conference paper at ICLR 2021
Table A.1: Summary of hyperparameters and distributions used in experiments. The uniform random
coverage distribution Dunifsamples states and actions uniformly at random, and samples the next
state from the transition dynamics. Random policy unitakes uniform random actions. The synthetic
expert policy was trained with PPO on the ground-truth reward. Mixture samples actions from
eitherunior, switching between them at each time step with probability 0:05. Warmstart Size is
the size of the dataset used to compute initialization parameters described in section A.1.2.
Parameter Value In experiment
Coverage Distribution DRandom transitions Dunif GridWorld
Rollouts from uni PointMass ,HalfCheetah ,Hopper
uni,andMixture PointMaze
Bootstrap Samples 10 000 All
Discount
 0:99 All
EPIC
State DistributionDS Marginalized from D All
Action Distribution DA Marginalized from D All
Seeds 30 All
SamplesNV 32 768 All
Mean Samples NM 32 768 All
NPEC
Seeds 3 All
Total Time Steps 1106All
Optimizer Adam All
Learning Rate 1102All
Batch Size 4096 All
Warmstart Size 16 386 All
Loss` ` (x;y) = (xy)2All
ERC
Episodes 131 072 All
Episode Return
Seeds 9 All
See Table A.2 for the policy training hyperparameters
seeds, and then compute a bootstrapped conﬁdence interval for each seed. We use 30seeds for
EPIC, but only 9seeds for computing Episode Return and 3seeds for NPEC due to their greater
computational requirements. In ERC, computing the distance is very fast, so we instead apply
bootstrapping to the collected episodes , computing the ERC distance for each bootstrapped episode
sample.
A.2 E XPERIMENTS
A.2.1 H YPERPARAMETERS FOR APPROXIMATE DISTANCES
Table A.1 summarizes the hyperparameters and distributions used to compute the distances between
reward functions. Most parameters are the same across all environments. We use a coverage
distribution of uniform random transitions Dunifin the simple GridWorld environment with known
determinstic dynamics. In other environments, the coverage distribution is sampled from rollouts of a
policy. We use a random policy uniforPointMass ,HalfCheetah andHopper in the hand-
designed reward experiments (section 6.1). In PointMaze , we compare three coverage distributions
(section 6.2) induced by rollouts of uni, an expert policy and a Mixture of the two policies,
sampling actions from either uniorand switching between them with probability 0:05per time
step.
13
Published as a conference paper at ICLR 2021
Table A.2: Hyperparameters for proximal policy optimisation (PPO) [ 19]. We used the implemen-
tation and default hyperparameters from Hill et al. [9]. PPO was used to train expert policies on
ground-truth reward and to optimize learned reward functions for evaluation.
Parameter Value In environment
Total Time Steps 1106All
Seeds 9 All
Batch Size 4096 All
Discount
 0:99 All
Entropy Coefﬁcient 0:01 All
Learning Rate 3104All
Value Function Coefﬁcient 0:5 All
Gradient Clipping Threshold 0:5 All
Ratio Clipping Thrsehold 0:2 All
Lambda (GAE) 0:95 All
Minibatches 4 All
Optimization Epochs 4 All
Parallel Environments 8 All
Table A.3: Hyperparameters for adversarial inverse reinforcement learning (AIRL) used in Wang
et al. [24].
Parameter Value
RL Algorithm PPO [19]
Total Time Steps 1 000 000
Discount
 0:99
Demonstration Time Steps 100 000
Generator Batch Size 2048
Discriminator Batch Size 50
Entropy Weight 1:0
Reward Function Architecture MLP, two 32-unit hidden layers
Potential Function Architecture MLP, two 32-unit hidden layers
Table A.4: Hyperparameters for preference comparison used in our implementation of Christiano
et al. [6].
Parameter Value Range Tested
Total Time Steps 5106[1;10106]
Batch Size 10 000 [500 ;250 000]
Trajectory Length 5 [1 ;100]
Learning Rate 1102[1104;1101]
Discount
 0:99
Reward Function Architecture MLP, two 32-unit hidden layers
Output L2 Regularization Weight 1104
Table A.5: Hyperparameters for regression used in our implementation of Christiano et al. [6,target
method from section 3.3].
Parameter Value Range Tested
Total Time Steps 10106[1;20106]
Batch Size 4096 [256 ;16 384]
Learning Rate 2102[1103;1101]
Discount
 0:99
Reward Function Architecture MLP, two 32-unit hidden layers
14
Published as a conference paper at ICLR 2021
A.2.2 T RAINING LEARNED REWARD MODELS
For the experiments on learned reward functions (sections 6.3 and 6.2), we trained reward mod-
els using adversarial inverse reinforcement learning (AIRL; 8), preference comparison [ 6] and
by regression onto the ground-truth reward [ target method from 6, section 3.3]. For AIRL, we
use an existing open-source implementation [ 24]. We developed new implementations for prefer-
ence comparison and regression, available at https://github.com/HumanCompatibleAI/
evaluating-rewards . We also use the RL algorithm proximal policy optimization (PPO; 19)
on the ground-truth reward to train expert policies to provide demonstrations for AIRL. We use 9
seeds, taking rollouts from the seed with the highest ground-truth return.
Our hyperparameters for PPO in Table A.2 were based on the defaults in Stable Baselines [ 9]. We
only modiﬁed the batch size and learning rate, and disabled value function clipping to match the
original PPO implementation.
Our AIRL hyperparameters in Table A.3 likewise match the defaults, except for increasing the total
number of timesteps to 106. Due to the high variance of AIRL, we trained 5 seeds, selecting the one
with the highest ground-truth return. While this does introduce a positive bias for our AIRL results,
in spite of this AIRL performed worse in our tests than other algorithms. Moreover, the goal in this
paper is to evaluate distance metrics, not reward learning algorithms.
For preference comparison we performed a sweep over batch size, trajectory length and learning
rate to decide on the hyperparameters in Table A.4. Total time steps was selected once diminishing
returns were observed in loss curves. The exact value of the regularization weight was found to be
unimportant, largely controlling the scale of the output at convergence.
Finally, for regression we performed a sweep over batch size, learning rate and total time steps to
decide on the hyperparameters in Table A.5. We found batch size and learning rate to be relatively
unimportant with many combinations performing well, but regression was found to converge slowly
but steadily requiring a relatively large 10106time steps for good performance in our environments.
All algorithms are trained on synthetic data generated from the ground-truth reward function. AIRL
is provided with a large demonstration dataset of 100 000 time steps from an expert policy trained
on the ground-truth reward (see Table A.3). In preference comparison and regression, each batch is
sampled afresh from the coverage distribution speciﬁed in Table A.1 and labeled according to the
ground-truth reward.
A.2.3 C OMPUTING INFRASTRUCTURE
Experiments were conducted on a workstation (Intel i9-7920X CPU with 64 GB of RAM), and
a small number of r5.24xlarge AWS VM instances, with 48 CPU cores on an Intel Skylake
processor and 768 GB of RAM. It takes less than three weeks of compute on a single r5.24xlarge
instance to run all the experiments described in this paper.
A.2.4 C OMPARING HAND -DESIGNED REWARD FUNCTIONS
We compute distances between hand-designed reward functions in four environments: GridWorld ,
PointMass ,HalfCheetah andHopper . The reward functions for GridWorld are described
in Figure A.1, and the distances are reported in Figure A.2. We report the approximate distances and
conﬁdence intervals between reward functions in the other environments in Figures A.3, A.4 and A.5.
We ﬁnd the (approximate) EPIC distance closely matches our intuitions for similarity between the
reward functions. NPEC often produces similar results to EPIC, but unfortunately is dogged by opti-
mization error. This is particularly notable in higher-dimensional environments like HalfCheetah
andHopper , where the NPEC distance often exceeds the theoretical upper bound of 1:0and the
conﬁdence interval width is frequently larger than 0:2.
By contrast, ERC distance generally has a tight conﬁdence interval, but systematically fails in
the presence of shaping. For example, it conﬁdently assigns large distances between equivalent
reward pairs in PointMass such as S
-D
. However, ERC produces reasonable results in
HalfCheetah andHopper where rewards are all similarly shaped. In fact, ERC picks up on
a detail in Hopper that EPIC misses: whereas EPIC assigns a distance of around 0:71between
15
Published as a conference paper at ICLR 2021
Distance Wall-Clock Environment Reward # of 95% CI Width
Metric Time Time Steps Queries Seeds Max Mean
EPIC Quick 17 s 8192 1 :671073 0:023 04 0:008 60
EPIC 6738 s 65 536 1 :0710930 0:005 58 0:002 34
NPEC 29 769 s 7 :501077:501073 0:315 91 0:066 20
ERC 1376 s 6 :551066:55106— 0:015 81 0:005 33
RL (PPO) 14 745 s 7 :501077:501073 — —
Table A.6: Time and resources taken by different metrics to perform 25distance comparisons
onPointMass , and the conﬁdence interval widths obtained (smaller is better). Methods EPIC ,
NPEC andERC correspond to Figures 2(a), (b) and (c) respectively. EPIC Quick is an abbreviated
version with fewer samples. RL (PPO) is estimated from the time taken using PPO to train a single
policy (16m:23s) until convergence ( 106time steps). EPIC samples NM+NVtime steps from the
environment and performs NMNVreward queries. In EPIC Quick ,NM=NV= 4096 ; inEPIC ,
NM=NV= 302768 . Other methods query the reward once per environment time step.
all rewards of different types (running vs backﬂipping), ERC assigns lower distances when the
rewards are in the same direction (forward or backward). Given this, ERC may be attractive in some
circumstances, especially given the ease of implementation. However, we would caution against
using it in isolation due to the likelihood of misleading results in the presence of shaping.
A.2.5 C OMPARING LEARNED REWARD FUNCTIONS
Previously, we reported the mean approximate distance from a ground-truth reward of four learned
reward models in PointMaze (Table 2). Since these distances are approximate, we report 95%
lower and upper bounds computed via bootstrapping in Table A.7. We also include the relative
difference of the upper and lower bounds from the mean, ﬁnding the relative difference to be fairly
consistent across reward models for a given algorithm and coverage distribution pair. The relative
difference is less than 1% for all EPIC and ERC distances. However, NPEC conﬁdence intervals can
be as wide as 50%: this is due to the method’s high variance, and the small number of seeds we were
able to run because of the method’s computational expense.
A.2.6 R UNTIME OF DISTANCE METRICS
We report the empirical runtime for EPIC and baselines in Table A.6, performing 25pairwise com-
parisons across 5reward functions in PointMass . These comparisons were run on an unloaded
machine running Ubuntu 20.04 (kernel 5.4.0-52) with an Intel i9-7920X CPU and 64 GB of RAM.
We report sequential runtimes: runtimes for all methods could be decreased further by parallelizing
across seeds. The algorithms were conﬁgured to use 8parallel environments for sampling. Inference
and training took place on CPU. All methods used the same TensorFlow conﬁguration, parallelizing
operations across threads both within and between operations. We found GPUs offered no perfor-
mance beneﬁt in this setting, and in some cases even increased runtime. This is due to the ﬁxed cost
of CPU-GPU communication, and the relatively small size of the observations.
We ﬁnd that in just 17seconds EPIC can provide results with a 95% conﬁdence interval <0:023,
an order of magnitude tighter than NPEC running for over 8hours. Training policies for all learned
rewards in this environment using PPO is comparatively slow, taking over 4hours even with only 3
seeds. While ERC is relatively fast, it takes a large number of samples to achieve tight conﬁdence
intervals. Moreover, since PointMass has stochastic initial states, ERC can take on arbitrary values
under shaping, as discussed in sections 5.1 and 6.3.
16
Published as a conference paper at ICLR 2021
012
000
000
100
000
100
00
00
10
000
000
00
00
00Sparse Reward
-1-1-1
-1-1-1
3-1-1
-4-4-4
0-4-4
-4-4
-4-4
0-4
222
222
22
22
22Dense Reward
012
000
000
-100
000
-100
00
00
-10
000
000
00
00
00Penalty Reward
000
010
000
010
000
00
01
00
000
010
00
10
00Center Reward
0 1 2012
-100
-10-1
40-1
-10-1
40-1
-10
-10
40
-100
-10-1
00
0-1
0-1Path Reward
0 1 2-400
-40-1
40-1
-40-1
40-1
-40
-40
40
-400
-40-1
00
0-1
0-1Cliff Reward
-4-3-2-101234
-4-3-2-101234
-4-3-2-101234
Figure A.1: Heatmaps of reward functions R(s;a;s0)for a33deterministic gridworld. R(s;stay;s)
is given by the central circle in cell s.R(s;a;s0)is given by the triangular wedge in cell sadjacent
to cells0in directiona. Optimal action(s) (for inﬁnite horizon, discount
= 0:99) have bold labels
against a hatched background. See Figure A.2 for the distance between all reward pairs.
17
Published as a conference paper at ICLR 2021
SparseDenseCenterPenaltyPathCliff
RBSparse
Dense
Center
Penalty
Path
CliffRA0.0000 0.0000 0.7500 1.0000 0.1602 0.3676
0.0000 0.0000 0.7500 1.0000 0.1602 0.3676
0.7500 0.7500 0.0000 0.6614 0.7071 0.6692
1.0000 1.0000 0.6614 0.0000 0.9871 0.9300
0.1602 0.1602 0.7071 0.9871 0.0000 0.2672
0.3676 0.3676 0.6692 0.9300 0.2672 0.0000
0.00.20.40.60.81.0
D(RA,RB)
(a) EPIC
SparseDenseCenterPenaltyPathCliff
RBSparse
Dense
Center
Penalty
Path
CliffRA0.0000 0.0000 1.0016 1.0009 0.3068 0.7038
0.0000 0.0000 1.0016 1.0000 0.3068 0.7038
1.0011 1.0133 0.0000 0.9853 1.0016 1.0004
1.0009 1.0213 0.9824 0.0000 1.0010 1.0010
0.3069 0.3069 1.0023 1.0000 0.0000 0.5476
0.7031 0.7031 1.0025 1.0000 0.5468 0.0000
0.00.20.40.60.81.0
D(RA,RB)
(b) NPEC
Figure A.2: Distances (EPIC, top; NPEC, bottom) between hand-designed reward functions for the
33deterministic Gridworld environment. EPIC and NPEC produce similar results, but EPIC
more clearly discriminates between rewards whereas NPEC distance tends to saturate. For example,
the NPEC distance from Penalty to other rewards lies in the very narrow [0:98;1:0]range, whereas
EPIC uses the wider [0:66;1:0]range. See Figure A.1 for deﬁnitions of each reward. Distances are
computed using tabular algorithms. We do not report conﬁdence intervals since these algorithms are
deterministic and exact up to ﬂoating point error.
18
Published as a conference paper at ICLR 2021
S
S
D
D
M
RBS
S
D
D
M
RA0.00 0.51 0.00 0.51 0.63
0.51 0.00 0.51 0.00 0.55
0.00 0.51 0.00 0.51 0.63
0.51 0.00 0.51 0.00 0.55
0.63 0.55 0.63 0.55 0.00
(a) EPIC Median
S
S
D
D
M
RB0.00 0.88 0.22 0.86 0.96
0.92 0.00 0.94 0.04 0.74
0.58 0.95 0.00 0.81 0.97
0.88 0.04 0.84 0.00 0.58
0.93 0.79 0.80 0.62 0.00 (b) NPEC Median
S
S
D
D
M
RB0.00 0.41 0.56 0.56 0.57
0.41 0.00 0.61 0.55 0.62
0.56 0.61 0.00 0.11 0.64
0.56 0.55 0.11 0.00 0.64
0.57 0.62 0.64 0.64 0.00
0.00.51.0
¯D(RA,RB) (c) ERC Median
S
S
D
D
M
RBS
S
D
D
M
RA6e-52e-39e-52e-39e-4
2e-36e-52e-35e-52e-3
9e-52e-37e-52e-39e-4
2e-35e-52e-35e-52e-3
9e-42e-39e-42e-35e-5
(d) EPIC CI Width
S
S
D
D
M
RB2e-59e-33e-22e-22e-1
9e-29e-72e-14e-31e-1
1e-11e-21e-56e-31e-1
6e-22e-39e-22e-66e-2
1e-11e-13e-11e-24e-7 (e) NPEC CI Width
S
S
D
D
M
RB7e-91e-24e-34e-33e-3
1e-27e-96e-35e-34e-3
4e-36e-31e-81e-34e-3
4e-35e-31e-31e-84e-3
3e-34e-34e-34e-37e-9
0.00.51.0
DW(RA,RB) (f) ERC CI Width
Figure A.3: Approximate distances between hand-designed reward functions in PointMass . The
coverage distribution Dis sampled from rollouts of a policy unitaking actions uniformly at random.
Key: The agent has position x2R, velocity _x2Rand can accelerate x2R, producing future
positionx02R.
 quadratic penalty on control x2,
 no control penalty. SisSparse(x) =
1[jxj<0:05],Dis shaped Dense(x;x0) = Sparse( x)+jx0jjxj, while MisMagnitude( x) =jxj.
Conﬁdence Interval (CI) : 95% CI computed by bootstraping over 10 000 samples.
RB
RA0.00 0.17 1.00 0.98
0.17 0.00 0.98 0.94
1.00 0.98 0.00 0.17
0.98 0.94 0.17 0.00
(a) EPIC Median
RB0.01 0.03 1.17 1.11
0.03 0.01 1.13 1.19
1.15 1.09 0.01 0.03
1.08 1.08 0.03 0.01 (b) NPEC Median
RB0.00 0.02 1.00 1.00
0.02 0.00 1.00 1.00
1.00 1.00 0.00 0.02
1.00 1.00 0.02 0.00
0.000.250.500.751.00
¯D(RA,RB) (c) ERC Median
RB
RA5e-5 6e-4 1e-8 1e-4
6e-4 5e-5 1e-4 4e-4
1e-8 1e-4 5e-5 7e-4
1e-4 4e-4 7e-4 6e-5
(d) EPIC CI Width
RB1e-2 1e-2 1e-1 3e-1
2e-2 1e-2 1e-2 1e-1
2e-1 2e-1 1e-2 1e-2
3e-1 8e-2 9e-3 1e-2 (e) NPEC CI Width
RB1e-8 2e-4 1e-16 3e-6
2e-4 1e-8 3e-6 1e-5
1e-16 3e-6 1e-8 2e-4
3e-6 1e-5 2e-4 1e-8
0.000.250.500.751.00
DW(RA,RB) (f) ERC CI Width
Figure A.4: Approximate distances between hand-designed reward functions in HalfCheetah .
The coverage distribution Dis sampled from rollouts of a policy unitaking actions uniformly at
random. Key:
 is a reward proportional to the change in center of mass and moving forward is
rewarded when
 to the right, and moving backward is rewarded when
 to the left.
 quadratic
control penalty,
 no control penalty. Conﬁdence Interval (CI) : 95% CI computed by bootstraping
over10 000 samples.
19
Published as a conference paper at ICLR 2021
RB
RA0.00 0.00 0.99 0.99 0.71 0.71 0.71 0.71
0.00 0.00 0.99 0.99 0.71 0.71 0.71 0.71
0.99 0.99 0.00 0.00 0.71 0.71 0.71 0.71
0.99 0.99 0.00 0.00 0.71 0.71 0.71 0.71
0.71 0.71 0.71 0.71 0.00 0.00 1.00 1.00
0.71 0.71 0.71 0.71 0.00 0.00 1.00 1.00
0.71 0.71 0.71 0.71 1.00 1.00 0.00 0.00
0.71 0.71 0.71 0.71 1.00 1.00 0.00 0.00
(a) EPIC Median
RB0.00 0.00 1.06 1.01 1.20 1.24 0.95 1.00
0.00 0.00 0.92 1.40 1.24 1.15 0.97 1.02
1.25 1.46 0.00 0.00 1.02 0.99 0.95 1.08
0.94 1.13 0.00 0.00 1.03 1.00 0.95 0.99
1.73 1.42 1.02 1.18 0.00 0.00 1.03 0.98
1.29 1.27 1.12 1.17 0.00 0.00 0.97 0.92
1.41 1.14 1.36 1.52 1.07 1.03 0.00 0.00
1.18 1.65 1.16 1.69 1.09 1.01 0.00 0.00 (b) NPEC Median
RB0.00 0.00 0.97 0.97 0.37 0.37 0.93 0.93
0.00 0.00 0.97 0.97 0.37 0.37 0.93 0.93
0.97 0.97 0.00 0.00 0.88 0.88 0.47 0.47
0.97 0.97 0.00 0.00 0.88 0.88 0.47 0.47
0.37 0.37 0.88 0.88 0.00 0.00 1.00 1.00
0.37 0.37 0.88 0.88 0.00 0.00 1.00 1.00
0.93 0.93 0.47 0.47 1.00 1.00 0.00 0.00
0.93 0.93 0.47 0.47 1.00 1.00 0.00 0.00
0.00.20.40.60.81.0
¯D(RA,RB) (c) ERC Median
RB
RA6e-5 8e-5 2e-4 3e-4 1e-3 1e-3 1e-3 1e-3
8e-5 5e-5 2e-4 2e-4 1e-3 1e-3 1e-3 1e-3
3e-4 2e-4 6e-5 8e-5 1e-3 1e-3 1e-3 1e-3
2e-4 3e-4 8e-5 6e-5 1e-3 1e-3 1e-3 1e-3
1e-3 1e-3 1e-3 1e-3 5e-5 2e-5 1e-8 4e-8
1e-3 1e-3 1e-3 1e-3 1e-5 6e-5 4e-8 1e-7
1e-3 1e-3 1e-3 1e-3 1e-8 4e-8 5e-5 1e-5
1e-3 1e-3 1e-3 1e-3 4e-8 1e-7 1e-5 6e-5
(d) EPIC CI Width
RB3e-3 3e-3 8e-1 4e-1 3e-1 5e-2 8e-2 1e-1
3e-3 3e-3 9e-1 2e03e-1 3e-1 8e-2 2e-1
4e-1 1e04e-3 4e-3 2e-1 2e-1 1e-1 3e-1
4e-1 6e-1 2e-3 4e-3 2e-1 2e-1 6e-2 1e-1
7e-1 1e-1 3e-1 3e-1 2e-4 1e-5 2e-1 3e-1
4e-1 6e-1 7e-1 4e-1 3e-4 2e-4 9e-2 2e-1
4e-2 4e-1 6e-1 2e-1 3e-1 2e-1 2e-4 3e-4
1e0 2e07e-1 7e-1 4e-1 1e-1 1e-4 2e-4 (e) NPEC CI Width
RB7e-9 5e-7 7e-4 7e-4 4e-3 4e-3 2e-3 2e-3
5e-7 7e-9 7e-4 7e-4 4e-3 4e-3 2e-3 2e-3
7e-4 7e-4 1e-8 6e-7 2e-3 2e-3 4e-3 4e-3
7e-4 7e-4 6e-7 1e-8 2e-3 2e-3 4e-3 4e-3
4e-3 4e-3 2e-3 2e-3 1e-8 2e-61e-16 4e-10
4e-3 4e-3 2e-3 2e-3 2e-6 1e-84e-10 2e-9
2e-3 2e-3 4e-3 4e-31e-16 4e-10 1e-8 2e-6
2e-3 2e-3 4e-3 4e-34e-10 2e-9 2e-6 1e-8
0.00.20.40.60.81.0
DW(RA,RB) (f) ERC CI Width
Figure A.5: Approximate distances between hand-designed reward functions in Hopper . The
coverage distribution Dis sampled from rollouts of a policy unitaking actions uniformly at random.
Key:
is a reward proportional to the change in center of mass and
 is the backﬂip reward deﬁned
in Amodei et al. [2, footnote ]. Moving forward is rewarded when
 or
 is to the right, and moving
backward is rewarded when
 or
 is to the left.
 quadratic control penalty,
 no control penalty.
Conﬁdence Interval (CI) : 95% CI computed by bootstraping over 10 000 samples.
20
Published as a conference paper at ICLR 2021
Table A.7: Approximate distances of reward functions from the ground-truth ( GT). We report the 95%
bootstrapped lower and upper bounds, the mean, and a 95% bound on the relative error from the mean.
Distances ( 1000scale) use coverage distribution Dfrom rollouts in the PointMaze-Train
environment of: a uniform random policy uni, an expertand a Mixture of these policies. DS
andDAare computed by marginalizing D.
(a) 95% lower bound DLOWof approximate distance.
Reward 1000DLOW
EPIC1000DLOW
NPEC1000DLOW
ERCEpisode Return
Function uniMixuniMixuniMix Train Test
GT 0:03 0:02<0.01 0:02 1:43<0.01 0:00 0:00 0:004:465:82
Regress 35:5 33:6 26:0 1:22 38:8 0:33 9:94 90:2 2:424:75:63
Pref 68:3 100 56:6 7:02 1239 9:25 24:7 358 19 :55:264:88
AIRL SO 570 519 402 734 1645 424 547 521 238 27:322:7
AIRL SA 774 930 894 956 723 952 802 720 963 29:928
Mirage 3:49 0:02 381 0 :17 4:03 481 25 :8<0.01 16228:426:2
(b) Mean approximate distance D. Results are the same as Table 2.
Reward 1000DEPIC 1000DNPEC 1000DERC Episode Return
Function uniMixuniMixuniMix Train Test
GT 0:06 0:05 0:04 0:04 3:17 0:01 0:00 0:00 0:005:196:59
Regress 35:8 33:7 26:1 1:42 38:9 0:35 9:99 90:7 2:435:476:3
Pref 68:7 100 56:8 8:51 1333 9 :74 24:9 360 19 :65:575:04
AIRL SO 572 520 404 817 2706 488 549 523 240 27:322:7
AIRL SA 776 930 894 1067 2040 1039 803 722 964 30:729
Mirage 17:0 0:05 397 0 :68 6:30 597 35 :3<0.01 16630:429:1
(c) 95% upper bound DUPof approximate distance.
Reward 1000DUP
EPIC1000DUP
NPEC1000DUP
ERCEpisode Return
Function uniMixuniMixuniMix Train Test
GT 0:09 0:07 0:07 0:06 6:14 0:01 <0.01 <0.01 <0.01 6:047:14
Regress 36:1 33:7 26:2 1:53 39:1 0:37 10:0 91:2 2:446:266:83
Pref 69:1 101 57:1 10:0 1432 10 :1 25:2 361 19 :75:95:22
AIRL SO 574 520 407 982 3984 532 551 526 242 27:322:8
AIRL SA 779 930 895 1241 4378 1124 805 724 964 31:729:8
Mirage 35:2 0:09 414 1 :66 10:8 821 45 :4<0.01 1713231:4
(d) Relative 95% conﬁdence interval DREL=maxUpper
Mean1;1Lower
Meanin percent. The population mean
is contained within DREL%of the sample mean in Table A.7b with 95% probability.
Reward DREL
EPIC%DREL
NPEC%DREL
ERC% Episode Return
Function uniMixuniMixuniMix Train Test
GT 50:0 62:5 80:0 61:8 94:0 29:7inf inf inf 0:16 0:12
Regress 0:81 0:14 0:40 14:2 0:42 7:48 0:53 0:55 0:57 0:14 0:11
Pref 0:61 0:14 0:44 17:5 7:49 5:02 0:90 0:48 0:48 0:06 0:04
AIRL SO 0:38 0:08 0:67 20:2 47:2 13:2 0:34 0:40 0:69 <0.01 <0.01
AIRL SA 0:35 0:02 0:08 16:3 115 8:42 0:23 0:26 0:04 0:03 0:04
Mirage 108 65:5 4:17 142 70 :9 37:5 28:5 0:55 2:66 0:07 0:10
21
Published as a conference paper at ICLR 2021
Table A.8: Approximate distances of reward functions from the ground-truth ( GT) under pathological
coverage distributions. We report the 95% bootstrapped lower and upper bounds, the mean, and a
95% bound on the relative error from the mean. Distances ( 1000scale) use four different coverage
distributionsD.independently samples states, actions and next states from the marginal distri-
butions of rollouts from the uniform random policy uniin the PointMaze-Train environment.
Indindependently samples the components of states and next states from N(0;1), and actions from
U[1;1].Jailconsists of rollouts of unirestricted to a small 0:090:09“jail” square that excludes
the goal state 0:5distance away. badare rollouts in PointMaze-Train of a policy that goes to
the corner opposite the goal state. andIndare not supported by ERC since they do not produce
complete episodes.
(a) 95% lower bound DLOWof approximate distance.
Reward 1000DLOW
EPIC1000DLOW
NPEC1000DLOW
ERC
Function Ind Jailbad Ind Jailbad Jailbad
Regress 127 398 705 205 87 :6 590 2433 898 809 456
Pref 146 433 462 349 97 :4 632 661 221 372 332
AIRL SO 570 541 712 710 697 821 957 621 751 543
AIRL SA 768 628 558 669 720 960 940 2355 428 753
Mirage 9:22 0:03<0.01 0:02 0:41 0:05 11:2 31:3 <0.01 0:02
(b) Mean approximate distance D. Results are the same as Table 2.
Reward 1000DEPIC 1000DNPEC 1000DERC
Function Ind Jailbad Ind Jailbad Jailbad
Regress 128 398 705 206 97 :2 591 2549 921 810 458
Pref 147 433 463 349 117 633 683 237 374 333
AIRL SO 573 541 713 710 826 823 988 852 753 545
AIRL SA 771 628 558 669 859 962 964 2694 430 754
Mirage 42:4 0:06 0:03 0:05 1:41 0:25 18:3 39 <0.01 0:02
(c) 95% upper bound DUPof approximate distance.
Reward 1000DUP
EPIC1000DUP
NPEC1000DUP
ERC
Function Ind Jailbad Ind Jailbad Jailbad
Regress 129 398 706 206 106 593 2654 948 812 460
Pref 148 433 464 349 132 635 705 265 376 335
AIRL SO 576 541 713 710 939 825 1047 1021 755 547
AIRL SA 774 628 559 669 1015 963 981 3012 432 756
Mirage 85:9 0:09 0:05 0:09 3:25 0:46 28:6 45:3 <0.01 0:02
(d) Relative 95% conﬁdence interval DREL=maxUpper
Mean1;1Lower
Meanin percent. The population mean
is contained within DREL%of the sample mean in Table A.8b with 95% probability.
Reward DREL
EPIC% DREL
NPEC%DREL
ERC%
Function Ind Jailbad Ind Jailbad Jailbad
Regress 0:79 0:01 0:08 0:06 9:81 0:24 4:54 2:89 0:18 0:42
Pref 0:76<0.01 0:16 0:07 16:9 0:35 3:35 11:7 0:47 0:48
AIRL SO 0:48<0.01 0:07 0:02 15:6 0:28 6:01 27:1 0:23 0:38
AIRL SA 0:38<0.01 0:13 0:03 18:2 0:22 2:47 12:6 0:45 0:23
Mirage 103 50 80 83 :3 131 85 :4 56:4 19:7 0:54 0:55
22
Published as a conference paper at ICLR 2021
0 20 40 60 80 100
Regress Training Progress (%)0.00.51.0Distance
0102030
RegretEPIC NPEC ERC RL Train RL Test
(a) Comparisons of Regress using all distance algorithms.
0 20 40 60 80 100
Pref Training Progress (%)0.00.51.0Distance
0102030
RegretEPIC NPEC ERC RL Train RL Test
(b) Comparisons of Pref using all distance algorithms.
0 20 40 60 80 100
AIRL SO Training Progress (%)0.00.51.0Distance
102030
RegretEPIC NPEC ERC RL Train RL Test
(c) Comparisons of AIRL SO using all distance algorithms.
0 20 40 60 80 100
AIRL SA Training Progress (%)0.00.51.0Distance 2030
RegretEPIC NPEC ERC RL Train RL Test
(d) Comparisons of AIRL SA using all distance algorithms.
Figure A.6: Distance of reward checkpoints from the ground-truth in PointMaze and policy
regret for reward checkpoints during reward model training. Each point evaluates a reward function
checkpoint from a single seed. EPIC, NPEC and ERC distance use the Mixture distribution. Regret
is computed by running RL on the checkpoint. The shaded region represents the bootstrapped 95%
conﬁdence interval for the distance or regret at that checkpoint, calculated following Section A.1.3.
23
Published as a conference paper at ICLR 2021
0 20 40 60 80 100
Reward Model Training Progress (%)0.00.51.0DistanceAIRL SA AIRL SO Pref Regress
(a) Comparisons using EPIC on all reward models.
0 20 40 60 80 100
Reward Model Training Progress (%)0.00.51.0DistanceAIRL SA AIRL SO Pref Regress
(b) Comparisons using NPEC on all reward models.
0 20 40 60 80 100
Reward Model Training Progress (%)0.00.51.0DistanceAIRL SA AIRL SO Pref Regress
(c) Comparisons using ERC on all reward models.
0 20 40 60 80 100
Reward Model Training Progress (%)020RegretAIRL SA AIRL SO
RL TrainPref
RL TestRegress
(d) Comparisons using Episode Return on all reward models.
Figure A.7: Distance of reward checkpoints from the ground-truth in PointMaze and policy
regret for reward checkpoints during reward model training. Each point evaluates a reward function
checkpoint from a single seed. EPIC, NPEC and ERC distance use the Mixture distribution. Regret
is computed by running RL on the checkpoint. The shaded region represents the bootstrapped 95%
conﬁdence interval for the distance or regret at that checkpoint, calculated following Section A.1.3.
24
Published as a conference paper at ICLR 2021
A.3 P ROOFS
A.3.1 B ACKGROUND
Proposition 3.4. The binary relation is an equivalence relation. Let RA;RB;RC:SAS! R
be bounded reward functions. Then is reﬂexive, RARA; symmetric, RARBimplies
RBRA; and transitive, (RARB)^(RBRC)impliesRARC.
Proof.RARAsince choosing = 1>0and(s) = 0 , a bounded potential function, we have
RA(s;a;s0) =RA(s;a;s0) +
(s0)(s)for alls;s02S anda2A.
SupposeRARB. Then there exists some  >0and a bounded potential function  :S!R
such thatRB(s;a;s0) =RA(s;a;s0) +
(s0)(s)for alls;s02S anda2A. Rearranging:
RA(s;a;s0) =1
RB(s;a;s0) +
1
(s0)
1
(s)
: (12)
Since1
>0and0(s) =1
(s)is a bounded potential function, it follows that RBRA.
Finally, suppose RARBandRBRC. Then there exists some 1;2>0and bounded potential
functions 1;2:S!Rsuch that for all s;s02S anda2A:
RB(s;a;s0) =1RA(s;a;s0) +
1(s0)1(s); (13)
RC(s;a;s0) =2RB(s;a;s0) +
2(s0)2(s): (14)
Substituting the expression for RBinto the expression for RC:
RC(s;a;s0) =2(1RA(s;a;s0) +
1(s0)1(s)) +
2(s0)2(s) (15)
=12RA(s;a;s0) +
(21(s0) +  2(s0))(21(s) +  2(s)) (16)
=RA(s;a;s0) +
(s0)(s); (17)
where=12>0and(s) =21(s) +  2(s)is bounded. Thus RARC.
A.3.2 E QUIVALENT -POLICY INVARIANT COMPARISON (EPIC) PSEUDOMETRIC
Proposition 4.2 (The Canonically Shaped Reward is Invariant to Shaping) .LetR:SAS! R
be a reward function and  :S!Ra potential function. Let
2[0;1]be a discount rate, and
DS2(S)andDA2(A)be distributions over states and actions. Let R0denoteRshaped by
:R0(s;a;s0) =R(s;a;s0) +
(s0)(s). Then the canonically shaped R0andRare equal:
CDS;DA(R0) =CDS;DA(R).
Proof. Lets;a;s02SAS . Then by substituting in the deﬁnition of R0and using linearity of
expectations:
CDS;DA(R0) (s;a;s0),R0(s;a;s0) +E[
R0(s0;A;S0)R0(s;A;S0)
R0(S;A;S0)](18)
= (R(s;a;s0) +
(s0)(s)) (19)
+E

R(s0;A;S0) +
2(S0)
(s0)
E[R(s;A;S0) +
(S0)(s)]
E

R(S;A;S0) +
2(S0)
(S)
=R(s;a;s0) +E[
R(s0;A;S0)R(s;A;S0)
R(S;A;S0)] (20)
+ (
(s0)(s))E[
(s0)(s)]
+E

2(S0)
(S0)
E

2(S0)
(S)
=R(s;a;s0) +E[
R(s0;A;S0)R(s;A;S0)
R(S;A;S0)] (21)
,CDS;DA(R) (s;a;s0); (22)
where the penultimate step uses E[(S0)] =E[(S)]sinceSandS0are identically distributed.
25
Published as a conference paper at ICLR 2021
Proposition 4.3. LetSandAbe ﬁnite, withjSj 2. LetDS2(S)andDA2(A). Let
R;  :SAS! Rbe reward functions, with (s;a;s0) =I[(s;a;s0) = (x;u;x0)],2R,
x;x02S andu2A. Let DS;DA(R)(s;a;s0) =CDS;DA(R) (s;a;s0)R(s;a;s0). Then:
kDS;DA(R+)DS;DA(R)k1=(1 +
DS(x))DA(u)DS(x0): (3)
Proof. Observe that:
DS;DA(R)(s;a;s0) =E[
R(s0;A;S0)R(s;A;S0)
R(S;A;S0)]; (23)
whereSandS0are random variables independently sampled from DS, andAindependently sampled
fromDA.
Then:
DS;DA(R+)DS;DA(R) = DS;DA(): (24)
Now:
LHS,kDS;DA(R+)DS;DA(R)k1(25)
= max
s;s02SjE[
(s0;A;S0)(s;A;S0)
(S;A;S0)]j (26)
= max
s;s02Sj(
I[x=s0]DA(u)DS(x0) (27)
I[x=s]DA(u)DS(x0)
DS(x)DA(u)DS(x0))j (28)
= max
s;s02SjDA(u)DS(x0) (
I[x=s0]I[x=s]
DS(x))j (29)
=(1 +
DS(x))DA(u)DS(x0); (30)
where the ﬁnal step follows by substituting s=xands06=x(usingjSj 2).
Lemma 4.5. The Pearson distance Dis a pseudometric. Moreover, let a;b2(0;1),c;d2Rand
X;Y be random variables. Then it follows that 0D(aX+c;bY +d) =D(X;Y )1.
Proof. For a non-constant random variable V, deﬁne a standardized (zero mean and unit variance)
version:
Z(V) =VE[V]r
Eh
(VE[V])2i: (31)
The Pearson correlation coefﬁcient on random variables AandBis equal to the expected product of
these standardized random variables:
(A;B) =E[Z(A)Z(B)]: (32)
LetW,X,Ybe random variables.
Identity . Have(X;X ) = 1 , soD(X;X ) = 0 .
Symmetry . Have(X;Y ) =(Y;X)by commutativity of multiplication, so D(X;Y ) =
D(Y;X).
Triangle Inequality . For any random variables A;B :
Eh
(Z(A)Z(B))2i
=E
Z(A)22Z(A)Z(B) +Z(B)2
(33)
=E
Z(A)2
+E
Z(B)2
2E[Z(A)Z(B)] (34)
= 22E[Z(A)Z(B)] (35)
= 2 (1(A;B)) (36)
= 4D(A;B)2: (37)
26
Published as a conference paper at ICLR 2021
So:
4D(W;Y )2=Eh
(Z(W)Z(Y))2i
(38)
=Eh
(Z(W)Z(X) +Z(X)Z(Y))2i
(39)
=Eh
(Z(W)Z(X))2i
+Eh
(Z(X)Z(Y))2i
(40)
+ 2E[(Z(W)Z(X)) (Z(X)Z(Y))]
= 4D(W;X )2+ 4D(X;Y )2+ 8E[(Z(W)Z(X)) (Z(X)Z(Y))]:(41)
SincehA;Bi=E[AB]is an inner product over R, it follows by the Cauchy-Schwarz inequality that
E[AB]p
E[A2]E[B2]. So:
D(W;Y )2D(W;X )2+D(X;Y )2+ 2D(W;X )D(X;Y ) (42)
= (D(W;X ) +D(X;Y ))2: (43)
Taking the square root of both sides:
D(W;Y )D(W;X ) +D(X;Y ); (44)
as required.
Positive Afﬁne Invariant and Bounded D(aX+c;bY +d) =D(X;Y )is immediate from
(X;Y )invariant to positive afﬁne transformations. Have 1(X;Y )1, so01(X;Y )
2thus0D(X;Y )1.
Theorem 4.7. The Equivalent-Policy Invariant Comparison distance is a pseudometric.
Proof. The result follows from Dbeing a pseudometric. Let RA,RBandRCbe reward functions
mapping from transitions SAS to real numbers R.
Identity . Have:
DEPIC(RA;RA) =D(CDS;DA(RA) (S;A;S0);CDS;DA(RA) (S;A;S0)) = 0; (45)
sinceD(X;X ) = 0 .
Symmetry . Have:
DEPIC(RA;RB) =D(CDS;DA(RA) (S;A;S0);CDS;DA(RB) (S;A;S0)) (46)
=D(CDS;DA(RB) (S;A;S0);CDS;DA(RA) (S;A;S0)) (47)
=DEPIC(RB;RA); (48)
sinceD(X;Y ) =D(Y;X).
Triangle Inequality . Have:
DEPIC(RA;RC) =D(CDS;DA(RA) (S;A;S0);CDS;DA(RC) (S;A;S0)) (49)
D(CDS;DA(RA) (S;A;S0);CDS;DA(RB) (S;A;S0)) (50)
+D(CDS;DA(RB) (S;A;S0);CDS;DA(RC) (S;A;S0)) (51)
=DEPIC(RA;RB) +DEPIC(RB;RC); (52)
sinceD(X;Z)D(X;Y ) +D(Y;Z).
Theorem 4.8. LetRA,R0
A,RB;R0
B:SAS! Rbe reward functions such that R0
ARA
andR0
BRB. Then 0DEPIC(R0
A;R0
B) =DEPIC(RA;RB)1.
Proof. SinceDEPIC is deﬁned in terms of D, the bounds 0DEPIC(R0
A;R0
B)and
DEPIC(RA;RB)1are immediate from the bounds in lemma 4.5.
SinceR0
ARAandR0
BRB, we can write for X2fA;Bg:
R
X(s;a;s0) =XRX(s;a;s0); (53)
R0
X(s;a;s0) =R
X(s;a;s0) +
X(s0)X(s); (54)
27
Published as a conference paper at ICLR 2021
for some scaling factor X>0and potential function X:S!R.
By proposition 4.2:
CDS;DA(R0
X) =CDS;DA
R
X
: (55)
Moreover, since CDS;DA(R)is deﬁned as an expectation over Rand expectations are linear:
CDS;DA
R
X
=XCDS;DA(RX): (56)
Unrolling the deﬁnition of DEPIC and applying this result gives:
DEPIC(R0
A;R0
B) =D(CDS;DA(R0
A) (S;A;S0);CDS;DA(R0
B) (S;A;S0)) (57)
=D(ACDS;DA(RA) (S;A;S0);BCDS;DA(RB) (S;A;S0))eqs. 55 and 56
=D(CDS;DA(RA) (S;A;S0);CDS;DA(RB) (S;A;S0)) lemma 4.5
=DEPIC(RA;RB):
A.3.3 N EAREST POINT IN EQUIVALENCE CLASS (NPEC) PREMETRIC
Proposition A.1. (1)DLp;Dis a metric in Lpspace, where functions fandgare identiﬁed if they
agree almost everywhere on D. (2)DLp;Dis a pseudometric if functions are identiﬁed only if they
agree at all points.
Proof. (1)DLp;Dis a metric in the Lpspace since Lpis a norm in the Lpspace, and
d(x;y) =kxykis always a metric. (2) As f=gat all points implies f=galmost every-
where, certainly DLp;D(R;R) = 0 . Symmetry and triangle inequality do not depend on identity so
still hold.
Proposition A.2 (Properties of DU
NPEC ).LetRA;RB;RC:SAS! Rbe bounded reward
functions, and 0. ThenDU
NPEC :
•Is invariant under in source :
DU
NPEC (RA;RC) =DU
NPEC (RB;RC)ifRARB.
•Invariant under scale-preserving in target :
DU
NPEC (RA;RB) =DU
NPEC (RA;RC)ifRBRCZero .
•Scalable in target :
DU
NPEC (RA;RB) =DU
NPEC (RA;RB).
•Bounded :
DU
NPEC (RA;RB)DU
NPEC (Zero;RB).
Proof. We will show each case in turn.
Invariance under in source
IfRARB, then:
DU
NPEC (RA;RC),inf
RRADLp;D(R;RC) (58)
= inf
RRBDLp;D(R;RC) (59)
,DU
NPEC (RB;RC); (60)
(61)
sinceRRAif and only if RRBasis an equivalence relation.
Invariance under scale-preserving in target
IfRBRCZero , then we can write RB(s;a;s0)RC(s;a;s0) =
(s0)(s)for some
potential function  :S!R. Deﬁnef(R)(s;a;s0) =R(s;a;s0)
(s0) + (s). Then for any
28
Published as a conference paper at ICLR 2021
reward function R:SAS! R:
DLp;D(R;RB),
E
s;a;s0D
jR(s;a;s0)RB(s;a;s0)jp1=p
=
E
s;a;s0D
jR(s;a;s0)(RC(s;a;s0) +
(s0)(s))jp1=p
=
E
s;a;s0D
j(R(s;a;s0)
(s0) + (s))RC(s;a;s0)jp1=p
=
E
s;a;s0D
jf(R)(s;a;s0)RC(s;a;s0)jp1=p
,DLp;D(f(R);RC); (62)
Crucially, note f(R)is a bijection on the equivalence class [R]. Now, substituting this into the
expression for the NPEC premetric:
DU
NPEC (RA;RB),inf
RRADLp;D(R;RB)
= inf
RRADLp;D(f(R);RC) eq. 62
= inf
f(R)RADLp;D(f(R);RC)fbijection on [R]
= inf
RRADLp;D(R;RC) fbijection on [R]
,DU
NPEC (RA;RC): (63)
Scalable in target
First, note that DLp;Dis absolutely scalable in both arguments:
DLp;D(RA;RB),
E
s;a;s0D
jRA(s;a;s0)RB(s;a;s0)jp1=p
=
E
s;a;s0D
jjpjRA(s;a;s0)RB(s;a;s0)jp1=p
jjabsolutely scalable
=
jjpE
s;a;s0D
jRA(s;a;s0)RB(s;a;s0)jp1=p
Elinear
=jj
E
s;a;s0D
jRA(s;a;s0)RB(s;a;s0)jp1=p
,jjDLp;D(RA;RB): (64)
Now, for>0, applying this to DU
NPEC :
DU
NPEC (RA;RB),inf
RRADLp;D(R;RB) (65)
= inf
RRADLp;D(R;RB) RR (66)
= inf
RRADLp;D(R;RB) eq. 64 (67)
=inf
RRADLp;D(R;RB) (68)
,DU
NPEC (RA;RB): (69)
29
Published as a conference paper at ICLR 2021
In the case= 0, then:
DU
NPEC (RA;Zero ),inf
RRADLp;D(R;Zero ) (70)
= inf
RRADLp;D1
2R;Zero
R1
2R (71)
= inf
RRA1
2DLp;D(R;Zero ) (72)
=1
2inf
RRADLp;D(R;Zero ) (73)
=1
2DU
NPEC (RA;Zero ): (74)
Rearranging, we have:
DU
NPEC (RA;Zero ) = 0: (75)
Bounded
Letd,DNPEC (Zero;RB). Then for any >0, there exists some potential function  :S!R
such that the Lpdistance of the potential shaping R(s;a;s0),
(s)(s)fromRBsatisﬁes:
DLp;D(R;RB)d+: (76)
Let2[0;1]. Deﬁne:
R0
(s;a;s0),RA(s;a;s0) +R(s;a;s0): (77)
Now:
DLp;D(R0
;R),
E
s;a;s0D
jR0
(s;a;s0)R(s;a;s0)jp1=p
(78)
=
E
s;a;s0D
jRA(s;a;s0)jp1=p
(79)
=
jjpE
s;a;s0D
jRA(s;a;s0)jp1=p
(80)
=jj
E
s;a;s0D
jRA(s;a;s0)jp1=p
(81)
=jjDLp;D(RA;Zero ): (82)
SinceRAis bounded,DLp;D(RA;Zero )must be ﬁnite, so:
lim
!0+DLp;D(R0
;R) = 0: (83)
It follows that for any >0there exists some >0such that:
DLp;D(R0
;R): (84)
Note thatRAR0
for all>0. So:
DNPEC (RA;RB)DLp;D(R0
;RB) (85)
DLp;D(R0
;R) +DLp;D(R;RB) prop. A.1 (86)
+ (d+) eq. 76 and eq. 84 (87)
=d+ 2: (88)
Since>0can be made arbitrarily small, it follows:
DNPEC (RA;RB)d,DNPEC (Zero;RB); (89)
completing the proof.
30
Published as a conference paper at ICLR 2021
Theorem 5.4. DNPEC is a premetric on the space of bounded reward functions. Moreover, let
RA;RA0;RB;RB0:SAS ! Rbe bounded reward functions such that RARA0and
RBRB0. Then 0DNPEC (RA0;RB0) =DNPEC (RA;RB)1.
Proof. We will ﬁrst prove DNPEC is a premetric, and then prove it is invariant and bounded.
Premetric
First, we will show that DNPEC is a premetric.
Respects identity: DNPEC (RA;RA) = 0
IfDU
NPEC (Zero;RA) = 0 thenDNPEC (RA;RA) = 0 as required. Suppose from now on that
DU
NPEC (RA;RA)6= 0. It follows from prop A.1 that DLp;D(RA;RA) = 0 . SinceXX,0is an
upper bound for DU
NPEC (RA;RA). By prop A.1 DLp;Dis non-negative, so this is also a lower bound
forDU
NPEC (RA;RA). SoDU
NPEC (RA;RA) = 0 and:
DNPEC (RA;RA) =DU
NPEC (RA;RA)
DU
NPEC (Zero;RA)=0
DU
NPEC (Zero;RA)= 0: (90)
Well-deﬁned: DNPEC (RA;RB)0
By prop A.1, it follows that DLp;D(R;RB)0for all reward functions R:SAS . Thus
0is a lower bound for fDLp;D(R;RB)jR:SASg , and thus certainly a lower bound for
fDLp;D(R;Y)jRXgfor any reward function X. Since the inﬁmum is the greatest lower bound,
it follows that for any reward function X:
DU
NPEC (X;RB),inf
RXDLp;D(R;RB)0: (91)
In the case that DU
NPEC (Zero;RB) = 0 , thenDNPEC (RA;RB) = 0 which is non-negative. From
now on, suppose that DU
NPEC (Zero;RB)6= 0. The quotient of a non-negative value with a positive
value is non-negative, so:
DNPEC (RA;RB) =DU
NPEC (RA;RB)
DU
NPEC (Zero;RB)0: (92)
Invariant and Bounded
SinceR0
BRB, we haveR0
BRBZero for some >0. By proposition A.2, DU
NPEC is
invariant under scale-preserving in target and scalable in target. That is, for any reward R:
DU
NPEC (R;R0
B) =DU
NPEC (R;RB) =DU
NPEC (R;RB): (93)
In particular, DU
NPEC (Zero;RB0) =DU
NPEC (Zero;RB). As > 0, it follows that
DU
NPEC (Zero;RB0) = 0()DU
NPEC (Zero;RB) = 0 .
SupposeDU
NPEC (Zero;RB) = 0 . ThenDNPEC (R;RB) = 0 =DNPEC (R;RB0)for any reward
R, so the result trivially holds. From now on, suppose DU
NPEC (Zero;RB)6= 0.
By proposition A.2, DU
NPEC is invariant toin source. That is, DU
NPEC (RA;RB) =
DU
NPEC (R0
A;RB), so:
DNPEC (R0
A;RB) =DU
NPEC (R0
A;RB)
DU
NPEC (Zero;RB)=DU
NPEC (RA;RB)
DU
NPEC (Zero;RB)=DNPEC (RA;RB):(94)
By eq. (93):
DNPEC (RA;R0
B) =DU
NPEC (RA;RB)
DU
NPEC (Zero;RB)=DU
NPEC (RA;RB)
DU
NPEC (Zero;RB)=DNPEC (RA;RB):(95)
SinceDNPEC is a premetric it is non-negative. By the boundedness property of proposition A.2,
DU
NPEC (R;RB)DU
NPEC (Zero;RB), so:
DNPEC (RA;RB) =DU
NPEC (RA;RB)
DU
NPEC (Zero;RB)1; (96)
which completes the proof.
31
Published as a conference paper at ICLR 2021
Note whenDLp;Dis a metric, then DNPEC (X;Y ) = 0 if and only if X=Y.
Proposition A.3. DNPEC isnotsymmetric in the undiscounted case.
Proof. We will provide a counterexample showing that DNPEC is not symmetric.
Choose the state space Sto be binaryf0;1gand the actionsAto be the singleton f0g. Choose the
coverage distribution Dto be uniform on s0!sfors2S. Take
= 1, i.e. undiscounted. Note that
as the successor state is always the same as the start state, potential shaping has no effect on DLp;D,
so WLOG we will assume potential shaping is always zero.
Now, takeRA(s) = 2sandRB(s) = 1 . Takep= 1 for theLpdistance. Observe that
DLp;D(Zero;RA) =1
2(j0j+j2j) = 1 andDLp;D(Zero;RB) =1
2(j1j+j1j) = 1 . Since poten-
tial shaping has no effect, DU
NPEC (Zero;R) =DLp;D(Zero;R)and soDNPEC (Zero;RA) = 1
andDNPEC (Zero;RB) = 1 .
Now:
DU
NPEC (RA;RB) = inf
>0DLp;D(RA;RB) (97)
= inf
>01
2(j1j+j21j) (98)
=1
2; (99)
with the inﬁmum attained at =1
2. But:
DU
NPEC (RB;RA) = inf
>0DLp;D(RB;RA) (100)
= inf
>01
2f() (101)
=1
2inf
>0f(); (102)
where:
f() =jj+j2j; > 0: (103)
Note that:
f() =22(0;2];
222(2;1):(104)
Sof()2on all of its domain, thus:
DU
NPEC (RB;RA) = 1: (105)
Consequently:
DNPEC (RA;RB) =1
26= 1 =DNPEC (RB;RA); (106)
soDNPEC is not symmetric.
A.4 F ULL NORMALIZATION VARIANT OF EPIC
Previously, we used the Pearson distance Dto compare the canonicalized rewards. Pearson distance
is naturally invariant to scaling. An alternative is to explicitly normalize the canonicalized rewards,
and then compare them using any metric over functions.
Deﬁnition A.4 (Normalized Reward) .LetR:SAS! Rbe a bounded reward function. Let
kkbe a norm on the vector space of reward functions over the real ﬁeld. Then the normalized Ris:
RN(s;a;s0) =R(s;a;s0)
kRk(107)
Note that (R)N=RNfor any>0as norms are absolutely homogeneous.
We say a reward is standardized if it has been canonicalized and then normalized.
32
Published as a conference paper at ICLR 2021
Deﬁnition A.5 (Standardized Reward) .LetR:SAS! Rbe a bounded reward function.
Then the standardized Ris:
RS= (CDS;DA(R))N: (108)
Now, we can deﬁne a pseudometric based on the direct distance between the standardized rewards.
Deﬁnition A.6 (Direct Distance Standardized Reward) .LetDbe some coverage distribution over
transitionssa!s0. LetDSandDAbe some distributions over states SandArespectively. Let
S;A;S0be random variables jointly sampled from D. The Direct Distance Standardized Reward
pseudometric between two reward functions RAandRBis theLpdistance between their standardized
versions overD:
DDDSR (RA;RB) =1
2DLp;D
RS
A(S;A;S0);RS
B(S;A;S0)
; (109)
where the normalization step, RN, uses theLpnorm.
For brevity, we omit the proof that DDDSR is a pseudometric, but this follows from DLp;Dbeing
a pseudometric in a similar fashion to theorem 4.7. Note it additionally is invariant to equivalence
classes, similarly to EPIC.
Theorem A.7. LetRA,RA0,RBandRB0be reward functions mapping from transitions SAS
to real numbers Rsuch thatRARA0andRBRB0. Then:
0DDDSR (R0
A;R0
B) =DDDSR (RA;RB)1: (110)
Proof. The invariance under the equivalence class follows from RSbeing invariant to potential
shaping and scale in R. The non-negativity follows from DLp;Dbeing a pseudometric. The upper
bound follows from the rewards being normalized to norm 1and the triangle inequality:
DDDSR (RA;RB) =1
2kRS
ARS
Bk (111)
1
2
kRS
Ak+kRS
Bk
(112)
=1
2(1 + 1) (113)
= 1:
Since both DDSR and EPIC are pseudometrics and invariant on equivalent rewards, it is interesting
to consider the connection between them. In fact, under the L2norm, then DDSR recovers EPIC.
First, we will show that canonical shaping centers the reward functions.
Lemma A.8 (The Canonically Shaped Reward is Mean Zero) .LetRbe a reward function mapping
from transitionsSAS to real numbers R. Then:
E[CDS;DA(R) (S;A;S0)] = 0: (114)
Proof. LetX,UandX0be random variables that are independent of S,AandS0but identically
distributed.
LHS,E[CDS;DA(R) (S;A;S0)] (115)
=E[R(S;A;S0) +
R(S0;U;X0)R(S;U;X0)
R(X;U;X0)] (116)
=E[R(S;A;S0)] +
E[R(S0;U;X0)]E[R(S;U;X0)]
E[R(X;U;X0)] (117)
=E[R(S;U;X0)] +
E[R(X;U;X0)]E[R(S;U;X0)]
E[R(X;U;X0)] (118)
= 0; (119)
where the penultimate step follows since Ais identically distributed to U, andS0is identically
distributed to X0and therefore to X.
Theorem A.9. DDDSR withp= 2 is equivalent to DEPIC . LetRAandRBbe reward functions
mapping from transitions SAS to real numbers R. Then:
DDDSR (RA;RB) =DEPIC(RA;RB): (120)
33
Published as a conference paper at ICLR 2021
Proof. Recall from the proof of lemma 4.5 that:
D(U;V) =1
2r
Eh
(Z(U)Z(V))2i
(121)
=1
2kZ(U)Z(V)k2; (122)
wherekk2is theL2norm (treating the random variables as functions on a measure space) and
Z(U)is a centered (zero-mean) and rescaled (unit variance) random variable. By lemma A.8, the
canonically shaped reward functions are already centered under the joint distribution DSDADS,
and normalization by the L2norm also ensures they have unit variance. Consequently:
DEPIC(RA;RB) =D(CDS;DA(RA) (S;A;S0);CDS;DA(RB) (S;A;S0)) (123)
=1
2

(CDS;DA(RA) (S;A;S0))N(CDS;DA(RB) (S;A;S0))N

2(124)
=1
2

RS
A(S;A;S0)RS
B(S;A;S0)

2(125)
=1
2DLp;D
RS
A(S;A;S0);RS
B(S;A;S0)
(126)
=DDDSR (RA;RB); (127)
completing the proof.
A.5 R EGRET BOUND
In this section, we derive an upper bound on the regret in terms of the EPIC distance. Speciﬁ-
cally, given two reward functions RAandRBwith optimal policies
Aand
B, we show that the
regret (under reward RA) of using policy
Binstead of a policy
Ais bounded by a function of
DEPIC(RA;RB). First, in section A.5.1 we derive a bound for MDPs with ﬁnite state and action
spaces. In section A.6 we then present an alternative bound for MDPs with arbitrary state and action
spaces and Lipschitz reward functions. Finally, in section A.7 we show that in both cases the regret
tends to 0asDEPIC(RA;RB)!0.
A.5.1 D ISCRETE MDP S
We start in lemma A.10 by showing that L2distance upper bounds L1distance. Next, in lemma A.11
we show regret is bounded by the L1distance between reward functions using an argument similar
to [21]. Then in lemma A.12 we relate regret bounds for standardized rewards RSto the original
rewardR. Finally, in theorem 4.9 we use section A.4 to express DEPIC in terms of the L2distance
on standardized rewards, deriving a bound on regret in terms of the EPIC distance.
Lemma A.10. Let(
;F;P)be a probability space and f:
!Ra measurable function whose
absolute value raised to the n-th power for n2f1;2ghas a ﬁnite expectation. Then the L1norm of
fis bounded above by the L2norm:
kfk1kfk2: (128)
Proof. LetXbe a random variable sampled from P, and consider the variance of f(X):
Eh
(jf(X)jE[jf(X)j])2i
=Eh
jf(X)j22jf(X)jE[jf(X)j] +E[jf(X)j]2i
(129)
=E
jf(X)j2
2E[jf(X)j]E[jf(X)j] +E[jf(X)j]2(130)
=E
jf(X)j2
E[jf(X)j]2(131)
0: (132)
Rearranging terms, we have
kfk2
2=E
jf(X)j2
E[jf(X)j]2=kfk2
1: (133)
Taking the square root of both sides gives:
kfk1kfk2; (134)
as required.
34
Published as a conference paper at ICLR 2021
Lemma A.11. LetMbe an MDP\R with ﬁnite state and action spaces SandA. LetRA;RB:
SAS! Rbe rewards. Let
Aand
Bbe policies optimal for rewards RAandRBinM. Let
D(t;st;at;st+1)denote the distribution over trajectories that policy induces inMat time stept.
LetD(s;a;s0)be the (stationary) coverage distribution over transitions SAS used to compute
DEPIC . Suppose that there exists some K > 0such thatKD(st;at;s0t+1)D(t;st;at;s0t+1)
for all time steps t2N, triplesst;at;st+12SAS and policies 2f
A;
Bg. Then the
regret under RAfrom executing
Boptimal forRBinstead of
Ais at most:
GRA(
A)GRA(
B)2K
1
DL1;D(RA;RB): (135)
Proof. NotingGRA()is maximized when =
A, it is immediate that
GRA(
A)GRA(
B) =jGRA(
A)GRA(
B)j (136)
=j(GRA(
A)GRB(
B)) + (GRB(
B)GRA(
B))j (137)
jGRA(
A)GRB(
B)j+jGRB(
B)GRA(
B)j: (138)
We will show that both these terms are bounded above byK
1
DL1;D(RA;RB), from which the
result follows.
First, we will show that for policy 2f
A;
Bg:
jGRA()GRB()jK
1
DL1;D(RA;RB): (139)
LetTbe the horizon of M. This may be inﬁnite ( T=1) when
 <1; note sinceSAS is
bounded, so are RA;RBso the discounted inﬁnite returns GRA();GRB()converge (as do their
differences). Writing = (s0;a0;s1;a1;), we have for any policy :
,jGRA()GRB()j (140)
=ED"TX
t=0
t(RA(st;at;st+1)RB(st;at;st+1))#(141)
E
D"TX
t=0
tjRA(st;at;st+1)RB(st;at;st+1)j#
(142)
=TX
t=0
tE
st;at;st+1D[jRA(st;at;st+1)RB(st;at;st+1)j] (143)
=TX
t=0
tX
st;at;st+12SASD(t;st;at;st+1)jRA(st;at;st+1)RB(st;at;st+1)j:(144)
Let2f
A;
Bg. By assumption,D(t;st;at;s0t+1)KD(st;at;s0t+1), so:
KTX
t=0
tX
st;at;st+12SASD(st;at;st+1)jRA(st;at;st+1)RB(st;at;st+1)j(145)
=KTX
t=0
tDL1;D(RA;RB) (146)
=K
1
DL1;D(RA;RB); (147)
as required.
In particular, substituting =
Bgives:
jGRB(
B)GRA(
B)j=jGRA(
B)GRB(
B)jK
1
DL1;D(RA;RB): (148)
35
Published as a conference paper at ICLR 2021
Rearranging gives:
GRA(
B)GRB(
B)K
1
DL1;D(RA;RB): (149)
So certainly:
GRA(
A) = max
GRA()GRB(
B)K
1
DL1;D(RA;RB): (150)
By a symmetric argument, substituting =
Agives:
GRB(
B) = max
GRB()GRA(
A)K
1
DL1;D(RA;RB): (151)
Eqs. 150 and 151 respectively give GRB(
B)GRA(
A)K
1
andGRA(
A)GRB(
B)K
1
.
Combining these gives:
jGRA(
A)GRB(
B)jK
1
DL1;D(RA;RB): (152)
Substituting inequalities 148 and 152 into eq. 138 yields the required result.
Note that ifD=Dunif, uniform overSAS , thenKjSj2jAj.
Lemma A.12. LetMbe an MDP\R with state and action spaces SandA. LetRA;RB:SAS!
Rbe bounded rewards. Let
Aand
Bbe policies optimal for rewards RAandRBinM. Suppose
the regret under the standardized reward RS
Afrom executing
Binstead of
Ais upper bounded by
someU2R:
GRS
A(
A)GRS
A(
B)U: (153)
Then the regret under the original reward RAis bounded by:
GRA(
A)GRA(B)4UkRAk2: (154)
Proof. Recall that
RS=CDS;DA(R)
kCDS;DA(R)k2; (155)
whereCDS;DA(R)is simplyRshaped with some (bounded) potential . It follows that:
GRS() =1
kCDS;DA(R)k2GCDS;DA(R)() (156)
=1
kCDS;DA(R)k2(GR()Es0d0[(s0)]); (157)
wheres0depends only on the initial state distribution d0.†Sinces0does not depend on , the terms
cancel when taking the difference in returns:
GRS
A(
A)GRS
A(
B) =1
kCDS;DA(RA)k2(GRA(
A)GRA(
B)): (158)
Combining this with eq 153 gives
GRA(
A)GRA(
B)UkCDS;DA(RA)k2: (159)
Finally, we will bound kCDS;DA(RA)k2in terms ofkRAk2, completing the proof. Recall:
CDS;DA(R) (s;a;s0) =R(s;a;s0) +E[
R(s0;A;S0)R(s;A;S0)
R(S;A;S0)];(160)
whereSandS0are random variables independently sampled from DSandAsampled fromDA. By
the triangle inequality on the L2norm and linearity of expectations, we have:
kCDS;DA(R)k2kRk2+
kfk2+kgk2+
jcj; (161)
†In the ﬁnite-horizon case, there is also a term
T(sT), wheresTis the ﬁxed terminal state. Since
sTis ﬁxed, it also cancels in eq. 158. This term can be neglected in the discounted inﬁnite-horizon case as

T(sT)!0asT!1 for any bounded .
36
Published as a conference paper at ICLR 2021
wheref(s;a;s0) =E[R(s0;A;S0)],g(s;a;s0) =E[R(s;A;S0)]andc=E[R(S;A;S0)]. Letting
X0be a random variable sampled from DSindependently from SandS0, have
kfk2
2=EX0h
E[R(X0;A;S0)]2i
(162)
EX0
E
R(X0;A;S0)2
(163)
=E
R(X0;A;S0)2
(164)
=kRk2
2: (165)
Sokfk2kRk2and, by an analogous argument, kgk2kRk2. Similarly
jcj=jE[R(S;A;S0)]j (166)
E[jR(S;A;S0)j] (167)
=kRk1 (168)
kRk2 lemma A.10 : (169)
Combining these results, we have
kCDS;DA(R)k24kRk2: (170)
Substituting eq. 170 into eq. 159 yields:
GRA(
A)GRA(
B)4UkRAk2; (171)
as required.
Theorem 4.9. LetMbe a
-discounted MDP\R with ﬁnite state and action spaces SandA.
LetRA;RB:SAS ! Rbe rewards, and
A;
Bbe respective optimal policies. Let
D(t;st;at;st+1)denote the distribution over transitions SAS induced by policy at timet,
andD(s;a;s0)be the coverage distribution used to compute DEPIC . Suppose there exists K > 0such
thatKD(st;at;st+1)D(t;st;at;st+1)for all times t2N, triples (st;at;st+1)2SAS
and policies 2f
A;
Bg. Then the regret under RAfrom executing
Binstead of
Ais at most
GRA(
A)GRA(
B)16KkRAk2(1
)1DEPIC(RA;RB);
whereGR()is the return of policy under reward R.
Proof. Recall from section A.4 that:
DEPIC(RA;RB) =1
2

RS
A(S;A;S0)RS
B(S;A;S0)

2: (172)
Applying lemma A.10 we obtain:
DL1;D(RS
A;RS
B) =

RS
A(S;A;S0)RS
B(S;A;S0)

12DEPIC(RA;RB): (173)
Note that
Ais optimal for RS
Aand
Bis optimal for RS
Bsince the set of optimal policies for RSis
the same as for R. Applying lemma A.11 and eq. 173 gives
GRS
A(
A)GRS
A(
B)2K
1
DL1;D(RS
A;RS
B)4K
1
DEPIC(RA;RB): (174)
SinceSAS is bounded,RAandRBmust be bounded, so we can apply lemma A.12, giving:
GRA(
A)GRA(
B)16KkRAk2
1
DEPIC(RA;RB); (175)
completing the proof.
37
Published as a conference paper at ICLR 2021
A.6 L IPSCHITZ REWARD FUNCTIONS
In this section, we generalize the previous results to MDPs with continuous state and action spaces.
The challenge is that even though the spaces may be continuous, the distribution Dinduced by an
optimal policy may only have support on some measure zero set of transitions B. However, the
expectation over a continuous distribution Dis unaffected by the reward at any measure zero subset
of points. Accordingly, the reward can be varied arbitrarily on transitions B– causing arbitrarily
small or large regret – while leaving the EPIC distance ﬁxed. To rule out this pathological case, we
assume the rewards are Lipschitz smooth. This guarantees that if the expected difference between
rewards is small on a given region, then all points in this region have bounded reward difference.
We start by deﬁning a relaxation of the Wasserstein distance Win deﬁnition A.13. In lemma A.14
we then bound the expected value under distribution in terms of the expected value under alternative
distributionplusW(;). Next, in lemma A.15 we bound the regret in terms of the L1distance
between the rewards plus W; this is analogous to lemma A.11 in the ﬁnite case. Finally, in
theorem A.16 we use the previous results to bound the regret in terms of the EPIC distance plus W.
Deﬁnition A.13. LetSbe some set and let ;be probability measures on Swith ﬁnite ﬁrst moment.
We deﬁne the relaxed Wasserstein distance betweenandby:
W(;), inf
p2(;)Z
kxykdp(x;y); (176)
where (;)is the set of probability measures on SSsatisfying for all x;y2S:
Z
Sp(x;y)dy=(x); (177)
Z
Sp(x;y)dx(y): (178)
Note thatW1is equal to the (unrelaxed) Wasserstein distance (in the `1norm).
Lemma A.14. LetSbe some set and let ;be probability measures on S. Letf:S!Rbe an
L-Lipschitz function on the `1normkk1. Then, for any 1:
EX[jf(X)j]EY[jf(Y)j] +LW(;): (179)
Proof. Letp2(;). Then:
EX[jf(X)j],Z
jf(x)jd(x) deﬁnition of E(180)
=Z
jf(x)jdp(x;y) is a marginal of p(181)
Z
jf(y)j+Lkxykdp(x;y) f L-Lipschitz (182)
=Z
jf(y)jdp(x;y) +LZ
kxykdp(x;y) (183)
=Z
jf(y)jZ
p(x;y)dxdy +LZ
kxykdp(x;y) (184)
Z
jf(y)j(y)dy+LZ
kxykdp(x;y) eq. 178 (185)
=EY[jf(Y)j] +LZ
kxykdp(x;y) deﬁnition of E:(186)
Since this holds for all choices of p, we can take the inﬁmum of both sides, giving:
EX[jf(X)j]EY[jf(Y)j] +L inf
p2(;)Z
kxykdp(x;y) (187)
=EY[jf(Y)j] +LW(;):
38
Published as a conference paper at ICLR 2021
Lemma A.15. LetMbe an MDP\R with state and action spaces SandA. LetRA;RB:SAS!
RbeL-Lipschitz, bounded rewards on the `1normkk1. Let
Aand
Bbe policies optimal for
rewardsRAandRBinM. LetD;t(st;at;st+1)denote the distribution over trajectories that policy
induces inMat time stept. LetD(s;a;s0)be the (stationary) coverage distribution over transitions
SAS used to compute DEPIC . Let1, and letB(t) = max2f
A;
BgW(D;t;D).
Then the regret under RAfrom executing
Boptimal forRBinstead of
Ais at most:
GRA(
A)GRA(
B)2
1
DL1;D(RA;RB) + 4L1X
t=0
tB(t): (188)
Proof. By the same argument as lemma A.11 up to eq. 144, we have for any policy :
jGRA()GRB()j1X
t=0
tDL1;D;t(RA;RB): (189)
Letf(s;a;s0) =RA(s;a;s0)RB(s;a;s0), and notefis2L-Lipschitz and bounded since RAand
RBare bothL-Lipschitz and bounded. Now, by lemma A.14, letting =D;tand=D, we have:
DL1;D;t(RA;RB)DL1;D(RA;RB) + 2LW(D;t;D): (190)
So, for2f
A;
Bg, it follows that
jGRA()GRB()j
1
DL1;D(RA;RB) + 2L1X
t=0
tB(t): (191)
By the same argument as for eq. 148 to 152 in lemma A.11, it follows that
GRA(
A)GRA(
B)2
1
DL1;D(RA;RB) + 4L1X
t=0
tB(t); (192)
completing the proof.
Theorem A.16. LetMbe an MDP\R with state and action spaces SandA. LetRA;RB:SA
S!Rbe bounded, L-Lipschitz rewards on the `1normkk1. Let
Aand
Bbe policies optimal for
rewardsRAandRBinM. LetD(t;st;at;st+1)denote the distribution over trajectories that policy
induces inMat time stept. LetD(s;a;s0)be the (stationary) coverage distribution over transitions
SAS used to compute DEPIC . Let1, and letB(t) = max2
A;
BW(D;t;D). Then
the regret under RAfrom executing
Boptimal forRBinstead of
Ais at most:
GRA(
A)GRA(
B)16kRAk2

1
DEPIC(RA;RB) +L1X
t=0
tB(t)!
: (193)
Proof. The proof for theorem 4.9 holds in the general setting up to eq. 173. Applying lemma A.15 to
eq. 173 gives
GRS
A(
A)GRS
A(
B)2
1
DL1;D(RS
A;RS
B) + 4L1X
t=0
tB(t) (194)
4
1
DEPIC(RA;RB) + 4L1X
t=0
tB(t): (195)
Applying lemma A.12 yields
GRA(
A)GRA(
B)16kRAk2

1
DEPIC(RA;RB) +L1X
t=0
tB(t)!
; (196)
as required.
39
Published as a conference paper at ICLR 2021
A.7 L IMITING BEHAVIOR OF REGRET
The regret bound for ﬁnite MDPs, Theorem 4.9, directly implies that, as EPIC distance tends to 0,
the regret also tends to 0. By contrast, our regret bound in theorem A.16 for (possibly continuous)
MDPs with Lipschitz reward functions includes the relaxed Wasserstein distance Was an additive
term. At ﬁrst glance, it might therefore appear possible for the regret to be positive even with a
zero EPIC distance. However, in this section we will show that in fact the regret tends to 0as
DEPIC(RA;RB)!0in the Lipschitz case as well as the ﬁnite case.
We show in lemma A.17 that if the expectation of a non-negative function over a totally bounded
measurable metric space Mtends to zero under one distribution with adequate support, then it also
tends to zero under all other distributions. For example, taking Mto be a hypercube in Euclidean
space with the Lebesque measure satisﬁes these assumptions. We conclude in theorem A.18 by
showing the regret tends to 0as the EPIC distance tends to 0.
Lemma A.17. LetM= (S;d)be a totally bounded metric space, where d(x;y) =kxyk. Let
(S;A; )be a measure space on Swith the Borel -algebraAand measure . Letp;q2(S)be
probability density functions on S. Let >0such thatp(s)for alls2S. Letfn:S!R
be a sequence of L-Lipschitz functions on norm kk. Suppose limn!1EXp[jfn(X)j] = 0 . Then
limn!1EYq[jfn(Y)j] = 0 .
Proof. SinceMis totally bounded, for each r>0there exists a ﬁnite collection of open balls in S
of radiusrwhose union contains M. LetBr(c) =fs2Sjksck<rg, the open ball of radius r
centered atc. LetC(r)denote some ﬁnite collection of Q(r)open balls:
C(r) =fBr(cr;n)jn2f1;;Q(r)gg; (197)
such thatS
B2C(r)B=S.
It is possible for some balls Br(cn)to have measure zero, (Br(cn)) = 0 , such as ifScontains an
isolated point cn. DeﬁneP(r)to be the subset of C(r)with positive measure:
P(r) =fB2C(r)j(B)>0g; (198)
and letpr;1;;pr;Q0(r)denote the centers of the balls in P. SinceP(r)is a ﬁnite collection, it must
have a minimum measure:
(r) = min
B2P(B): (199)
Moreover, by construction of P,(r)>0.
LetS0(r)be the union only over balls of positive measure:
S0(r) =[
B2P(r)B: (200)
Now, letD(r) =SnS0(r), comprising the (ﬁnite number of) measure zero balls in C(r). Since mea-
sures are countably additive, it follows that D(r)is itself measure zero: (D(r)) = 0 . Consequently:
Z
Sg(s)d=Z
S0(r)g(s)d; (201)
for any measurable function g:S!R.
Since limn!1EXp[jfn(X)j] = 0 , for allr>0there exists some Nr2Nsuch that for all nNr:
EXp[jfn(X)j]<Lr (r): (202)
By Lipschitz continuity, for any s;s02S:
jfn(s0)jjfn(s)jLks0sk: (203)
In particular, since any point s2S0(r)is at mostrdistance from some ball center pr;i, then
jfn(pr;i)jjfn(s)jLr. So if there exists s2S0(r)such thatjfn(s)j3Lr, then there must
exist a ball center pr;iwithjfn(pr;i)j2Lr. Then for any point s02Br(pr;i):
jfn(s0)jjfn(pr;i)jLrLr: (204)
40
Published as a conference paper at ICLR 2021
Now, we have:
EXp[jfn(X)j],Z
s2Sjfn(s)jp(s)d(s) (205)
=Z
s2S0(r)jfn(s)jp(s)d(s) eq. 201 (206)
Z
s2Br(pr;i)jfn(s)jp(s)d(s) non-negativity ofjfn(s)j (207)
Z
s2Br(pr;i)jfn(s)jd(s) p(s) (208)
LrZ
s2Br(pr;i)1d(s) eq. 204 (209)
=Lr (Br(pr;i)) integrating w.r.t.  (210)
Lr (r) (r)minimum of (Br(pr;i)): (211)
But this contradicts eq. 202, and so can only hold if n < Nr. It follows that for all nNrand
s2S0(r), we havejfn(s)j<3Lr, and so in particular:
EYq[jfn(Y)j]<3Lr: (212)
Let>0. Chooser=
3L. Then for all nNr,EYq[jfn(Y)j]<. It follows that:
lim
n!1EYq[jfn(Y)j] = 0; (213)
completing the proof.
Theorem A.18. LetMbe an MDP\R with state and action spaces SandA. LetRA;RB:SA
S!Rbe bounded rewards on some norm kkonSAS . Let
Aand
Bbe policies optimal
for rewards RAandRBinM. LetD(t;st;at;st+1)denote the distribution over trajectories that
policyinduces inMat time step t. LetD(s;a;s0)be the (stationary) coverage distribution over
transitionsSAS used to compute DEPIC .
Suppose that either:
1.Discrete :SandAarediscrete . Moreover, suppose that there exists some K > 0such
thatKD(st;at;s0t+1)D(t;st;at;s0t+1)for all time steps t2N, triplesst;at;st+12
SAS and policies 2f
A;
Bg.
2.Lipschitz :(SAS;d)is a totally bounded measurable metric space where d(x;y) =
kxyk. Moreover, RAandRBareL-Lipschitz onkk. Furthermore, suppose there
exists some  > 0such thatD(s;a;s0)for alls;a;s02 SAS , and that
D(t;st;at;st+1)is a non-degenerate probability density function (i.e. no single point has
positive measure).
Then asDEPIC(RA;RB)!0,GRA(
A)GRA(
B)!0.
Proof. In case (1) Discrete , by theorem 4.9:
GRA(
A)GRA(
B)16KkRAk2
1
DEPIC(RA;RB): (214)
Moreover, by optimality of
Awe have 0GRA(
A)GRA(
B). So by the squeeze theorem, as
DEPIC(RA;RB)!0,GRA(
A)GRA(
B)!0.
From now on, suppose we are in case (2) Lipschitz . By the same argument as lemma A.11 up to
eq. 144, we have for any policy :
GRS
A()GRS
B()1X
t=0
tDL1;D;t(RS
A;RS
B): (215)
41
Published as a conference paper at ICLR 2021
Applying lemma A.12 we have:
jGRA()GRB()j4kRAk21X
t=0
tDL1;D;t(RS
A;RS
B): (216)
By equation 173, we know that DL1;D(RS
A;RS
B)!0asDEPIC(RA;RB)!0. By lemma A.17,
we know that DL1;D;t(RS
A;RS
B)!0asDL1;D(RS
A;RS
B)!0. So we can conclude that as
DEPIC(RA;RB)!0:
jGRA()GRB()j!0; (217)
as required.
42

Adam Gleave
Michael Dennis
Shane Legg
Stuart Russell
Jan Leike