Inferring Reward Functions from Demonstrators with Unknown Biases

Under review as a conference paper at ICLR 2019
INFERRING REWARD FUNCTIONS
FROM DEMONSTRATORS WITH UNKNOWN BIASES
Anonymous authors
Paper under double-blind review
ABSTRACT
Our goal is to infer reward functions from demonstrations. In order to infer the
correct reward function, we must account for the systematic ways in which the
demonstrator is suboptimal. Prior work in inverse reinforcement learning can
account for speciﬁc, known biases, but cannot handle demonstrators with unknown
biases. In this work, we explore the idea of learning the demonstrator’s planning
algorithm (including their unknown biases), along with their reward function.
What makes this challenging is that any demonstration could be explained either
by positing a term in the reward function, or by positing a particular systematic
bias. We explore what assumptions are sufﬁcient for avoiding this impossibility
result: either access to tasks with known rewards which enable estimating the
planner separately, or that the demonstrator is sufﬁciently close to optimal that
this can serve as a regularizer. In our exploration with synthetic models of human
biases, we ﬁnd that it is possible to adapt to different biases and perform better
than assuming a ﬁxed model of the demonstrator, such as Boltzmann rationality.
1 I NTRODUCTION
Our ultimate goal is to enable the design of agents that optimize for the right reward function. Unfor-
tunately, designing reward functions is challenging (Amodei et al., 2017) and can have unintended
side-effects (Hadﬁeld-Menell et al., 2017). Inverse Reinforcement Learning (Russell, 1998; Ng et al.,
2000; Abbeel & Ng, 2004) aims to bypass the need for reward design by learning the reward from
observed demonstrations of good behavior.
Existing IRL algorithms typically make the assumption that the demonstrator is either optimal, or
Boltzmann rational, i.e. taking better actions with higher probability (Ziebart et al., 2008; Finn et al.,
2016). However, there is a rich literature showing that humans are notoptimal, and are biased in
systematic ways. Consider a grad student who starts writing a paper a month in advance, expecting
it to take two weeks, but then misses the deadline. Should we infer that they prefer to lose sleep to
pursue a deadline that they then miss? Of course not – this is a classic case of the planning fallacy
(Buehler et al., 1994). It is not clear what exactly went wrong – perhaps the grad student did not have
a good dynamics model (not realizing that Amazon spot instances can be suddenly terminated), or
they failed to account for the uncertainty in their dynamics model (expecting that it wouldn’t happen
to them), or they optimized too much for short term reward (watching Netﬂix), and so on – we do not
know what the bias is. Ideally, even with this unknown bias, we would like our system to infer that
the grad student failed to plan appropriately, and that they would prefer to meet the deadline.
This leads to a tradeoff between the expressivity of our model of the demonstrator, and the ability of
the algorithm to learn the true reward function with few samples. On one extreme, if we know exactly
what bias the demonstrator has, we can account for it to infer the true reward function quickly and
accurately, as in Evans et al. (2016); Evans & Goodman (2015); Zheng et al. (2014); Majumdar et al.
(2017) (myopia and hyperbolic time discounting, sparse noise, and risk-sensitivity respectively). Even
suboptimal trajectories or failures (Shiarlis et al., 2016) can be thought of as a biased demonstrator,
where the bias is the speciﬁc model of failure. Unfortunately, people are complex, and it is unlikely
that we will know exactly which bias a person is displaying, or even the space of possible biases that
people might have. If we are incorrect about the bias, we will have a mis-speciﬁed model, and the
reward function will get garbage values that explain the "noise" resulting from the mis-speciﬁcation
(Steinhardt, 2017; Steinhardt & Evans, 2017). On the other extreme, when the demonstrator could be
1
Under review as a conference paper at ICLR 2019
any function mapping reward functions to policies, it is impossible to learn the true reward function
even with inﬁnite data, because there are always alternative explanations for the observed policy
(Armstrong & Mindermann, 2017; Christiano, 2015). We could consider asking the human to rate
each demonstration (Burchﬁel et al., 2016), but the human could be biased in these ratings as well.
In this paper, we focus on learning rewards when the demonstrator’s bias is unknown . We seek to
avoid impossibility results by introducing a set of assumptions that are weaker than assuming a known
bias, the core of which is that the demonstrator plans their actions similarly in similar tasks. This
leads us to investigate the idea that one could actually learn, rather than assume, the demonstrator’s
planning algorithm across multiple tasks, including any biases they may have.
We analyze the feasibility of this idea in two settings, via two algorithms. In both algorithms, we
can navigate the tradeoff between model expressivity and sample efﬁciency through the choice of
architecture for the differentiable planner we learn, and the space of reward functions to consider.
In the ﬁrst setting, we assume that we have access to the true reward functions in some subset of
tasks – that means we can learn the demonstrator’s planning algorithm on those tasks, after which
we can infer rewards from demonstrations in new tasks. In the more realistic setting where we have
no access to any reward function, we assume that the demonstrator is "close" to optimal. This is
a weaker version of the assumption of Boltzmann rationality that still allows us to learn particular
systematic biases. We operationalize this assumption by initializing the differentiable planner to
mimic the optimal agent, and then ﬁnetuning it to allow it to account for speciﬁc biases. We show that
this initialization is essential for good performance, as we would expect from the impossibility result.
In summary, this paper makes three contributions:
1.Assumptions that are more realistic than the assumption of Boltzmann rationality that allow
us to avoid known pitfalls of reward inference.
2.Algorithms that leverage these assumptions to infer reward functions from suboptimal
demonstrations.
3.An experimental evaluation demonstrating there is hope in the idea of learning better rewards
by learning biases too, despite the theoretical difﬁculty of the problem.
2 E XAMPLES OF BIASES
To put this work in context, we start with some examples of the kind of biases a general algorithm
should be able to capture and account for. While we use these for illustrative purposes, the whole
point of our work is that humans might have systematic suboptimalities completely different from
these examples. We don’t know all the possible biases a priori.
Running Example. We illustrate the effects of these biases on a simple 2D navigation task in ﬁgure 1.
There are multiple salient locations, each of which has a desirability score (which can be negative, in
which case the agent wants to avoid those locations). The agent can move in any of the four cardinal
directions, or stay in its current position. Every movement action has a chance of failing and causing
the agent to move in a direction orthogonal to the one it chose. Despite their simplicity, there are
several ways in which human-like suboptimal behavior can manifest in these environments.
Time inconsistency. Would you prefer to get $100 in 30 days, or $110 in 31 days? Faced with this
question, people typically choose the latter. However, thirty days later, when faced with the choice
of getting $100 now, or $110 tomorrow, they sometimes choose to take the $100. This reversal of
preferences over time would never happen with an optimal agent that maximizes expected sum of
discounted rewards. Researchers model this phenomenon using hyperbolic time discounting , in which
future rewards are discounted more aggressively than exponentially. This leads to a followup question
– how do humans make long-term plans, given that their future self will have different preferences?
Prior work has considered a spectrum from naive agents that assume their future self will have the
same preferences as they do, to sophisticated agents that perfectly understand how their preferences
will change over time and make plans that take such change into account (Frederick et al., 2002).
In ﬁgure 1, when going to a high reward, both the naive andsophisticated hyperbolic time discounters
can be "tempted" by a proximate smaller reward. The naive agent fails to anticipate the temptation,
and so once it gets near the smaller positive reward, it caves in to the temptation and stays there. The
2
Under review as a conference paper at ICLR 2019
Figure 1: The plans of our synthetic agents on two navigation environments. Actual trajectories could differ
due to randomness in the transitions. Green squares indicate positive reward while red squares indicate negative
reward, with darker colors indicating higher magnitude of reward.
sophisticated agent explicitly plans to avoid the temptation – it does not collect the smaller reward
and instead takes a longer, more dangerous path around the smaller reward to get to the large reward.
Incorrect estimates of probabilities. Humans are notoriously bad at judging probabilities. The
availability heuristic (Tversky & Kahneman, 1973) refers to the human tendency to rate events as
more likely if they are easier to recall. The recency effect is a similar effect where recent events are
judged to be more probable. These biases are complicated and depend heavily on context and don’t
obviously transfer to our task. So, we use two simpliﬁed models – an overconﬁdent agent, which
expects that the most likely next state is more likely than it actually is, leading it to take risks, and an
underconﬁdent agent, which is overly cautious. In ﬁgure 1, the overconﬁdent agent takes the shortest
path to the reward, underestimating the risk of slipping into the large region of negative reward, while
the underconﬁdent agent plans a circuitous route around negative reward that it is unlikely to have
actually encountered.
Bounded computation. Researchers have studied models of bounded rationality , where humans are
assumed to be rational subject to the constraint that they have a bounded amount of computation. This
can be thought of as an explanation that many other heuristics and biases are actually computational
shortcuts that allow us to reach reasonably good decisions without too much cost (Kahneman, 2003).
In our task, we model computation bounds as a small time horizon for planning, leading to myopic
behavior. In ﬁgure 1, the myopic agent can only see close rewards, and goes directly to them, never
even realizing the possibility of going to the highest reward.
3 P ROBLEM : LEARNING REWARDS OF DEMONSTRATORS WITH UNKNOWN
BIASES
Notation. A (ﬁnite-horizon) Markov Decision Process (MDP) (Puterman, 2014) is a tuple
hS;A;T;r;Hi.Sis a set of states. Ais a set of actions. Tis a probability distribution over
the next state, given the previous state and action. We write this as T(st+1jst;a).ris a reward
function that maps states and actions to rewards r:SA!R.H2Z+is the ﬁnite planning
horizon for the agent. Since we are interested in the setting where the reward function ris unknown,
we will factor MDPs into world models w=hS;A;T;Hiand reward functions r.
Instead of having access to a reward function, we observe the behavior of an demonstrator, who
performs the task well but may be suboptimal in systematic ways. In particular, we observe the
demonstrator’s policy:S!A.
Estimating Biases and Rewards. Given a world model wand the demonstrator’s policy Dwhich
may exhibit an unknown bias, determine the rewardrthat the demonstrator is optimizing.
We might hope to solve this problem without any additional assumptions. However, this problem is
unsolvable – Armstrong & Mindermann (2017) prove an impossibility result showing that for any
potential reward function r0, there is some planner D0such thatD0(w;r0) =D(w;r). The proof is
simple – simply set D0(w;r0) =for anyr0, that isD0always returns regardless of reward.
3
Under review as a conference paper at ICLR 2019
Inverse reinforcement learning assumes that the demonstrator is (approximately) optimal to get
around this issue. It is common to assume Boltzmann rationality, where the probability of an action is
proportional to the exponent of its expected value (Baker et al., 2006), i.e. P(ajs)/eQ(s;a), where
Qis the optimal Q function that satisﬁes the Bellman equation:
Q(s;a) =R(s;a) +
X
s0h
T(s0js;a) max
a0Q(s0;a0)i
(1)
However, we know that humans are systematically suboptimal, and so we would like to relax this
assumption and try other, more realistic assumptions. The pathological solutions in the impossibility
result occur partly because the demonstrator can have arbitrary behavior on different environments.
While we certainly want the demonstrator to adapt to different environments, the algorithm that
the demonstrator uses to determine their policy should stay ﬁxed across similar environments. This
imposes structure on the demonstrator’s planner that can eliminate some possibilities.
Assumption 1: The demonstrator plans in the same way for sufﬁciently similar environments.
To formalize this, we assume that there is a space of world models Wwith the same set of states
Sand actions A, and a space of reward functions RSA!R. The demonstrator can plan
a stochastic policy given a world model and reward, that is, they are modeled as D: (WR)!
(S!A![0;1]). We callDthe planning algorithm used by the demonstrator, or planner for short.
Of course, if Dcan be any function with this type signature, it can still map any arbitrary (w;r)pair
to any arbitrary policy D(w;r), but we will further ensure that Dis simple (through regularization).
Given a list of world models W= [w1:::wn]and reward functions R= [r1:::rn], we deﬁne
D(W;R )to be the list of the demonstrator’s policies [D(w1;r1):::D (wn;rn)].
Note that this is a strong assumption and it does limit the scope of our work: while it is reasonable to
believe that people plan in the same way for variations of the same task, they likely have different
biases for different tasks, because they may have domain-speciﬁc heuristics. The setting of multiple
tasks has been studied before (Gleave & Habryka, 2018; Dimitrakakis & Rothkopf, 2011; Choi &
Kim, 2012), though not for the purpose of inferring systematic biases.
This assumption leads to a slightly easier problem, of recovering rewards from multiple tasks:
Estimating Biases and Rewards for Multiple Tasks. Given a list of world models Wand the
demonstrator’s policies D=D(W;R )which may exhibit an unknown bias, determine the list of
reward functions R(one for each w2W) that the demonstrator was optimizing.
Since the person uses the same planner across all tasks, an agent can have an easier time recovering
rewards for each task by leveraging the common structure across the tasks. This is especially
appealing for agents that would get to observe people for some period of time before trying to assist
them. Unfortunately, Assumption 1 is not sufﬁcient to solve this problem. Consider the case where
the demonstrator is optimal. Given the assumptions so far, we could infer that the demonstrator is
minimizing expected reward, i.e. that the reward they are optimizing is r, since that perfectly
predictsD. This is very bad, as we could infer a reward that incentivizes the worst possible behavior!
When humans take action, we typically assume that they are doing something that is reasonable for
achieving their goals, even if it is not optimal:
Assumption 2a: The demonstrator is "close" to optimal.
We explore this assumption in section 4.3 to solve the problem of estimating biases and rewards for
multiple tasks. We also explore an alternative approach, based on the fact that we have strong priors
about what humans are trying to optimize for, which allow us to infer how good they are at achieving
their goals. If we see some tasks where we know the demonstrator’s reward function and policy, we
can infer that the demonstrator is not minimizing expected reward.
Assumption 2b: We know what reward function the demonstrator is optimizing for some tasks.
Estimating Biases and Rewards with Access to Tasks with Known Rewards. Given a list of
world models W, a list of the demonstrator’s policies D=D(W;R ), a list of world models Wknown
with known rewards Rknown and a list of the demonstrator’s policies known =D(Wknown;Rknown),
determine the reward functions RthatDwas optimizing.
4
Under review as a conference paper at ICLR 2019
(a)Training the planner f. We hold the world model
w, rewardr, and policyﬁxed, and update the planner
fwith gradient descent.
(b)Training the reward R. We hold the world model
w, plannerf, and policy ﬁxed, and update the
rewardrwith gradient descent.
Figure 2: The architecture and operations on it that we use in our algorithms.
We will present algorithms for both settings in the next section. While our problem formulations
above assume that we have access to full policies D, none of our algorithms rely on this assumption
– it is easy to modify them to work with trajectories instead.
4 A LGORITHMS TO ESTIMATE BIASES AND REWARDS
The key idea behind our algorithms is to learn a model of how the demonstrator plans, and invert the
model’s "understanding" using backpropagation to infer the reward from actions.
4.1 A RCHITECTURE
We model the demonstrator planning algorithm Dusing a differentiable planner f, which is a neural
net that can express planning algorithms whose parameters can be updated using gradient descent.
fhas the same type as the demonstrator’s planner D, namely (WR)!(S!A![0;1]). Thus,
the inputs to the differentiable planner fare a world model w2Wand a reward function r2R;
the output is a stochastic policy 2(S!A![0;1]). We determine how well hf;Rimatches the
demonstrator’s policy Dwith the cross entropy loss L(f(W;R );D) =P
iL(f(wi;ri);D;i).
While our algorithms can work with any differentiable planner, in this work we use a value iteration
network (VIN) (Tamar et al., 2016). A VIN is a fully differentiable neural network that embeds an
approximate value iteration algorithm inside a feed-forward classiﬁcation network. For environments
where transitions only depend on "nearby" states (as in navigation tasks), the Bellman update can be
performed using an appropriate convolution, and the computation of values from Q-values can be
done with a max-pooling layer. By leaving the ﬁlters for the convolutions unspeciﬁed, the VIN can
automatically learn the transition probabilities. Of course, the VIN is merely one architecture for a
differentiable planner; we could equally well use other planners (Srinivas et al., 2018; Pascanu et al.,
2017; Guez et al., 2018). As research in this area advances, our work stands to beneﬁt.
By choosing this particular architecture we are making another assumption:
Assumption 3: The inductive bias of fis well-suited to the task of learning the planner D.
In our experiments, this assumption says that the demonstrator’s planner is well-modeled by a Value
Iteration Network. This is not a good assumption currently, but we expect that as more research is
done on differentiable planners, they will become more sophisticated and will become better able to
learn human planning algorithms.
The components of our algorithms. There are two main operations that we make use of on this
architecture, which we illustrate in ﬁgure 2. First, given world models W, reward functions R(either
known or hypothesized), and the demonstrator’s policies D, we can train a corresponding planner
using gradient descent (ﬁgure 2a):
= min
0L(f0(W;R );D) (TRAIN -PLANNER )
5
Under review as a conference paper at ICLR 2019
Second, given world models W, demonstrator’s policies D, and some planner parameters , we can
infer the corresponding reward functions using gradient descent (ﬁgure 2b):
R= min
R0L(f(W;R0);D) (TRAIN -REWARD )
We can also perform both of these at the same time by training the planner parameters and rewards
jointly given world models Wand the demonstrator’s policies D:
R;= min
R0;0L(f0(W;R0);D) (TRAIN -JOINTLY )
4.2 L EARNING THE PLANNER FROM KNOWN REWARDS FIRST (ASSUMPTION 2B)
First, we tackle the simpler setting when we have access to a set of tasks with known rewards. We
illustrate this in Algorithm 1. We ﬁrst train the planner on the world models for which we have
rewards, which lets us learn a model of how the demonstrator plans, including any systematic biases
they may have. Then, we use the learned planner weights to infer the reward on the world models for
which we don’t know the reward.
Algorithm 1 Estimating biases and rewards with access to tasks with known rewards.
1:function IRL-WITH -REWARDS (W,D,Wknown ,Rknown ,known )
2: TRAIN -PLANNER (Wknown;Rknown;known)
3: return TRAIN -REWARD (W;; D)
4.3 L EARNING THE PLANNER AND REWARDS SIMULTANEOUSLY (ASSUMPTION 2A)
The assumption that we have some tasks with known rewards is very strong, and may not hold in
practice. How could we infer rewards without making this assumption? The obvious answer is to
trainandRjointly as in Equation TRAIN -JOINTLY . However, since we have discarded Assumption
2b, we can once again learn the planner that minimizes expected reward, and infer that the reward is
r. We must instead use Assumption 2a, that the demonstrator is "close" to optimal.
To use this assumption in our algorithm, we simulate data from an optimal agent with randomly
generated world models and rewards, and use this to train the planner to mimic an optimal agent.
After initializing this way, we can then train the planner and reward jointly. This gives us Algorithm 2.
We show in section 5.3 that the initialization, enacting assumption 2a, is crucial for good performance.
Algorithm 2 Estimating biases and rewards for multiple tasks with no known rewards.
1:function IRL-WITHOUT -REWARDS (W,D)
2:Wsim;Rsim Generate random world models and rewards
3: sim Run optimal agent on hWsim;Rsimi
4:init TRAIN -PLANNER (Wsim;Rsim;sim)
5:Rinit TRAIN -REWARD (W;; D)
6:;R TRAIN -JOINTLY (W;D) .usinginitandRinitas initializations
7: returnR
5 E VALUATION
We evaluate our algorithms by simulating demonstrators with different biases, and testing whether
the same method can correctly infer reward for all these demonstrators. Details on the experiment
setup are provided in the supplementary material.
5.1 E VALUATING REWARD INFERENCE
Hypothesis. The key idea behind this work is that accounting for unknown systematic bias should
outperform the assumption of a particular inaccurate bias, e.g. noisy rationality or the lack thereof.
6
Under review as a conference paper at ICLR 2019
Figure 3: Reward obtained when planning with the inferred reward, as a percentage of the maximum possible
reward, for different bias models and algorithms.
Manipulated variables. In order to test this, we manipulate whether we learn the demonstrator
model or assume it. To avoid confounds introduced by changing the inference algorithm, we use the
same algorithm for both. In the learning case, we train the planner on the ground truth demonstrator
data; in the assume case, we train it on data generated from a) a Boltzmann-rational demonstrator;
and b) an optimal demonstrator – these are the two models commonly assumed by IRL algorithms.
Keeping the algorithm the same enables us to isolate the effect of adapting to an unknown model
from the effect of having to use an approximate differentiable planner rather than a perfect one. We
will quantify the second effect, i.e. the approximation error introduced by the VIN, in section 5.2
In the setting where we learn the bias, we further manipulate whether we have access to known
rewards for some tasks or not – i.e. whether we use Algorithm 1 or Algorithm 2.
Finally, we manipulate the actual bias of the demonstrator. Following Evans et al. (2016), we
implement the myopic, naive and sophisticated synthetic demonstrators as modiﬁcations of the value
iteration algorithm. Similarly, we implement the overconﬁdent and underconﬁdent demonstrators by
modifying the transition probability distributions used to plan in value iteration. We also include an
optimal demonstrator, and stochastic (Boltzmann) versions of all demonstrators.
Dependent measures. We measure the reward obtained by planning optimally with the inferred
reward function, as a percentage of the maximum possible reward that could be obtained.
Findings. Figure 3 shows our results comparing learning a demonstrator model with assuming an
optimal or a Boltzmann demonstrator. The top left subﬁgure plots what happens on average, across
all synthetic demonstrators we tested. The results provide support to our hypothesis: both learning
methods (orange) outperform assuming a model (gray). Looking at the breakdown per demonstrator,
we see that assuming optimal does not do well when the demonstrator has any noise (bottom graph).
Similarly, assuming Boltzmann does not do well when the demonstrator is not noisy (top graph). The
learning methods tend to perform on par with the best of two choices. In some cases, like the naive
and sophisticated hyperbolic discounters, especially the noisy ones, the learning methods outperform
both optimal and Boltzmann assumptions. The optimal assumption performs well in some of the
non-noisy cases, because our demonstrator bias models are almost deterministic (they only break ties
randomly). So, as long as the demonstrator eventually reaches the best reward location, the reward
inference works well.
Interestingly, Algorithm 1 does not always outperform Algorithm 2, despite it having access to
known rewards. We believe this has to do with the fact that Algorithm 2 exploits Assumption 2a
(demonstrator close to optimal) and initializes from training on simulated optimal demonstrator data.
Algorithm 1 does not rely on this assumption and therefore does not beneﬁt from this initialization,
whereas the assumption is correct for most of the models we test.
5.2 T RADEOFF BETWEEN BEING ADAPTIVE TO BIAS VS .USING EXACT PLANNING
Our paper is not about a practical solution to be used right now, but rather an investigation of the
viability of an idea. The core reason behind this is that to be adaptive to different kinds of biases we
7
Under review as a conference paper at ICLR 2019
Figure 4: Percent reward obtained for different bias models using variations of Algorithm 2, which does not get
access to any known rewards. These algorithms can vary along two dimensions – whether they are initialized
with the assumption that the demonstrator is rational, and whether they train the planner and reward jointly or
with coordinate ascent. The original version of Algorithm 2 does initialize, and trains jointly.
might see, we learn a model of the demonstrator’s planning algorithm via a differentiable planner.
Unfortunately, this causes our planning to be approximate – whatever beneﬁt we get from the adapting
to biases, we lose because of the approximation. But these planners will become more practical, they
can make this idea practical as well.
To quantify this loss, we replace the VIN with a differentiable exact model of the demonstrator,
and infer the reward by backpropagating through the exact model. Since value iteration is not
differentiable, we implement soft value iteration, where max operations are replaced with logsumexp
operations, and measure percent reward obtained when inferring rewards for an optimal demonstrator.
Results. With an exact model of the demonstrator, we get (98:10:1)% of the maximum reward when
performing optimal planning on the inferred rewards, while we get (86:91:6)% with Algorithm 1
and(86:21:6)% with Algorithm 2. Again, better planners would improve both algorithms.
5.3 H OW IMPORTANT ARE THE VARIOUS PARTS OF THE ALGORITHM ?
Algorithm 2 was predicated on Assumption 2a, that the demonstrator’s planner was "close" to rational,
which motivated the initialization step where the planner is trained to mimic an optimal agent. We test
how important this is by modifying Algorithm 2 to infer rewards without an initialization (removing
lines 2-5). We include versions of the algorithm where we perform coordinate ascent by alternating
planner training and reward training instead of training the planner and reward jointly.
Results. Figure 4 shows the results for a subset of demonstrators (full results are in the supplementary
material). We can see that the initialization is indeed crucial for good performance, as expected. It
also turns out that the joint training outperforms coordinate ascent.
6 D ISCUSSION
Summary. In this work, we considered the very general problem of inverse reinforcement learning
when the demonstrator has an unknown bias. In this problem we face a severe tradeoff between
the ability to adapt to unknown biases, and the ability to infer any reward function. We introduced
assumptions that are weaker than the typical assumption of noisy rationality, and tested algorithms
that leverage these assumptions to perform better on average across different kinds of biases than
assuming a speciﬁc bias known a priori.
Limitations and future work. We see this paper as taking a step towards exploring the idea of
learning planners from demonstrations to learn reward functions. This is not yet a practical idea,
due to the practical limitations of differentiable planners. Thus, to analyze how feasible this is with
different possible biases and with ground truth reward functions, we resorted to synthetic demonstrator
data as opposed to human data.
Further, our assumption that the demonstrator has the same bias across many tasks is key to our work,
but is very strong. We could extend this work by using meta-learning to learn a prior over planners.
We will also need to infer the demonstrator’s beliefs as in Baker & Tenenbaum (2014), for which
we could use TOMNets (Rabinowitz et al., 2018). We are excited to look into this, and into what
additional inductive bias we could leverage, in our future work.
8
Under review as a conference paper at ICLR 2019
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In
Proceedings of the twenty-ﬁrst international conference on Machine learning , pp. 1. ACM, 2004.
Dario Amodei, Paul Christiano, and Alex Ray. Learning from human preferences. https://blog.
openai.com/deep-reinforcement-learning-from-human-preferences/ ,
2017.
Stuart Armstrong and Sören Mindermann. Impossibility of deducing preferences and rationality from
human policy. arXiv preprint arXiv:1712.05812 , 2017.
Chris Baker, Rebecca Saxe, and Joshua B Tenenbaum. Bayesian models of human action understand-
ing. In Advances in neural information processing systems , pp. 99–106, 2006.
Chris L Baker and Joshua B Tenenbaum. Modeling human plan recognition using bayesian theory of
mind. Plan, activity, and intent recognition: Theory and practice , pp. 177–204, 2014.
Roger Buehler, Dale Grifﬁn, and Michael Ross. Exploring the" planning fallacy": Why people
underestimate their task completion times. Journal of personality and social psychology , 67(3):
366, 1994.
Benjamin Burchﬁel, Carlo Tomasi, and Ronald Parr. Distance minimization for reward learning from
scored trajectories. In AAAI , pp. 3330–3336, 2016.
Jaedeug Choi and Kee-Eung Kim. Nonparametric bayesian inverse reinforcement learning for
multiple reward functions. In Advances in Neural Information Processing Systems , pp. 305–313,
2012.
Paul Christiano. The easy goal inference problem is still hard. https://ai-alignment.com/
the-easy-goal-inference-problem-is-still-hard-fad030e0a876 , 2015.
Christos Dimitrakakis and Constantin A Rothkopf. Bayesian multitask inverse reinforcement learning.
InEuropean Workshop on Reinforcement Learning , pp. 273–284. Springer, 2011.
Owain Evans and Noah D Goodman. Learning the preferences of bounded agents. In NIPS Workshop
on Bounded Optimality , volume 6, 2015.
Owain Evans, Andreas Stuhlmüller, and Noah D Goodman. Learning the preferences of ignorant,
inconsistent agents. In AAAI , pp. 323–329, 2016.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control
via policy optimization. In International Conference on Machine Learning , pp. 49–58, 2016.
Shane Frederick, George Loewenstein, and Ted O’donoghue. Time discounting and time preference:
A critical review. Journal of economic literature , 40(2):351–401, 2002.
Adam Gleave and Oliver Habryka. Multi-task maximum entropy inverse reinforcement learning.
arXiv preprint arXiv:1805.08882 , 2018.
Arthur Guez, Théophane Weber, Ioannis Antonoglou, Karen Simonyan, Oriol Vinyals, Daan Wierstra,
Rémi Munos, and David Silver. Learning to search with mctsnets. arXiv preprint arXiv:1802.04697 ,
2018.
Dylan Hadﬁeld-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse
reward design. In Advances in Neural Information Processing Systems , pp. 6768–6777, 2017.
Daniel Kahneman. A perspective on judgment and choice: mapping bounded rationality. American
psychologist , 58(9):697, 2003.
Anirudha Majumdar, Sumeet Singh, Ajay Mandlekar, and Marco Pavone. Risk-sensitive inverse
reinforcement learning via coherent risk models. In Robotics: Science and Systems , 2017.
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, pp.
663–670, 2000.
9
Under review as a conference paper at ICLR 2019
Razvan Pascanu, Yujia Li, Oriol Vinyals, Nicolas Heess, Lars Buesing, Sebastien Racanière, David
Reichert, Théophane Weber, Daan Wierstra, and Peter Battaglia. Learning model-based planning
from scratch. arXiv preprint arXiv:1707.06170 , 2017.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John
Wiley & Sons, 2014.
Neil C Rabinowitz, Frank Perbet, H Francis Song, Chiyuan Zhang, SM Eslami, and Matthew
Botvinick. Machine theory of mind. arXiv preprint arXiv:1802.07740 , 2018.
Stuart Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual
conference on Computational learning theory , pp. 101–103. ACM, 1998.
Kyriacos Shiarlis, Joao Messias, and Shimon Whiteson. Inverse reinforcement learning from failure.
InProceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems ,
pp. 1060–1068. International Foundation for Autonomous Agents and Multiagent Systems, 2016.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning
networks. arXiv preprint arXiv:1804.00645 , 2018.
Jacob Steinhardt. Latent variables and model mis-speciﬁcation.
https://jsteinhardt.wordpress.com/2017/01/10/
latent-variables-and-model-mis-specification/ , 2017.
Jacob Steinhardt and Owain Evans. Model mis-speciﬁcation and inverse reinforce-
ment learning. https://jsteinhardt.wordpress.com/2017/02/07/
model-mis-specification-and-inverse-reinforcement-learning/ , 2017.
Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks. In
Advances in Neural Information Processing Systems , pp. 2154–2162, 2016.
Amos Tversky and Daniel Kahneman. Availability: A heuristic for judging frequency and probability.
Cognitive psychology , 5(2):207–232, 1973.
Jiangchuan Zheng, Siyuan Liu, and Lionel M Ni. Robust bayesian inverse reinforcement learning
with sparse behavior noise. In AAAI , pp. 2198–2205, 2014.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse
reinforcement learning. In AAAI , volume 8, pp. 1433–1438. Chicago, IL, USA, 2008.
10
Under review as a conference paper at ICLR 2019
A A DDITIONAL EXPERIMENTAL DATA
In section 5, we presented data on the results of running various algorithms against a set of demon-
strators, reporting the reward obtained according to the true reward function when using the inferred
reward with an optimal planner, as a percentage of the maximum possible true reward. Table 1
shows the percentage reward obtained for all combinations of algorithms and demonstrators. We
also measure the accuracy of the planner and reward at predicting the demonstrator’s actions in new
gridworlds where the rewards are the same but the wall locations have changed. These results are
presented in Table 2. Note that there are often multiple optimal actions at a given state, which makes
it challenging to get high accuracy.
Table 1: Percent reward obtained when the algorithm (column) is used to infer the bias of the demonstrator
(row). The optimal and Boltzmann algorithms assume a ﬁxed model of the demonstrator and train the VIN to
mimic the model before performing reward inference (and were used in ﬁgure 3). We also include the four
ﬂavors of Algorithm 2 that were plotted in ﬁgure 4. The VI algorithm uses a differentiable implementation of
soft value iteration as the planner instead of a VIN (used in section 5.2). The demonstrators are the optimal
agent, the biased agents of ﬁgure 1, and versions of each of these agents with Boltzmann noise.
Agent Optimal Boltzmann Algorithm 1 Coord w/ init Joint w/ init Coord w/o init Joint w/o init VI
Average 67:02:7 79:70:8 89:50:7 85:00:5 86:40:63:90:7 2:61:0 71:93:0
Optimal 87:31:0 73:92:3 86:91:6 86:21:6 88:51:14:21:2 2:63:7 98:10:1
Naive 86:40:9 74:41:6 91:10:8 84:61:2 87:50:93:21:3 2:63:7 96:10:1
Sophisticated 87:51:1 77:11:6 91:81:3 83:61:3 87:91:03:61:4 2:63:7 96:70:1
Myopic 82:80:8 77:01:2 81:02:8 80:60:8 82:61:05:52:8 2:63:7 87:50:2
Overconﬁdent 87:51:2 70:71:7 82:11:4 83:91:5 86:71:22:71:1 2:63:7 97:50:1
Underconﬁdent 88:00:8 74:71:6 86:71:2 86:11:5 88:51:02:41:4 2:63:7 98:90:2
Boltzmann 8:51:0 90:71:3 91:40:8 88:41:6 91:30:93:01:9 2:63:7 8:70:1
B-Naive 52:82:3 77:32:9 98:50:1 82:52:4 75:82:98:34:5 2:63:7 47:70:2
B-Sophisticated 51:52:1 74:52:8 98:80:2 80:11:5 77:02:38:73:9 2:63:7 48:00:2
B-Myopic 77:71:1 90:80:6 95:61:0 91:50:6 91:90:52:42:1 2:63:7 83:40:1
B-Overconﬁdent 7:00:9 84:12:3 79:22:3 81:42:8 86:31:30:81:6 2:63:7 8:70:1
B-Underconﬁdent 86:70:9 91:30:7 91:20:7 90:71:0 92:40:81:81:2 2:63:7 92:10:1
Table 2: Accuracy when predicting the demonstrator’s actions (row) on new gridworlds using the planner and
reward inferred by the algorithm (column). Algorithms and demonstrators are the same as in Table 1.
Agent Optimal Boltzmann Algorithm 1 Coord w/ init Joint w/ init Coord w/o init Joint w/o init VI
Optimal 61:30:4 59:80:4 62:00:3 62:80:2 63:60:3 63:00:2 72:40:1 25:70:1
Naive 60:10:3 59:40:3 58:60:3 61:30:3 61:80:3 61:00:3 71:10:1 24:90:1
Sophisticated 60:50:4 59:20:4 59:30:3 61:00:3 62:00:4 61:20:3 71:20:1 24:90:1
Myopic 54:10:4 53:50:5 54:90:5 55:60:2 56:10:3 56:00:1 62:80:1 20:40:1
Overconﬁdent 61:60:4 60:10:4 61:80:4 63:30:3 63:70:3 63:10:2 72:80:1 25:90:1
Underconﬁdent 60:90:4 59:50:4 61:40:3 62:40:3 62:90:3 62:50:3 72:00:1 25:50:1
Boltzmann 56:71:1 60:50:4 60:90:3 60:30:2 60:80:3 62:30:3 67:10:5 24:20:1
B-Naive 56:60:8 59:80:8 60:40:1 60:30:2 60:50:7 59:90:3 68:50:3 23:70:1
B-Sophisticated 57:60:7 60:20:7 60:50:2 60:50:2 61:20:3 60:10:3 68:50:3 23:70:1
B-Myopic 56:30:2 56:90:4 55:90:2 56:50:2 57:00:2 56:30:1 62:40:1 20:30:0
B-Overconﬁdent 56:91:1 60:70:4 61:30:3 60:90:2 61:60:3 62:70:2 68:00:5 24:20:1
B-Underconﬁdent 62:40:3 63:10:4 63:40:2 63:00:1 63:60:1 63:50:2 72:20:1 25:40:1
B E XPERIMENT DETAILS
All results are averaged over 10 runs with different seeds, on randomly generated 14x14 gridworlds
that have 7 squares with non-zero rewards. We ensure that all such squares can be reached from the
start state, and that at least half of the positions in grid are not walls.
We use a Value Iteration Network with 10 iterations as our differentiable planner, and set the space of
rewards to be S!R; that is, any state can be mapped to any reward, but the reward is assumed not
to depend on the action. We added an extra convolutional layer to the initial part of the VIN (which
learns the proxy reward) as initial experiments showed that this could better learn an optimal planner
for our gridworlds. We apply L2 regularization to the VIN with scale 0.0001, and do not regularize
the reward.
For all experiments, we kept the number of demonstrations ﬁxed to 8000. For Algorithm 1, this was
split into 7000 policies with rewards that were used to train the planner, and 1000 on which rewards
had to be inferred. Note that this does notinclude any simulated data – for example, Algorithm 2
would get 8000 biased policies, and would also simulate a further 7000 policies from an optimal
agent in order to initialize the planner and reward.
11

Rohin Shah
Noah Gundotra
Pieter Abbeel
Anca Dragan


Toward Evaluating Robustness of Deep Reinforcement Learning with Continuous Control

Published as a conference paper at ICLR 2020
TOWARD EVALUATING ROBUSTNESS OF DEEP REIN -
FORCEMENT LEARNING WITH CONTINUOUS CONTROL
Tsui-Wei Weng1;, Krishnamurthy (Dj) Dvijotham2;|, Jonathan Uesato2;|, Kai Xiao1;|;
Sven Gowal2;|, Robert Stanforth2;|, Pushmeet Kohli2
1MIT,2DeepMind
twweng@mit.edu, fdvij,juesatog@google.com, kaix@mit.edu,
fsgowal,stanforth,pushmeet g@google.com
ABSTRACT
Deep reinforcement learning has achieved great success in many previously difﬁ-
cult reinforcement learning tasks, yet recent studies show that deep RL agents are
also unavoidably susceptible to adversarial perturbations, similar to deep neural
networks in classiﬁcation tasks. Prior works mostly focus on model-free adver-
sarial attacks and agents with discrete actions. In this work, we study the problem
of continuous control agents in deep RL with adversarial attacks and propose the
ﬁrst two-step algorithm based on learned model dynamics. Extensive experiments
on various MuJoCo domains (Cartpole, Fish, Walker, Humanoid) demonstrate that
our proposed framework is much more effective and efﬁcient than model-free at-
tacks baselines in degrading agent performance as well as driving agents to unsafe
states.
1 I NTRODUCTION
Deep reinforcement learning (RL) has revolutionized the ﬁelds of AI and machine learning over
the last decade. The introduction of deep learning has achieved unprecedented success in solving
many problems that were intractable in the ﬁeld of RL, such as playing Atari games from pixels and
performing robotic control tasks (Mnih et al., 2015; Lillicrap et al., 2015; Tassa et al., 2018). Un-
fortunately, similar to the case of deep neural network classiﬁers with adversarial examples, recent
studies show that deep RL agents are also vulnerable to adversarial attacks.
A commonly-used threat model allows the adversary to manipulate the agent’s observations at every
time step, where the goal of the adversary is to decrease the agent’s total accumulated reward. As a
pioneering work in this ﬁeld, (Huang et al., 2017) show that by leveraging the FGSM attack on each
time frame, an agent’s average reward can be signiﬁcantly decreased with small input adversarial
perturbations in ﬁve Atari games. (Lin et al., 2017) further improve the efﬁciency of the attack
in (Huang et al., 2017) by leveraging heuristics of detecting a good time to attack and luring agents
to bad states with sample-based Monte-Carlo planning on a trained generative video prediction
model.
Since the agents have discrete actions in Atari games (Huang et al., 2017; Lin et al., 2017), the prob-
lem of attacking Atari agents often reduces to the problem of ﬁnding adversarial examples on image
classiﬁers, also pointed out in (Huang et al., 2017), where the adversaries intend to craft the input
perturbations that would drive agent’s new action to deviate from its nominal action. However, for
agents with continuous actions, the above strategies can not be directly applied. Recently, (Uesato
et al., 2018) studied the problem of adversarial testing for continuous control domains in a similar
but slightly different setting. Their goal was to efﬁciently and effectively ﬁnd catastrophic failure
given a trained agent and to predict its failure probability. The key to success in (Uesato et al., 2018)
is the availability of agent training history. However, such information may not always be accessible
to the users, analysts, and adversaries.
Besides, although it may not be surprising that adversarial attacks exist for the deep RL agents as
adversarial attacks have been shown to be possible for neural network models in various supervised
Work done during summer internship at DeepMind, UK.|Equal contributions.
1
Published as a conference paper at ICLR 2020
learning tasks. However, the vulnerability of RL agents can not be easily discovered by existing
baselines which are model-free and build upon random searches and heuristics – this is also veriﬁed
by our extensive experiments on various domains (e.g. walker, humanoid, cartpole, and ﬁsh), where
the agents still achieve close to their original best rewards even with baseline attacks at every time
step. Hence it is important and necessary to have a systematic methodology to design non-trivial ad-
versarial attacks, which can efﬁciently andeffectively discover the vulnerabilities of deep RL agents
– this is indeed the motivation of this work.
This paper takes a ﬁrst step toward this direction by proposing the ﬁrst sample-efﬁcient model-based
adversarial attack. Speciﬁcally, we study the robustness of deep RL agents in a more challenging set-
ting where the agent has continuous actions and its training history is not available. We consider the
threat models where the adversary is allowed to manipulate an agent’s observations or actions with
small perturbations, and we propose a two-step algorithmic framework to ﬁnd efﬁcient adversarial
attacks based on learned dynamics models. Experimental results show that our proposed model-
based attack can successfully degrade agent performance and is also more effective and efﬁcient
than model-free attacks baselines.
The contributions of this paper are the following:
To the best of our knowledge, we propose the ﬁrst model-based attack on deep RL agents
with continuous actions. Our proposed attack algorithm is a general two-step algorithm and
can be directly applied to the two commonly-used threat models (observation manipulation
and action manipulation).
We study the efﬁciency and effectiveness of our proposed model-based attack with model-
free attack baselines based on random searches and heuristics. We show that our model-
based attack can degrade agent performance in numerous MuJoCo domains by up to 4in
terms of total reward and up to 4:6in terms of distance to unsafe states (smaller means
stronger attacks) compared to the model-free baselines.
Our proposed model-based attack also outperform all the baselines by a large margin in a
weaker adversary setting where the adversary cannot attack at every time step. In addition,
ablation study on the effect of planning length in our proposed technique suggests that our
method can still be effective even when the learned dynamics model is not very accurate.
2 B ACKGROUND
Adversarial attacks in reinforcement learning. Compared to the rich literature of adversarial
examples in image classiﬁcations (Szegedy et al., 2013) and other applications (including natural
language processing (Jia & Liang, 2017), speech (Carlini & Wagner, 2018), etc), there is relatively
little prior work studying adversarial examples in deep RL. One of the ﬁrst several works in this ﬁeld
are (Huang et al., 2017) and (Lin et al., 2017), where both works focus on deep RL agent in Atari
games with pixels-based inputs and discrete actions. In addition, both works assume the agent to
be attacked has accurate policy and the problem of ﬁnding adversarial perturbation of visual input
reduces to the same problem of ﬁnding adversarial examples on image classiﬁers. Hence, (Huang
et al., 2017) applied FGSM (Goodfellow et al., 2015) to ﬁnd adversarial perturbations and (Lin
et al., 2017) further improved the efﬁciency of the attack by heuristics of observing a good timing
to attack – when there is a large gap in agents action preference between most-likely and least-
likely action. In a similar direction, (Uesato et al., 2018) study the problem of adversarial testing
by leveraging rejection sampling and the agent training histories. With the availability of training
histories, (Uesato et al., 2018) successfully uncover bad initial states with much fewer samples
compared to conventional Monte-Carlo sampling techniques. Recent work by (Gleave et al., 2019)
consider an alternative setting where the agent is attacked by another agent (known as adversarial
policy), which is different from the two threat models considered in this paper. Finally, besides
adversarial attacks in deep RL, a recent work (Wang et al., 2019) study veriﬁcation of deep RL
agent under attacks, which is beyond the scope of this paper.
Learning dynamics models. Model-based RL methods ﬁrst acquire a predictive model of the
environment dynamics, and then use that model to make decisions (Atkeson & Santamaria, 1997).
These model-based methods tend to be more sample efﬁcient than their model-free counterparts,
2
Published as a conference paper at ICLR 2020
(a) Attack observations of agent.
 (b) Attack actions of agent.
Figure 1: Two commonly-used threat models.
and the learned dynamics models can be useful across different tasks. Various works have focused
on the most effective ways to learn and utilize dynamics models for planning in RL (Kurutach et al.,
2018; Chua et al., 2018; Chiappa et al., 2017; Fu et al., 2016).
3 P ROPOSED FRAMEWORK
In this section, we ﬁrst describe the problem setup and the two threat models considered in this
paper. Next, we present an algorithmic framework to rigorously design adversarial attacks on deep
RL agents with continuous actions.
3.1 P ROBLEM SETUP AND FORMULATION
Letsi2RNandai2RMbe the observation vector and action vector at time step i, and let
:RN!RMbe the deterministic policy (agent). Let f:RNRM!RNbe the dynamics
model of the system (environment) which takes current state-action pair (si;ai)as inputs and outputs
the next state si+1. We are now in the role of an adversary, and as an adversary, our goal is to drive
the agent to the (un-safe) target states starget within thebudget constraints.
We can formulate this goal into two optimization problems, as we will illustrate shortly below.
Within this formalism, we can consider two threat models:
Threat model (i): Observation manipulation. For the threat model of observation manipulation,
an adversary is allowed to manipulate the observation sithat the agent perceived within an budget:
ksik1; L ssi+ siUs; (1)
where si2RNis the crafted perturbation and Us2RN;Ls2RNare the observation limits.
Threat model (ii): Action manipulation. For the threat model of action manipulation, an adver-
sary can craft ai2RMsuch that
kaik1; L aai+ aiUa; (2)
whereUa2RM;La2RMare the limits of agent’s actions.
Our formulations. Given an initial state s0and a pre-trained policy , our (adversary) objective is
to minimize the total distance of each state sito the pre-deﬁned target state starget up to the unrolled
(planning) steps T. This can be written as the following optimization problems in Equations 3 and
4 for the Threat model (i) and(ii)respectively:
min
siTX
i=1d(si;starget)
s.t.ai=(si+ si); si+1=f(si;ai);Constraint (1) ; i2ZT;(3)
min
aiTX
i=1d(si;starget)
s.t.ai=(si); si+1=f(si;ai+ ai);Constraint (2) ; i2ZT:(4)
3
Published as a conference paper at ICLR 2020
A common choice of d(x;y)is the squared `2distancekxyk2
2andfis the learned dynamics
model of the system, and Tis the unrolled (planning) length using the dynamics models.
3.2 O UR ALGORITHM
In this section, we propose a two-step algorithm to solve Equations 3 and 4. The core of our pro-
posal consists of two important steps: learn a dynamics model fof the environment and deploy
optimization technique to solve Equations 3 and 4. We ﬁrst discuss the details of each factor, and
then present the full algorithm by the end of this section.
Step 1: learn a good dynamics model f.Ideally, iffis the exact (perfect) dynamics model of
the environment and assuming we have an optimization oracle to solve Equations 3 and 4, then
the solutions are indeed the optimal adversarial perturbations that give the minimal total loss with
-budget constraints. Thus, learning a good dynamics model can conceptually help on developing a
strong attack. Depending on the environment, different forms of fcan be applied. For example, if
the environment of concerned is close to a linear system, then we could let f(s;a) =As+Ba, where
AandBare unknown matrices to be learned from the sample trajectories (si;ai;si+1)pairs. For a
more complex environment, we could decide if we still want to use a simple linear model (the next
state prediction may be far deviate from the true next state and thus the learned dynamical model is
less useful) or instead switch to a non-linear model, e.g. neural networks, which usually has better
prediction power but may require more training samples. For either case, the model parameters
A;B or neural network parameters can be learned via standard supervised learning with the sample
trajectories pairs (si;ai;si+1).
Step 2: solve Equations 3 and 4. Once we learned a dynamical model f, the next immediate task
is to solve Equation 3 and 4 to compute the adversarial perturbations of observations/actions. When
the planning (unrolled) length T > 1, Equation 3 usually can not be directly solved by off-the-
shelf convex optimization toolbox since the deel RL policy is usually a non-linear and non-convex
neural network. Fortunately, we can incorporate the two equality constraints of Equation 3 into
the objective and with the remaining -budget constraint (Equation 1), Equation 3 can be solved
via projected gradient descent (PGD)1. Similarly, Equation 4 can be solved via PGD to get ai.
We note that, similar to the n-step model predictive control, our algorithm could use a much larger
planning (unrolled) length Twhen solving Equations 3 and 4 and then only apply the ﬁrst n(T)
adversarial perturbations on the agent over ntime steps. Besides, with the PGD framework, fis
not limited to feed-forward neural networks. Our proposed attack is summarized in Algorithm 2 for
Step 1 , and Algorithm 3 for Step 2 .
Algorithm 1 Collect trajectories
1:Input: pre-trained policy , MaxSampleSize ns, environment env
2:Output: a set of trajectory pairs S
3:k 0;S
4:s0 env.reset()
5:whilek<n sdo
6:ak (sk)
7:sk+1 env.step(ak)
8:S[f (sk;ak;sk+1)g
9:k k+ 1
10:end while
11:ReturnS
4 E XPERIMENTS
In this section, we conduct experiments on standard reinforcement learning environment for con-
tinuous control (Tassa et al., 2018). We demonstrate results on 4 different environments in Mu-
1Alternatively, standard optimal control methods such as Linear Quadratic Regulator (LQR) and iterative
Linear Quadratic Regulator (i-LQR) can also be applied to solve Equations 3 and 4 approximately.
4
Published as a conference paper at ICLR 2020
Algorithm 2 learn dynamics
1:Input: pre-trained policy , MaxSampleSize ns, environment env, trainable parameters W
2:Output: learned dynamical model f(s;a;W)
3:Sagent Collect trajectories (;ns;env)
4:Srandom Collect trajectories (random policy;ns;env)
5:f(s;a;W) supervised learning algorithm(Sagent[S random;W)
6:Returnf(s;a;W)
Algorithm 3 model based attack
1:Input: pre-trained policy , learned dynamical model f(s;a;W), threat model, maximum
perturbation magnitude , unroll length T, apply perturbation length n(T)
2:Output: a sequence of perturbation 1;:::; n
3:ifthreat model is observation manipulation (Eq. 1) then
4: Solve Eq. 3 with parameters ( ;f;;T ) via PGD to get 1;:::; T
5:else if threat model is action manipulation (Eq. 2) then
6: Solve Eq. 4 with parameters ( ;f;;T ) via PGD to get 1;:::; T
7:end if
8:Return1;:::; n
JoCo Tassa et al. (2018) and corresponding tasks: Cartpole-balance/swingup, Fish-upright, Walker-
stand/walk and Humanoid-stand/walk. For the deep RL agent, we train a state-of-the-art D4PG
agent (Barth-Maron et al., 2018) with default Gaussian noise N(0;0:3I)on the action and the score
of the agents without attacks is summarized in Appendix A.3. The organization is as follows: we
ﬁrst evaluate the effectiveness of our proposed model-based attack and three model-free baselines
in terms of both loss and reward. Next, we conduct ablation study on the key parameter of our
algorithm the planning length T, evaluate our algorithm on a weaker attack setting and also discuss
the efﬁciency of our proposed attack in terms of sample complexity.
Evaluations. We conduct experiments for 10 different runs, where the environment is reset to
different initial states in different runs. For each run, we attack the agent for one episode with 1000
time steps (the default time intervals is usually 10ms) and we compute the total loss and total return
reward. The total loss calculates the total distance of current state to the unsafe states and the total
return reward measures the true accumulative reward from the environment based on agent’s action.
Hence, the attack algorithm is stronger if the total return reward and the total loss are smaller.
Baselines. We compare our algorithm with the following model-free attack baselines with random
searches and heuristics:
rand-U : generatemrandomly perturbed trajectories from Uniform distribution with inter-
val[;]and return the trajectory with the smallest loss (or reward),
rand-B : generatemrandomly perturbed trajectories from Bernoulli distribution with prob-
ability 1=2and interval [;], and return the trajectory with the smallest loss (or reward),
ﬂip: generate perturbations by ﬂipping agent’s observations/actions within the budget in
`1norm.
Forrand-U andrand-B , they are similar to Monte-Carlo sampling methods, where we generate
msample trajectories from random noises and report the loss/reward of the best trajectory (with
minimum loss or reward among all the trajectories). We set m= 1000 throughout the experiments.
More details see Appendix A.2.
Our algorithm. A 4-layer feed-forward neural network with 1000 hidden neurons per layer is
trained as the dynamics model frespectively for the domains of Cartpole, Fish, Walker and Hu-
manoid. We use standard `2loss (without regularization) to learn a dynamics model f. Instead of
using recurrent neural network to represent f, we found that the 1-step prediction for dynamics with
the 4-layer feed-forward network is already good for the MuJoCo domains we are studying. Specif-
ically, for the Cartpole and Fish, we found that 1000 episodes ( 1e6training points) are sufﬁcient
5
Published as a conference paper at ICLR 2020
Figure 2: Video frames of best attacks in each baseline among 10 runs for the Walker.walk example.
Only our proposed attack can constantly make the Walker fall down (since we are minimizing its
head height to be zero).
to train a good dynamics model (the mean square error for both training and test losses are at the
order of 105for Cartpole and 102for Fish), while for the more complicated domain like Walker
and Humanoid, more training points ( 5e6) are required to achieve a low test MSE error (at the order
of101and100for Walker and Humanoid respectively). Consequently, we use larger planning
(unrolled) length for Cartpole and Fish (e.g. T= 10;20), while a smaller T(e.g. 3or5) is used
for Walker and Humanoid. Meanwhile, we focus on applying projected gradient descent (PGD) to
solve Equation 3 and 4. We use Adam as the optimizer with optimization steps equal to 30 and we
report the best result for each run from a combination of 6 learning rates, 2 unroll length fT1;T2g
andnsteps of applying PGD solution with nTi.
4.1 R ESULTS
For observation manipulation, we report the results on Walker, Humanoid and Cartpole domains
with tasks (stand, walk, balance, swingup) respectively. The unsafe states starget for Walker and
Humanoid are set to be zero head height, targeting the situation of falling down. For Cartpole, the
unsafe states are set to have 180pole angle, corresponding to the cartpole not swinging up and nor
balanced. For the Fish domain, the unsafe states for the upright task target the pose of swimming
ﬁsh to be not upright, e.g. zero projection on the z-axis.
The full results of both two threat models on observation manipulation and action manipulation are
shown in Table 1a, b and c, d respectively. Since the loss is deﬁned as the distance to the target
(unsafe) state, the lower the loss, the stronger the attack. It is clear that our proposed attack achieves
much lower loss in Table 1a & c than the other three model-free baselines, and the averaged ratio is
also listed in 1b & d. Notably, over the 10 runs, our proposed attack always outperforms baselines
for the threat model of observation perturbation and the Cartpole domain for the threat model of
action perturbation, while still superior to the baselines despite losing two times to the ﬂipbaseline
on the Fish domain.
To have a better sense on the numbers, we give some quick examples below. For instance, as shown
in Table 1a and b, we show that the average total loss of walker head height is almost unaffected
for the three baselines – if the walker successfully stand or walk, its head height usually has to be
greater than 1:2at every time step, which is 1440 for one episode – while our attack can successfully
lower the walker head height by achieving an average of total loss of 258(468) , which is roughly
0:51(0:68)per time step for the stand (walk) task. Similarly, for the humanoid results, a successful
humanoid usually has head height greater than 1:4, equivalently a total loss of 1960 for one episode,
and Table 1a shows that the d4pg agent is robust to the perturbations generated from the three model-
free baselines while being vulnerable to our proposed attack. Indeed, as shown in Figure 2, the
6
Published as a conference paper at ICLR 2020
Table 1: Compare three model-free attack baselines ( rand-U ,rand-B ,ﬂip) and our algorithm
(Ours ) in 4 different domains and tasks. We report the following statistics over 10 different runs:
mean, standard deviation, averaged ratio, and best attack (number of times having smallest loss over
10 different runs). Results show that our attack outperforms all the model-free attack baselines for
the observation manipulation threat model by a large margin for all the statistics. Our proposed
attack is also superior on the action manipulation threat model and win over most of the evaluation
metrics.
(a) Observation manipulation: mean and standard deviation (in parenthesis)
Total loss rand-U rand-B ﬂip Ours
Walkerstand 1462 (70) 1126 (86) 1458 (24) 258 (55)
walk 1517 (22) 1231 (31) 1601 (18) 466 (42)
Humanoidstand 1986 (28) 1808 (189) 1997 (5) 516 (318)
walk 1935 (22) 1921 (31) 1982 (9) 1457 (146)
Cartpolebalance 4000 (0.02) 3999 (0.04) 3989 (2) 2101 (64)
swingup 3530 (1) 3525 (1) 3516 (1) 2032 (172)
(b) Observation manipulation: averaged ratio and rank-1
Total loss (avg ratio) Ours/rand-U Ours/rand-B Ours/ﬂip best attack
Walkerstand 0.18 0.23 0.18 Ours: 10/10, others: 0/10
walk 0.31 0.38 0.29 Ours: 10/10, others: 0/10
Humanoidstand 0.26 0.29 0.26 Ours: 10/10, others: 0/10
walk 0.75 0.76 0.74 Ours: 10/10, others: 0/10
Cartpolebalance 0.53 0.53 0.53 Ours: 10/10, others: 0/10
swingup 0.58 0.58 0.58 Ours: 10/10, others: 0/10
(c) Action manipulation: mean and standard deviation (in parenthesis)
Total loss rand-U rand-B ﬂip Ours
Cartpolebalance 4000 (0.03) 3999 (0.08) 3046 (1005) 1917 (102)
swingup 3571 (1) 3487 (7) 1433 (4) 1388 (50)
Fish upright 935 (27) 936 (24) 907 (22) 824 (84)
(d) Action manipulation: averaged ratio and rank-1
Total loss (avg ratio) Ours/rand-U Ours/rand-B Ours/ﬂip best attack
Cartpolebalance 0.48 0.48 0.63 Ours: 10/10, others: 0/10
swingup 0.39 0.40 0.97 Ours: 10/10, others: 0/10
Fish upright 0.88 0.88 0.91 Ours: 8/10, ﬂip: 2/10
7
Published as a conference paper at ICLR 2020
Table 2: Compare three attack baselines ( rand-U ,rand-B ,ﬂip) and our algorithm ( Ours ) in three
different domains and tasks. Performance statistics of 10 different runs are reported.
(a) The mean and standard deviation (in parenthesis) over 10 different runs
Total reward rand-U rand-B ﬂip Ours
Walkerstand 937 (41) 744 (48) 993 (8) 235 (38)
walk 941 (23) 796 (21) 981 (9) 225 (50)
Humanoidstand 927 (21) 809 (85) 959 (5) 193 (114)
walk 934 (22) 913 (21) 966 (6) 608 (66)
Cartpolebalance 995 (0.17) 986 (0.16) 985 (3) 385 (6)
swingup 873 (0.75) 851 (2) 852 (0.29) 353 (61)
(b) Average ratio and number of times our algorithm being the best attack over 10 runs.
Total reward (avg ratio) Ours/rand-U Ours/rand-B Ours/ﬂip best attack
Walkerstand 0.25 0.32 0.24 Ours: 10/10, others: 0/10
walk 0.24 0.28 0.23 Ours: 10/10, others: 0/10
Humanoidstand 0.21 0.24 0.20 Ours: 10/10, others: 0/10
walk 0.65 0.67 0.63 Ours: 10/10, others: 0/10
Cartpolebalance 0.39 0.39 0.39 Ours: 10/10, others: 0/10
swingup 0.41 0.42 0.42 Ours: 10/10, others: 0/10
walker and humanoid falls down quickly (head height is close to zero) under our specially-designed
attack while remaining unaffected for all the other baselines.
4.2 D ISCUSSION
Evaluating on the total reward. Often times, the reward function is a complicated function and
its exact deﬁnition is often unavailable. Learning the reward function is also an active research ﬁeld,
which is not in the coverage of this paper. Nevertheless, as long as we have some knowledge of
unsafe states (which is often the case in practice), then we can deﬁne unsafe states that are related
to low reward and thus performing attacks based on unsafe states (i.e. minimizing the total loss
of distance to unsafe states) would naturally translate to decreasing the total reward of agent. As
demonstrated in Table 2, the results have the same trend of the total loss result in Table 1, where
our proposed attack signiﬁcantly outperforms all the other three baselines. In particular, our method
can lower the average total reward up to 4:96compared to the baselines result, while the baseline
results are close to the perfect total reward of 1000 .
Evaluating the effect of planning length. To investigate model effect over time, we perform ab-
lation studies on the planning/unroll length Tof our proposed model-based attack in three examples:
(I) cartpole.balance (II) walker.walk and (III) walker.stand.
(I) Cartpole balance. Our learned models are very accurate (test MSE error on the order of 106).
We observed that the prediction error of our learned model compared to the true model (the MuJoCo
simulator) is around 10% for 100 steps. Hence, we can choose Tto be very large (e.g. 20-100) and
our experiments show that the result of T= 100 is slightly better, see Appendix A.4.
(II) Walker walk. This task is much more complicated than (I), and our learned model is less accurate
(test MSE is 0:447). For 10 steps, the prediction error of our learned model compared to the true
model is already more than 100%, and hence using a small T for planning would be more reasonable.
Table 3a shows that T= 1 indeed gives the best attack results (decreases the loss by 3:2and
decreases the reward by 3:6compared to the best baseline (randB)) and the attack becomes less
powerful as T increases. Nevertheless, even with T= 10 , our proposed technique still outperforms
the best baseline (randB) by 1:4both in the total loss and total reward.
8
Published as a conference paper at ICLR 2020
Table 3: Ablation study on the planning length T. Compare 3 attack baselines ( rand-U ,rand-B ,
ﬂip) and our algorithm ( Ours ) and report performance statistics of 10 different runs.
(a) domain: Walker, task: walk (observation perturbation)
Walker.walk Total loss Total reward
mean std med min max mean std med min max
Ours,T= 1 468 79 489 286 567 222 45 227 135 300
Ours,T= 2 604 31 611 535 643 353 51 362 253 441
Ours,T= 5 761 65 771 617 837 483 60 496 348 540
Ours,T= 10 881 68 886 753 975 568 48 579 469 623
Ours,T= 15 874 93 891 723 1002 583 58 604 483 647
Ours,T= 20 937 62 950 804 993 634 41 638 559 687
rand-U 1517 22 1522 1461 1542 941 23 945 885 965
rand-B 1231 31 1234 1189 1272 796 21 796 766 824
ﬂip 1601 18 1604 1562 1619 981 9 984 961 991
(b) domain: Walker, task: stand (observation perturbation)
Walker.stand Total loss Total reward
mean std med min max mean std med min max
Ours,T= 1 322 84 319 202 453 257 67 265 163 366
Ours,T= 2 279 55 264 223 391 246 40 232 200 322
Ours,T= 5 163 53 154 93 246 193 27 188 154 238
Ours,T= 10 84 46 67 42 165 153 24 142 132 194
Ours,T= 15 101 40 82 57 157 164 23 152 143 201
Ours,T= 20 117 41 98 68 193 170 21 161 149 207
rand-U 1462 70 1454 1341 1561 938 41 932 866 999
rand-B 1126 86 1130 973 1244 744 48 744 664 809
ﬂip 1458 24 1451 1428 1501 993 8 997 979 999
Table 4: Less frequency attack. Report statistics of 10 different runs with different initial states in
the walker domain with task stand.
Total loss Total reward
mean std med min max mean std med min max
Ours 934 152 886 769 1187 648 95 622 559 799
rand-U 1511 35 1502 1468 1558 970 20 964 947 999
rand-B 1431 77 1430 1282 1541 924 41 923 840 981
ﬂip 1532 15 1537 1496 1546 996 5 999 984 1000
(III) Walker stand. The learned model is slightly more accurate than the (II) (test MSE is 0:089) in
this task. Interestingly, Table 3b show that with the more accurate walker.stand model (compared
to the walker.walk model), T= 10 gives the best avg total loss& reward , which are 13:4and
4:9smaller than the best baseline rand-B . Note that even with T= 1, the worst choice among
all our reported T, the result is still 3:5and2:9better than the best baselines, demonstrating the
effectiveness of our proposed approach.
The main takeaway from these experiments is that when the model is accurate, we can use larger T
in our proposed attack; while when the model is less accurate, smaller Tis more effective (as in the
Walker.walk example). However, even under the most unfavorable hyperparameters, our proposed
attack still outperforms all the baselines by a large margin.
Evaluating on the effectiveness of attack. We study the setting where attackers are less powerful
– they can only attack every 2 time steps instead of every transition. Table 4 shows that our proposed
9
Published as a conference paper at ICLR 2020
attack is indeed much stronger than the baselines even when the attackers power is limited to attack
every 2 time steps: (1) compared to the best results among three baselines, our attack gives 1:53
smaller avg total loss (2) the mean reward of all the baselines is close to perfect reward, while our
attacks can achieve 1:43smaller average total reward compared to the best baseline.
Evaluating on the efﬁciency of attack. We also study the efﬁciency of the attack in terms of
sample complexity, i.e. how many episodes do we need to perform an effective attack? Here we
adopt the convention in control suite (Tassa et al., 2018) where one episode corresponds to 1000 time
steps (samples) and learn the neural network dynamical model fwith different number of episodes.
Figure 3 in Appendix A.1 plots the total head height loss of the walker (task stand) for 3 base-
lines and our method with dynamical model ftrained with three different number of samples:
f5e5;1e6;5e6g, or equivalentlyf500;1000;5000gepisodes. We note that the sweep of hyper pa-
rameters is the same for all the three models, and the only difference is the number of training
samples. The results show that for the baselines rand-U andﬂip, the total losses are roughly at the
order of 1400 -1500 , while a stronger baseline rand-B still has total losses of 900-1200 . However, if
we solve Eq. equation 3 with ftrained by 5e5or1e6samples, the total losses can be decreased to
the order of 400-700and are already winning over the three baselines by a signiﬁcant margin. Same
as our expectation, if we use more samples (e.g. 5e6, which is 5-10times more), to learn a more
accurate dynamics model, then it is beneﬁcial to our attack method – the total losses can be further
decreased by more than 2and are at the order of 50-250over 10 different runs. See Appendix A.1
for more details.
Here we also give a comparison between our model-based attack to existing works (Uesato et al.,
2018; Gleave et al., 2019) on the sample complexity. In (Uesato et al., 2018), 3e5episodes of
training data is used to learn the adversarial value function, which is roughly 1000more data than
even our strongest adversary (with 5e3episodes). Similarly, (Gleave et al., 2019) use roughly 2e4
episodes to train an adversary via deep RL, which is roughly 4more data than ours2.
5 C ONCLUSIONS AND FUTURE WORKS
In this paper, we study the problem of adversarial attacks in deep RL with continuous control for two
commonly-used threat models. We proposed the ﬁrst model-based attack algorithm and showed that
our formulation can be easily solved by off-the-shelf gradient-based solvers. Extensive experiments
on 4 MuJoCo domains show that our proposed algorithm outperforms all model-free based attack
baselines by a large margin. We hope our discovery of the vulnerability of deep RL agent can bring
more safety awareness to researchers when they design algorithms to train deep RL agents.
There are several interesting future directions can be investigated based on this work, including
learning reward functions to facilitate a more effective attack, extending our current approach to
develop effective black-box attacks, and incorporating our proposed attack algorithm to adversarial
training of the deep RL agents. In particular, we think there are three important challenges that need
to be addressed to study adversarial training of RL agents along with our proposed attacks: (1) The
adversary and model need to be jointly updated. How do we balance these two updates, and make
sure the adversary is well-trained at each point in training? (2) How to avoid cycles in the training
process due to the agent overﬁtting to the current adversary? (3) How to ensure the adversary doesn’t
overly prevent exploration/balance unperturbed vs. robust performance?
ACKNOWLEDGEMENT
The authors thank Chongli Qin, Po-Sen Huang, Taylan Cemgil, Daniel J. Mankowitz, Nir Levine,
Alistair Muldal, Gabriel Barth-Maron, Matthew W. Hoffman, Yuval Tassa, Tom Erez and Jost Tobias
Springenberg for useful discussions and suggestions.
2It is only a qualitative comparison in the sample complexity regimes since the applications are not the
same. This is in-line with the theoretical perspective that model-based approaches are expected to be more
sample-efﬁcient than the model-free counterparts
10
Published as a conference paper at ICLR 2020
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar-
tin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920 ,
2018.
Christopher G Atkeson and Juan Carlos Santamaria. A comparison of direct and model-based rein-
forcement learning. International Conference on Robotics and Automation , 1997.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Alistair
Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy
gradients. arXiv preprint arXiv:1804.08617 , 2018.
Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on speech-to-
text. In 2018 IEEE Security and Privacy Workshops (SPW) , pp. 1–7. IEEE, 2018.
Silvia Chiappa, Sbastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment
simulators. International Conference on Learning Representations (ICLR) , 2017.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement
learning in a handful of trials using probabilistic dynamics models. Neural Information Process-
ing Systems (NIPS) , 2018.
Justin Fu, Sergey Levine, and Pieter Abbeel. One-shot learning of manipulation skills with online
dynamics adaptation and neural network priors. Intelligent Robots and Systems (IROS) , 2016.
Adam Gleave, Michael Dennis, Neel Kant, Cody Wild, Sergey Levine, and Stuart Russell. Adver-
sarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv:1905.10615 , 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial
examples. ICLR , 2015.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks
on neural network policies. arXiv preprint arXiv:1702.02284 , 2017.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In
Empirical Methods in Natural Language Processing (EMNLP), Outstanding paper award , 2017.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble
trust-region policy optimization. arXiv preprint arXiv:1802.10592 , 2018.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa,
David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv
preprint arXiv:1509.02971 , 2015.
Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu, and Min Sun. Tac-
tics of adversarial attack on deep reinforcement learning agents. arXiv preprint arXiv:1703.06748 ,
2017.
V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle-
mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level
control through deep reinforcement learning. Nature , 518(7540):529, 2015.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow,
and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud-
den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv
preprint arXiv:1801.00690 , 2018.
Jonathan Uesato, Ananya Kumar, Csaba Szepesvari, Tom Erez, Avraham Ruderman, Keith Ander-
son, Nicolas Heess, Pushmeet Kohli, et al. Rigorous agent evaluation: An adversarial approach
to uncover catastrophic failures. arXiv preprint arXiv:1812.01647 , 2018.
Yuh-Shyang Wang, Tsui-Wei Weng, and Luca Daniel. Veriﬁcation of neural network control policy
under persistent adversarial perturbation. arXiv preprint arXiv:1908.06353 , 2019.
11
Published as a conference paper at ICLR 2020
A A PPENDIX
A.1 M ORE ILLUSTRATION ON FIGURE 3
The meaning of Fig 3 is to show how the accuracy of the learned models affects our proposed
technique:
1. we ﬁrst learned 3 models with 3 different number of samples: 5e5, 1e6, 5e6 and we found
that with more training samples (e.g. 5e6, equivalently 5000 episodes), we are able to learn
a more accurate model than the one with 5e5 training samples;
2. we plot the attack results of total loss for our technique with 3 learned models (denoted as
PGD, num train) as well as the baselines (randU, randB, Flip) on 10 different runs (initial-
izations).
We show with the more accurate learned model (5e6 training samples), we are able to achieve a
stronger attack (the total losses are at the order of 50-200 over 10 different runs) than the less
accurate learned model (e.g. 5e5 training samples). However, even with a less accurate learned
model, the total losses are on the order of 400-700, which already outperforms the best baselines by
a margin of 1.3-2 times. This result in Fig 3 also suggest that a very accurate model isn’t necessarily
needed in our proposed method to achieve effective attack. Of course, if the learned model is more
accurate, then we are able to degrade agent’s performance even more.
Figure 3: Compare sample size on the Walker.stand in 10 different initialization in the environment.
The x-axis is the kth initialization and the y-axis is the total loss of corresponding initialization.
A.2 M ORE DETAILS ON BASELINE IMPLEMENTATIONS
For the baselines (rand-U and rand-B), the adversary generates 1000 trajectories with random noise
directly and we report the best loss/reward at the end of each episode. The detailed steps are listed
below:
Step 1: The perturbations are generated from a uniform distribution or a bernoulli distribution
within the range [-eps, eps] for each trajectory, and we record the total reward and total
loss for each trajectory from the true environment (the MuJoCo simulator)
Step 2: Take the best (lowest) total reward/loss among 1000 trajectories and report in Table 1 and
2.
12
Published as a conference paper at ICLR 2020
We note that here we assume the baseline adversary has an ”unfair advantage” since they have access
to the true reward (and then take the best attack result among 1000 trials), whereas our techniques
do not have access to this information. Without this advantage, the baseline adversaries (rand-B,
rand-U) may be weaker if they use their learned model to ﬁnd the best attack sequence. In any case,
Table 1 and 2 demonstrate that our proposed attack can successfully uncover vulnerabilities of deep
RL agents while the baselines cannot.
For the baseline ﬂip, we add the perturbation (with the opposite sign and magnitude ) on the original
state/action and project the perturbed state/action are within its limits.
A.3 S CORE OF LEARNED POLICY WITHOUT ATTACKS
We use default total timesteps = 1000, and the maximum total reward is 1000. We report the total
reward of the d4pg agents used in this paper below. The agents are well-trained and have total
reward close to 1000, which outperforms agents trained by other learning algorithms on the same
tasks (e.g. DDPG, A3C in Sec 6 (Tassa et al., 2018); PPO in Sec 5 (Abdolmaleki et al., 2018)),
and thus the agents in this paper can be regarded as state-of-the-art RL agents for these continuous
control domain tasks. The attack results in Table 1 and 2 in our manuscript are hence suggested to
be representative.
Domain Task Total reward
Walker stand 994
Walker walk 987
Humanoid stand 972
Humanoid walk 967
Cartpole balance 1000
Cartpole swingup 883
Fish upright 962
A.4 A DDITIONAL EXPERIMENTS ON ABLATION STUDY
Table 5: Cartpole balance (action perturbation)
Total loss mean std med min max
Ours,T= 20 2173 51 2189 2087 2239
Ours,T= 100 1951 113 1924 1851 2192
rand-U 4000 0 4000 4000 4000
rand-B 3999 0 3999 3999 3999
ﬂip 3046 1005 3074 2060 3999
13

Tsui-Wei Weng
Krishnamurthy (Dj) Dvijotham*
Jonathan Uesato*
Kai Xiao*
Sven Gowal*
Robert Stanforth*
Pushmeet Kohli