Why Can’t You Do That, HAL? Explaining Unsolvability of Planning Tasks.

Why Can’t You Do That HAL?
Explaining Unsolvability of Planning Tasks
Sarath Sreedharan1, Siddharth Srivastava1, David Smith ,Subbarao Kambhampati1
1CIDSE, Arizona State University, Tempe, AZ 85281 USA
ssreedh3@asu.edu, siddharths@asu.edu, david.smith@psresearch.xyz, rao@asu.edu
Abstract
Explainable planning is widely accepted as a pre-
requisite for autonomous agents to successfully
work with humans. While there has been a lot of
research on generating explanations of solutions to
planning problems, explaining the absence of so-
lutions remains a largely open and under-studied
problem, even though such situations can be the
hardest to understand or debug. In this paper, we
show that hierarchical abstractions can be used
to efﬁciently generate reasons for unsolvability of
planning problems. In contrast to related work on
computing certiﬁcates of unsolvability, we show
that our methods can generate compact, human-
understandable reasons for unsolvability. Empirical
analysis and user studies show the validity of our
methods as well as their computational efﬁcacy on
a number of benchmark planning domains.
1 Introduction
The ability to explain the rationale behind a decision is widely
seen as one of the basic skills needed by an autonomous agent
to truly collaborate with humans. At the very least we would
want our autonomous assistants to be capable of explaining
why a particular action/plan was chosen to achieve some ob-
jective and be able to explain why they consider some objec-
tives to be unachievable. For example, consider an automated
taxi scheduling system. A user asks for a taxi to pick up her
and three of her friends and the service comes back by say-
ing that it is not possible, and recommends instead using two
different taxis. In this scenario, the user would want to know
why a single taxi can’t pick up all four of them.
Most earlier works in explanation generation for planning
problems have focused on the problem of explaining why
a given plan or action was chosen, but do not address the
problem of explaining the unsolvability of a given planning
problem. The few works that have tried to address unsolv-
ability have mostly looked at generating certiﬁcates or proofs
of unsolvability (cf. [Eriksson, R ¨oger, and Helmert, 2018;
2017 ]) or identify some modiﬁcation of the planning prob-
lem that could make the problem solvable, i.e, an excuse
for the unsolvability of the problem (c.f [G¨obelbecker et al.,
2010 ]). Unfortunately the certiﬁcates/proofs considered bythese works are geared towards automatic veriﬁcation rather
than human understandability and for complex domains ex-
cuses generated by such systems may not be enough to un-
derstand why a problem was unsolvable in the ﬁrst place.
In this paper, we present a new approach for explaining
unsolvability of planning problems that builds on the well
known psychological insight that humans tend to decompose
sequential planning problems in terms of the subgoals they
need to achieve [Donnarumma, Maisto, and Pezzulo, 2016;
Cooper and Shallice, 2006; Simon and Newell, 1971 ]. We
will thus help the user understand the infeasibility of a given
planning problem by pointing out unreachable but necessary
subgoals. For example, in the earlier case, “Holding three pas-
sengers” is a subgoal that is required to reach the goal, but
one that can no longer be achieved due to new city regula-
tions. Thus the system could explain that the taxi can’t hold
more than two passengers at a time (and also notify the user
about the new city ordinance).
Unfortunately, this is not so straightforward, since by the
very nature of the problem, there exist no solutions and hence
its hard to extract meaningful non-trivial subgoals for the
problem. We can ﬁnd a way around this issue by noting the
fact that the user is asking for an explanation for unsolvability
either due to a lack of understanding of the task or because of
limitations in their inferential capabilities. Therefore, we can
try to capture the user’s expectations by considering abstrac-
tions of the given problem. In particular, we use state abstrac-
tions to generate potential solutions and subgoals at higher
levels of abstractions. Such an approach was used by [Sreed-
haran, Srivastava, and Kambhampati, 2018 ]to compute ex-
planations for user queries attuned to the level of expertise of
the user.
In section 3, we present our basic framework and discuss
how we can identify the appropriate level of abstraction and
unachievable subgoals for an unsolvable classical planning
problem. In the real world, a more challenging version of this
problem arises when the user provides plan advice (which
may include temporal preferences) on the type of solutions
expected. In section 4, we will see how explaining unsolv-
ability of planning problems with plan advice (c.f [Myers,
1996 ]) could be seen as establishing unsolvability of planning
problems with additional plan constraints. This is a capabil-
ity that is necessary to capture the fact that these explana-
tions are being provided within the context of a conversation.
Figure 1: A sample abstraction lattice. The lattice consists of mod-
els generated by projecting out rocks or soil samples. Dark blobs
are locations for soil samples, gray objects are rocks, and the goal
is marked in green. The problem is unsolvable in the most concrete
model but solvable in models where rocks are projected out.
The presence of this additional plan advice could either re-
ﬂect cases (1) where the original problem was solvable, but
the user’s requirements (i.e. expressed in the advice) renders
it unsolvable and (2) where the original problem was unsolv-
able and the user presents an outline for a solution in the form
of advice. Even in the second case, by taking into account
the human’s expected solution, we can provide a more tar-
geted explanation. Such additional advice are quite common
within the context of contrastive explanations [Miller, 2018 ],
where these advice specify alternative foils (in this case al-
ternative plans) expected by the user. By supporting refuta-
tion of such advice we also allow the possibility of leverag-
ing our approach for contrastive explanations. For evaluating
our approach, we will present a user study we ran to validate
the usefulness of such explanations for unsolvable problems
(with plan advice) and also note the computational efﬁciency
of our method for some standard planning benchmarks.
2 Background
We will assume that the autonomous agent uses a STRIPS
planning model [Fikes and Nilsson, 1971 ]that can be repre-
sented as a tuple of the form M=hF;A;I;Gi, whereFis
a set of propositional ﬂuents that deﬁne the state space SM
for the model, Agives the set of actions the robot has access
to,Ideﬁnes the initial state and Gthe goal. A state S2SM
corresponds to a unique value assignment for each state ﬂu-
ent and can be represented by the set of ﬂuents that are true
in that state. Each action a2Ais further deﬁned by a tuple
a=hpreca;addsa;delsaiand a plan is deﬁned as an action
sequence of the form =ha1;:::;ani. A plan is said to be
valid forM, if the result of executing a plan from the ini-
tial state satisﬁes the goal (denoted as (I)j=MG). For the
modelM, we will represent the set of all valid plans as M.
Each planning model Malso corresponds to a transition sys-
temT=hSM;I;SG;A;Ti, where SGis the subset of SM
where the goal Gis satisﬁed and TSMASM, such
thathS;a;S0i2T(denoted as Sa !S0) ifSprecaand
a(S) =S0. Each valid plan has a corresponding path in the
transition system from I to some state in SG.In this work, we will be focusing on state and action ab-
stractions induced by projecting out ﬂuents. Thus a model
M2is said to be an abstraction of M1(denoted byM1v
M2) if modelM2can be formed from M1by projecting out
a set of ﬂuents. Formally, M1vM 2if there exists some
PF, such that the transition system of M2is deﬁned
asT2=hSM2;I2;SG2;A;T 2i. Where, for every S2SM1,
there exist a state SnP2SM2,I2=InP,SG2is the
subset of SM2that satisfyG0=GnPand for every transi-
tionhS;a;S0i2T1, there existhSnP;a;S0nPi2T2. We
will denote an abstraction formed by projecting out Pfrom
the modelMasfP(M). An abstraction fP(M)is consid-
ered logically complete if for every such that(I)j=MG,
we have(IfP(M))j=fP(M)GfP(M). In this work, we will
only be looking at logically complete abstractions. For clas-
sical planning models, logically complete abstractions can be
formed by simply removing the abstracted out ﬂuents from
the domain model and problem descriptions.
Sreedharan, Srivastava, and Kambhampati (2018) note that
given a modelMand a set of propositions Pwe can deﬁne
an abstraction lattice, denoted as LM;P=hM;E;`i, where
each model in Mis an abstraction of Mformed by project-
ing out some subset of ﬂuents from P(wherePF) and
M 2 M. There exist an edge hM 1;M2i 2Ewith label
`(M1;M2) =p, ifffpg(M1) =M2, thus this structure
provides a way of capturing the ordering induced by v(where
elements higher up in the lattice are more abstract than the
ones at the bottom). Note that in the most general case, the
lattice need not be complete, that is jMj6= 2jPj. In fact we
do not assume that there is a single most abstract supremum
but rather the structure could have multiple maximal elements
(thus making it a meet semi-lattice rather than a pure lattice).
For convenience, we will treat the abstraction function f
for a given lattice as invertible and use f1
P(M)to repre-
sent the unique concrete node in the lattice that could have
been abstracted (by projecting out P) to generateM. We
will refer to f1
P(M)as the concretization of MforP. Fig-
ure 1 presents a simple conceptualization of an abstraction
lattice for the rover domain. The edges in the lattice corre-
spond to projecting out the presence of rocks or soil samples.
These earlier work uses such abstraction lattices to estimate
the user’s level of understanding of the given task, by search-
ing for the level of abstraction where an incorrect alternative
raised by the user (or foil) could be supported.
3 Our Approach
Before we start discussing the technical details of our ap-
proach, let us look at a sample explanatory scenario.
Example 1. Consider the following scenario where a rover
is tasked with collecting a rock sample and a soil sample
from the region illustrated in Figure 2. The rover can only tra-
verse the region via the waypoints marked on the map and its
maneuverability is affected by the conditions of the terrain.
The rover cannot easily traverse the region between P3 and
P4 without special precautions as the region is quite rocky.
Suppose a mission control operator is also keeping track of
the rover’s plan but may not have access to a map with the
same level of ﬁdelity or may have incomplete knowledge of
the rover’s capabilities. The rover reports to the mission con-
troller that in fact the task can not be solved. The mission
control operator is confused by the rover’s response and could
even ask
“Why don’t you collect the rock sample from P4 and Soil sam-
ple from P7?”
Here if the rover were to explain the reason as to why it
couldn’t achieve the goal, the rover would need to explain
that certain parts of the map are hard to traverse (particularly
the region around the rock sample) and because of this issue
it can never reach the location of the rock sample. Thus the
explanation in this case consist of two distinct parts, infor-
mation about the problem (i.e traversability of certain paths)
and the required subgoal that can no longer be achieved in the
light of this new information. In the proceeding sections, we
will layout our framework and discuss how we could leverage
it to generate such explanations.
The input to our approach thus includes an unsolvable
problemMR=hFR;AR;IR;GRi(in the above example
this would correspond to the complete rover model) and an
abstraction lattice LMR;P=hM;E;`i, where Mrepresents
the space of possible models that could be used to capture the
human’s understanding of the task (i.e by assuming the user
may or may not be aware of the ﬂuents in the set PFR). In
Example 1, Pcould include ﬂuents related traversability of
various paths or ﬂuents related to various rover capabilities.
Given this setting, our method for identifying explanations,
includes the following steps
• Identify the level of abstraction at which the explanation
should be provided (Section 3.1)
• Identify a sequence of necessary subgoals for the given
problem that can be reasoned about at the identiﬁed level
of abstraction (Section 3.2)
• Identify the ﬁrst unachievable subgoal in that sequence
(Section 3.3)
Intuitively, one could understand the three steps mentioned
above as follows. First, identify the level of detail at which un-
solvability of the problem needs to be discussed. The higher
the level of abstraction, the easier the user would ﬁnd it to un-
derstand and reason about the task, but the level of abstraction
should be detailed enough that the problem is actually unsolv-
able there. In most cases, this would mean ﬁnding the highest
level of abstraction where the problem is still unsolvable.
Now even if the system was to present the problem at this
desired level of abstraction, the user may be unable to grasp
the reason for unsolvability. Again, our method involves help-
ing the human in this process by pointing out a necessary
subgoal (i.e., any valid solution to that problem must achieve
the subgoal) that can’t be achieved at the current abstraction
level. Thus the second point relates to the challenge of ﬁnding
a sequence of subgoals (deﬁned by state ﬂuents present at the
explanatory level) for a given problem. In the third step, we
try to identify the ﬁrst subgoal in the sequence that is actually
unsolvable in the given level.
Given our approach, the ﬁnal explanatory message pro-
vided to the user would include model information that brings
their understanding of the task to the required level and infor-
mation on the speciﬁc subgoals (and previous ones that needto be achieved ﬁrst) that can no longer be achieved. In cases
where the unachievable subgoals are hard to understand for-
mulas or large disjunctions, we can also use these subgoals
to produce exemplar plans in the more abstract models and
illustrate their failures alongside the unachievable subgoals.
3.1 Identifying the Minimal Level of Abstraction
Required for Explanation
Following [Sreedharan, Srivastava, and Kambhampati, 2018 ],
we will assume that the human’s understanding of the
task could be approximated by a model MH =
hFH;AH;IH;GHi, such that, the model is part of the ab-
straction lattice (MR@MHandMH2M). While the
earlier work is able to use alternative plan provided by the
user to identify the human model, we instead use the fact that
the user expected the problem to be solvable to identify MH,
i.e.,9;(IMH)j=MHGMH.
We now need to abstract this human model to a level where
the problem is unsolvable (i.e the explanation level) by pro-
viding information about a certain subset of ﬂuents previously
missing from the human model (i.e information on their truth
values in the initial and goal state, and how they affect various
actions etc...). In the case of Example 1, this would include in-
formation on whether various paths are traversable and how
the traversability of a path is a precondition for the robot to
move across it. We will refer to the set of ﬂuents that the hu-
man needs to be informed about as explanatory ﬂuents ( E)
and for Example 1, it will be E=fcantraverse (?x;?y)g.
Deﬁnition 1. Given a human model MH, we deﬁne a set
of propositionsEto be explanatory ﬂuents iff1
E(MH)is
unsolvable, i.e,jf1
E(MH)j= 0.
Unfortunately, this is not an operational deﬁnition as we do
not have access to MH. Instead, we know that MHmust be
part of the lattice, and thus there exists a subset of the max-
imal elements of the lattice (denoted as Mabs) that is more
abstract thanMH. In this section, we will show how the ex-
planatory ﬂuents for models in this subset of Mabswould sat-
isfyMHas well.
The ﬁrst useful property to keep in mind is that if M1is
more concrete than M2then the models obtained by con-
cretizing each model with the same set of ﬂuents would main-
tain this relation (although they may get concretized to the
same model), i.e.,
Proposition 1. Given modelsM1,M2and a set of ﬂuents
0, ifM1vM 2, thenf1
0(M1)vf1
0(M2).
Next, it can be shown that any given set of explanatory
ﬂuents for an abstract model will be a valid explanatory ﬂuent
set for a more concrete model
Proposition 2. Given modelsM1,M2, ifM1vM 2, then
any explanationEforM2must also be an explanation for
M1.
To see why this proposition is true, let’s start from the
fact thatf1
E(M1)vf1
E(M2)and therefore f1
E(M1)
f1
E(M2). From the deﬁnition of explanation we know that
the concretization with respect to explanatory ﬂuents would
render the problem unsolvable (i.e jf1
E(M2)j= 0) and thus
Figure 2: The map for the rover mission planning problem. The rover is required to collect a rock sample and a soil sample and then return to
the original position P1. One of the rock samples is located in rough terrain (gray) that can not be traversed by the rover. The mission control
operator who is monitoring the plan is currently unaware of this detail.
jf1
E(M1))jmust also be empty and hence Eis an explana-
tion forM1.
Deﬁnition 2. Given an abstraction lattice L, letMabsbe its
maximal elements. Then the minimum abstraction set is de-
ﬁned as Mmin=fMjM2 Mabs^jMj>0g.
Note that for any model M12Mmin,MHv M 1,
this means by Proposition 2, any explanation that is valid
for models in Mmin, should leadMHto a node where the
problem is unsolvable. Now we can generate the explanation
(even the optimal one) by searching for a set of ﬂuents that
when introduced to the models M 2 Mmin will render
the problem f1
E(M)unsolvable. In this work, we will
mostly consider the use of uniform cost search to ﬁnd the
least costly set of explanatory ﬂuent, where the cost of each
ﬂuent reﬂects the cost of communicating information about
a particular ﬂuent. In this case, the search state consists of
sets of models (with Mminbeing the initial state), the actions
consist of the various ﬂuent concretizations, the edges of
the lattice deﬁne the successor functions and the goal test
involves verifying whether the problem is solvable in each
possible model in the current state.
3.2 Generating Subgoals of a Given Problem
Note that it would be hard to identify non-trivial subgoals for
the given problem in the node at which the problem was found
to be unsolvable (i.e f1
E(Mmin)) since there are no valid
plans in that model. Fortunately, we can use models more ab-
stract thanf1
E(Mmin)to generate such subgoals. We will
use planning landmarks [Hoffmann, Porteous, and Sebastia,
2004 ]extracted fromM, wherejMj>0, as subgoals. Intu-
itively, state landmarks (denoted as  =h;i) for a model
Mcan be thought of as a partially ordered set of formulas,
where the formulas and the ordering need to be satisﬁed by
every plan that is valid in M. We will only be considering
sound orderings (c.f [Richter, Helmert, and Westphal, 2008 ])
between landmarks, namely, (1) natural orderings (nat) -
nat0, thenmust be true before 0is made true in ev-
ery plan, (2) necessary orderings (nec) - ifnec0then
must be true in the step before 0is made true every timeand (3) greedy necessary orderings (gnec) - ifgnec0
thenmust be true in the step before 0is made true the ﬁrst
time. The landmark formulas may be disjunctive, conjunctive
or atomic landmarks.
Our use of landmarks as the way to identify subgoals is
further justiﬁed by the fact that logically complete abstrac-
tions conserve landmarks. Formally
Proposition 3. Given two models M1andM2, such that
M1vM 2, let1=h1;1iand2=h2;2ibe the
landmarks ofM1andM2respectively. Then for all 1
i;1
j2
1, such that1
i11
j(where1is some sound ordering),
we have2
iand2
jin2, where1
i21
j,1
ij=2
iand
1
jj=2
j.
This is true because 2
i12
jholds over all the plans
that are valid inM2, and therefore must also hold over all
plans inM1. Though inM1these landmark instances may be
captured by more constrained formulas, and additionally M1
may also contain landmarks that were absent from M2. Now
if we can show that in a particular model, a landmark gener-
ated from a more abstract model is unachievable (or the order-
ing from the previous level is unachievable) then 1
becomes
?(thereby meeting the above requirement). Thereafter, for
any model more concrete than M2, the formula correspond-
ing to that landmark must be ?. In other words, if for any
model a landmark is unachievable, then that landmark can’t
be achieved in any models more concrete than the current one.
So given the explanatory level, we can move one level up
in the lattice and make use of any of the well established
landmark extraction methods developed for classical plan-
ning problem to generate a sequence of potential subgoals for
MR.
3.3 Identifying Unachievable Sub-Goals
Now we need to ﬁnd the ﬁrst subgoal from the sequence that
can no longer be achieved in the models obtained by apply-
ing the explanatory ﬂuents ( f1
E(Mmin)) which will then
be presented to the user. For example, in the case of Figure
1, the unachievable subgoal would correspond to satisfying
atrover (5;4)(marked in red in M4).
It is important to note that ﬁnding the ﬁrst unachievable
subgoal is not as simple as testing the achievability of each
subgoal at the abstraction level identiﬁed by methods dis-
cussed in section 3.1. Instead, we need to make sure that
each subgoal is achievable while preserving the order of all
the previous subgoals. To test this we will introduce a new
compilation that allows us to express the problem of testing
achievement of a landmark formula as a planning problem.
Consider a planning model Mand the landmarks  =h;i
extracted from some model M0, whereM@M0. We will
assume that the formulas in are propositional logic for-
mulas over the state ﬂuents and are expressed in DNF. Each
2can be represented as a set of sets of ﬂuents (i.e, =
fc1;:::;ckgand eachciset takes the form ci=fp1;::pmg),
where each set of ﬂuents represent a conjunction over those
ﬂuents. For testing the achievability of any landmark 2,
we make an augmented model M=hF;A;I;Gi,
such that the landmark is achievable iffjMj>0. The
modelMcan be deﬁned as follows: F=F[Fmeta,
whereFmetacontains new meta ﬂuents for each possible
landmark02of the form
•achieved (0)keeps track of a landmark being achieved
and never gets removed
•unset (0)Says that the landmark has not been achieved
yet, usually set true in the initial state unless the land-
mark is true in the initial state
•firsttimeachieved (0)Says that the landmark has
been achieved for the ﬁrst time. This ﬂuent is set true
in the initial state if the landmark is already true there
The new action set A, will contain a copy of each action
inA. For each new action corresponding to a2A, we add
the following new effects to track the achievement of each
landmark
• for each02if the action has existing add effects for
a subset of predicates ^cjfor acj20, then we add the
conditional effects cond 1(0)! fachieved (0)gand
cond 2(0)!ffirsttimeachieved (0)g, where
cond 1(0) =cjn^cj[f^j^2^(^nec0)g[
fachieved (^)j^nat0gand
cond 2(0) =cond 1(0)[f^j^gnec0g[funset ()g
• We add a conditional delete effect to every ac-
tion of the form firsttimeachieved (0)!
(not(firsttimeachieved (0)))
The new goal would be deﬁned as G=
ffirsttimeachieved ()g.
This formulation allows us to test each landmark in the
given sequence and ﬁnd the ﬁrst one that can no longer be
achieved. To ensure completeness, we will return the ﬁnal
goal if all the previously extracted landmarks are still achiev-
able inf1
E(Mmin). Now given an ordered set of landmarks,
we can identify the ﬁrst unsolvable landmark by testing the
solvability of the Ffor each landmark in the order they ap-
pear in the sequence.
Since the above formulation is designed for DNF, we can
generate compilation for cases where the landmarks use ei-ther un-normalized formulas or CNF by converting them ﬁrst
into DNF formulas.
4 Planning Problems with Plan Advice
Let us now discuss how we could extend the methods pre-
sented in earlier sections to cases where the user provides plan
advice. In such cases,the user imposes certain restrictions on
the kind of solution they expect, either as an alternative to
the solution the system may come up with on its own or as
a guide to help the system come up with solutions when it
claims unsolvability.
As pointed out in [Myers, 1996 ], such advice can be
compiled into plan constraints in the original problem. A
number of approaches have been proposed to capture and
represent plan constraints [Bacchus and Kabanza, 2000;
Nau et al., 2001; Kambhampati, Knoblock, and Yang, 1995;
Baier and McIlraith, 2006 ], and each of these representational
choices has its unique strengths and weaknesses. In general,
we can see that these plan constraints as specify a partition-
ing of the space of all valid plans to either acceptable (i.e it
satisﬁes the constraints) or unacceptable. So we can deﬁne,
constraints as follows
Deﬁnition 3. The partition speciﬁed by a constrainton a
given set of plans that is speciﬁed by a membership function
: ![0;1], where is the set of all plans.
We will slightly abuse notation and for a given set of plans
^we will use (0)to denotefj2^^() = 1g(i.e
the subset of 0that satisﬁes the constraint). If we can as-
sume some upper bound on the possible length of plans in
(M)(which is guaranteed when we restrict our attention
to non-redundant plans for standard classical planning prob-
lems), then we can assert that there always exists a ﬁnite state
machine that captures the space of acceptable plans
Proposition 4. Given a constraint and a modelM, there
exists a ﬁnite state automaton F;M=h;SF;M;S0;;Ei,
where is the input alphabet, SF;Mdeﬁnes the FSA states,
S0is the initial state, is the transition function and Eis the
set of accepting states, such that (M) =L(F;M)\M,
whereL(F;M)is the set of strings accepted by F;M.
The existence ofF;Mcan be trivially shown by consider-
ing an FSA that has a path for each unique plan in F;M. We
believe that this formulation is general enough to capture al-
most all of the plan constraint speciﬁcations discussed in the
planning literature, including LTL based speciﬁcations, since
for classical planning problems these formulas are better un-
derstood in terms of LTLf[De Giacomo and Vardi, 2015 ]
which can be compiled into a ﬁnite state automaton.
We can useF;Mto build a new model (M)such that
a plan is valid in (M)if and only if the plan is valid
forMand satisﬁes the given speciﬁcation , i.e., 8;2
(M)iff2(M)
ForM=hF;A;I;Gi, we can deﬁne the new model
(M) =hF;A;I;Gias follows
•F=F[ fin-state- fSgjS2SF;Mg
•A=A[A
•I=I[ fin-state- fS0gg
Figure 3: The graph compares the time taken to generate the expla-
nation for three of the domains for increasing size of lattices.
•G=G[ fin-state- fSgjS2Eg
Aare the new meta actions responsible for simu-
lating the transitions deﬁned by :SF;M!
pow(SF;M). For example, if (S1;a) =fS1;S2g, where
acorresponds to an action in A, then we will have two
new actions a1
S1;a=hpreca[fin-state-fS1gg;addsa[
fin-state-fS2gg;delsa[ fin-state-fS1ggi anda2
S1;a=
hpreca[fin-state-fS1gg;addsa;delsagi. In cases like LTL,
the FSA state transitions may be induced by the satisfaction
of some formula, so the new meta action may have precondi-
tions corresponding to that formula, with no other effects but
changing the ﬂuent corresponding to the state transition.
The above formulation merely points out that there always
exists a way of generating (M)from the given speciﬁcation
andM. For many constraint types, there may exist more
efﬁcient ways of generating models that satisfy the require-
ments of(M).
Once we have access to (M), we should be able to use the
methods discussed in earlier sections to explain unsolvability
of(M)and hence why the constraint isn’t feasible. To facil-
itate such explanation, we will build an abstraction lattice for
the constrained problems LM;Psuch thatP\(FnF) =,
i.e the abstraction lattice only affects the ﬂuents from the orig-
inal problem and not the new ones introduced as part of the
compilation. In fact, we can induce such a lattice by consider-
ing the lattice generated for the original problem and then re-
placing each node in the lattice with the corresponding com-
piled problem, to see why this would induce a valid abstrac-
tion lattice, consider the following property
Proposition 5. Given modelsM1,M2and a constraint
speciﬁcation , ifM1vM 2, then(M1)v(M2).
To see why this is true, just assume that the reverse was
true, that(M2)is not a logically complete abstraction of
(M1). This means that there are plans in (M1)that are
not part of (M2). From the deﬁnition of (M2), we know
that(M2)= M2\L(F). If there exist a 2(M1),
such that62(M2), then62M2. Which meansM16v
M2, hence contradicting our assumptions.
Revisiting Example 1, the question asked by the user could
be seen as an advice, where the corresponding constraint
covers all plans where the rover performs the actions col-
lectrock sample P4 ,collect soilsample P7 , irrespective ofthe exact position and order in which the actions appears in
the plan. More generally, we could think of this plan advice as
being a special case of advice where the user wants to ensure
presence of certain actions in the plan with some partial or-
dering among them (eg: “Why don’t you pickup block B and
then C?”, “Make sure that you have cleared Room1 before
you move on to Room2 and Room3” etc..). Such advice could
be represented as partial plans [Kambhampati, Knoblock, and
Yang, 1995 ], which in general can be captured as a partially
ordered multiset of the form1^=h^A;6i, where ^Ais a mul-
tiset over grounded actions and 6deﬁnes ordering constraints
over these grounded actions. A plan =ha1;:::;aniis said
to be a candidate plan for the given partial plan ^, if there
exists a mapping function :^A![1;jj]that maps each
action ina2^Ato a position in the plan such that a=a(a)
and ifa < b fora;b2^A, then(a)< (b). Such partial
plans can be fairly easily compiled into a classical planning
model (such that it satisﬁes (M)) by extending the com-
pilation methods discussed in [Ramırez and Geffner, 2010 ],
without relying on an intermediate ﬁnite state machine.
The corresponding partial plan for the question speciﬁed
above would be
^=hfcollect rock sample P4;collect soilsample P7g;i
Let us assume that in this case the observer could be un-
aware of certain domain constraints such as the rover’s in-
ability to traverse certain regions on the map the fact that not
all rovers are capable of collecting rock and soil samples or
that they are not always equipped to store these samples. In
this case, possible user models can be captured by a lattice
build using the following ﬂuents P=f(can traverse ?x ?y) ,
(equipped forsoilsample ?r) ,(equipped forrock sample ?r) ,
(store of ?r)g. Now our approach would identify the user need
to be made aware of the fact that not all regions of the map are
equally traversable (i.e inform the user about cantraverse ?x
?y) and how its a precondition for move action), furthermore
given this property the robot can no longer reach the waypoint
4 which is required to complete this task (i.e the unreachable
landmark is (at rover0 waypoint4) ).
5 Evaluations
5.1 User Studies
Our ﬁrst concern with evaluating explanations based on land-
marks was to establish that they constitute meaningful expla-
nations for naive users. As a simple alternative to our explana-
tions, we consider providing to the user a set of potential solu-
tions (generated from a higher level of abstraction) and their
individual failures. For the study, we recruited around 120
master turkers from Amazon’s Mechanical Turk and tested
the following hypotheses
•H1- Users prefer concise explanations over ones that
enumerate a set of possible candidates for a given piece
of plan advice
1We are presenting a simpliﬁed deﬁnition of a partial plan.
The full deﬁnition allows for the representation of more complex
constraints than mere ordering constraints, such as contiguity con-
straints and interval protection constraints.
Domain Name jPjCoverage Average Runtime (secs) Explanation Cost Cost of explaining MR
Blocksworld 4 5/5 8.141 11.6 30.2
Satellite 8 5/5 19.15 6 43.6
Depots 5 5/5 20.229 13 51
Rover 8 5/5 263.635 7.5 15.75
Storage 7 5/5 50.348 20 55.8
Over-Rover8 5/5 2047.360 29.8 92.6
Over-tpp8 5/5 1065.542 842.8 881.2
Bottleneck3 5/5 504.431 60.8 66.2
Table 1: Table showing runtime for explanations generated for standard IPC domains. The explanation costs capture the number of unique
model updates (changes in effects/precondition etc..) corresponding to each explanation
•H2- Users prefer concise explanations that contain in-
formation about unachievable landmarks over ones that
only show the failure of a single exemplary plan
For the hypotheses, we presented the study participants with
a sample dialogue between two people over a logistics plan
to move a package from one location to another. The dia-
logue included a person (named Bob) presenting a plan to
another (named Alice), and Alice asks for an alternative pos-
sibility (i.e speciﬁes a constraint on the solution). Now the
challenge for Bob is to explain why the constrained problem
is unsolvable. For example, in one example Bob presents a
rather convoluted plan that involves the package being trans-
ferred through multiple trucks to a train and then to the ﬁnal
destination. This leads to Alice asking the package to be de-
livered via an airplane.
For H1, in addition to some model information that Bob
was unaware of, the potential explanations included either (a)
the information on the unachievable landmark, (b) landmark
information with the failure details of a speciﬁc exemplary
plan or (c) a set of plans that satisfy the constraints and their
corresponding failures. For the earlier example this meant
Bob explains to Alice the limited availability of Truck fuel
and (a) the impossibility of getting the package to the airport
or (b) the the impossibility of getting the package to the air-
port and a speciﬁc plan (eg: truck1 picks up package moves
to location two then to three ... ) along with its point of failure
(eg:truck1 runs out of fuel when it reaches location three ) or
(c) three example plans involving various trucks trying to get
the package to the airport and their speciﬁc points of failures
(each of which fails at different steps but before reaching the
airport).
For this study, we used 45 participants and each participant
was assigned one of three possible maps for each hypothesis
and was paid $1.25 for 10 mins. We used a control question
to ﬁlter participant responses, so as to ensure their quality.
Out of the 39 remaining responses, we found 94.8% of users
chose to select the more concise explanation (i.e (a) or (b)),
and 51.28% of the users chose explanations that involved just
landmarks.
For H2, we used 75 participants and presented each par-
ticipant with explanations that include (a) just landmark in-
formation, (b) landmark information with failure details of
an exemplary plan and (c) just the exemplary plan failure.
Here participants were paid $1 for 10 mins for H2 as the ex-
planatory options were much simpler. After ﬁltering usingthe control question, we found that out of 60 valid entries
75.4% of participants preferred explanations that included
landmark information ((a) or (b)) and 44.2% wanted both
landmarks and exemplary plan (i.e (b)). The supplementary
ﬁle at http://bit.ly/2HQ5sTv contains more details
on the study setup.
5.2 Empirical Studies
In this section, we will present the results of an empirical
evaluation of the computational characteristics of our ap-
proach. One big concern with the methods discussed in this
work is the fact that they involve solving multiple planning
problems. Thus we were interested in identifying the run-
time for generating explanations on a set of standard planning
benchmarks.
To evaluate our methods, we considered eight planning do-
mains and chose ﬁve problem instances for each of the do-
mains. For each domain, we used a subset of the domain
predicates to generate the abstraction lattice (i.e we set the
subset as the set of ﬂuents Pused to deﬁne the lattice). The
ﬁrst ﬁve domains and their problem instances consisted of
standard IPC domains and problem instances used in previ-
ous IPC competitions [International Planning Competition,
2011 ]. Each problem instance was made unsolvable by in-
cluding plan constraints that avoid a speciﬁc landmark of the
original problem. The constraints were coded using domain
control programs [Baier, Fritz, and McIlraith, 2007 ]of the
form
while :^ :(goal completed)
do any
done
Whereis the landmark formula and (goal completed)
is the goal ﬂuent (generated by a new goal action whose
preconditions are the original goals of the problem). The con-
straints ensure that any valid plan must avoid the landmark
and thereby rendering it unsolvable. The next three domains
were selected from the set used for the 2016 unsolvability
competition [Unsolvability International Planning Competi-
tion, 2016 ](these domains are marked with an asterix in the
results table). All landmarks were generated using the fast-
downward implementation of [Keyder, Richter, and Helmert,
2010 ].
Table 1 presents the results of our tests on these domains. It
shows the number of ﬂuents used to generate the lattice ( jPj),
the coverage (i.e the number of problems that were solved
within the timeout limit of 100 minutes), the average runtime,
the cost of the generated explanations and the cost of present-
ing the most concrete model to the user. For each scenario,
we created a complete lattice for all the ﬂuents considered
for abstraction (i.e jMj= 2jPj). The cost of the explanation
captures the amount of information to be provided to the user
as part of the explanation. This could include information re-
garding the various explanatory ﬂuents and is here captured
roughly by the number of places within the domain deﬁnition
where these ﬂuents appear. The cost also reﬂects the inferen-
tial overhead demanded from the user (since providing more
information translates to the user needing to understand the
domain at a much more concrete level).
For a sample explanation, consider the overconstrained
rover domain, where the rovers’ actions are limited by their
energy levels and the energy of the rover isn’t enough to ﬁn-
ish the task. In one of the instances where the rover energy
level is at 33 and the original problem had a goal consisting
of eight propositions (each referring to the need for commu-
nicating a particular soil sample, rock sample or sending an
image for different objectives), our approach was able to iden-
tify that the user needs to understand ﬂuents related to energy
((energy ?x ?y) and(energycost ?x ?y ?z) ) and identiﬁed two
subgoals out of the eight that it could not achieve.
Figure 3 presents the variations in average runtime for three
of the domains as the size of the lattice were increased (the X-
axis represents the number of ﬂuents that were used to build
the lattice and Y the runtime in seconds). Note that, in gen-
eral, the runtime increases as the lattice size increase due to
the increase in the search space, but in all three domains there
are points where the runtime decrease when the lattice size in-
creases. This is expected since with an increase in the size of
lattice, the planning problems whose unsolvability are being
tested becomes simpler.
6 Related Work
As discussed earlier, our methods for identifying the level of
explanations are based on the expertise level modeling ap-
proaches introduced in [Sreedharan, Srivastava, and Kamb-
hampati, 2018 ]. These two works are quite closely connected
and in fact, the contrastive explanations of the type studied in
the earlier paper, where the user presents alternative plans (i.e
the foils for the explanations) that are then refuted by the sys-
tem, is a special case of our approach for handling problems
with plan advice. The problems studied in that earlier paper
can be thought of as capturing cases where the advice only
allows for a single plan. Also, one could argue that people
would be more comfortable giving advices as foils rather than
full plans. Part of our explanations also try to reveal to the
user information about the current task that was previously
unknown to them. Thus our methods could also be under-
stood as an example of explanation as model-reconciliation
[Chakraborti et al., 2017 ]. Since our methods use abstrac-
tions, our approach doesn’t make too many demands on the
inferential capabilities of the user and hence can be applied
to much larger and more complex domains.
Another closely related direction has been the work done
on explaining unsynthesizability of hybrid controllers for agiven set of high-level task speciﬁcations [Raman and Kress-
Gazit, 2013 ]. The work tries to identify the subformulas of
the given speciﬁcation that lead to the unsynthesizability.
This particular approach is speciﬁc to the planning frame-
work detailed in [Finucane, Jing, and Kress-Gazit, 2010 ]and
the objective of the work parallels the goals of work like
[G¨obelbecker et al., 2010 ].
Outside of explanation generation, the work done in the
model checking community is closely related to our current
problem [Grumberg and Veith, 2008 ]. In fact, the hierarchical
approach to identifying a model that can invalidate the given
foil speciﬁcation, can be seen as a special case of the CE-
GAR based methods studied in the model-checking commu-
nity[Clarke et al., 2000 ]. Most work in this ﬁeld focuses on
developing methods for identifying whether a given program
meets some speciﬁcations and failures to meet speciﬁcation
are generally communicated via counterexamples.
Another related problem is that of identifying whether a
given problem is unsolvable. In our setting, we assume that
the system is capable of correctly identifying whether a given
problem is unsolvable or not and in general this can be a time
consuming process. Thankfully the problem of efﬁciently
identifying whether a given planning problem is unsolvable is
an active research area (cf. [Steinmetz and Hoffmann, 2017;
Kolobov, Weld, and others, 2010 ]) and solutions to this prob-
lem can be easily leveraged by our approach to improve the
overall efﬁciency of the system.
7 Conclusion and Future Directions
The work presented in this paper investigates the problem of
generating explanations for unsolvability of a given planning
problem. We also saw how the same methods apply when
dealing with problems with plan constraints. In addition to
extending these methods to more expressive domains, an in-
teresting extension would be to try tackling cases where the
current problem is solvable but all the solutions are too ex-
pensive. While this additional cost threshold could be seen as
a constraint, the setting becomes a lot more interesting when
the action costs are affected by the abstractions (c.f state de-
pendent costs [Geißer, Keller, and Mattm ¨uller, 2016 ]). With
respect to contrastive explanations, this would correspond to
cases where the alternative posed by the user is more expen-
sive than the plan proposed by the robot. Finally, an obvious
challenge to fully realize this method in practical scenarios
is to develop methods to convert user questions to plan con-
straints. Methods like [Tenorth et al., 2010 ]can be used to
convert natural language statements to constraints like partial
plans. Expert users can also directly write LTL and procedu-
ral programs as a way of interrogating the system.
Acknowledgments
This research is supported in part by the ONR grants
N00014-16-1-2892, N00014-18-1-2442, N00014-18-1-2840,
the AFOSR grant FA9550-18-1-0067, the NASA grant
NNX17AD06G and NSF grant 1844325. We also thank the
IJCAI reviewers for their comments and feedback.
References
[Bacchus and Kabanza, 2000 ]Bacchus, F., and Kabanza, F.
2000. Using temporal logics to express search control
knowledge for planning. Artiﬁcial Intelligence 116(1-
2):123–191.
[Baier and McIlraith, 2006 ]Baier, J. A., and McIlraith, S. A.
2006. Planning with ﬁrst-order temporally extended goals
using heuristic search. In AAAI , 788–795.
[Baier, Fritz, and McIlraith, 2007 ]Baier, J. A.; Fritz, C.; and
McIlraith, S. A. 2007. Exploiting procedural domain con-
trol knowledge in state-of-the-art planners. In ICAPS .
[Chakraborti et al., 2017 ]Chakraborti, T.; Sreedharan, S.;
Zhang, Y .; and Kambhampati, S. 2017. Plan explana-
tions as model reconciliation: Moving beyond explanation
as soliloquy. In IJCAI .
[Clarke et al., 2000 ]Clarke, E.; Grumberg, O.; Jha, S.; Lu,
Y .; and Veith, H. 2000. Counterexample-guided abstrac-
tion reﬁnement. In International Conference on Computer
Aided Veriﬁcation , 154–169. Springer.
[Cooper and Shallice, 2006 ]Cooper, R. P., and Shallice, T.
2006. Hierarchical schemas and goals in the control of
sequential behavior. Psychological Review .
[De Giacomo and Vardi, 2015 ]De Giacomo, G., and Vardi,
M. Y . 2015. Synthesis for ltl and ldl on ﬁnite traces. In
IJCAI , volume 15, 1558–1564.
[Donnarumma, Maisto, and Pezzulo, 2016 ]Donnarumma,
F.; Maisto, D.; and Pezzulo, G. 2016. Problem solving
as probabilistic inference with subgoaling: explaining
human successes and pitfalls in the tower of hanoi. PLoS
computational biology 12(4):e1004864.
[Eriksson, R ¨oger, and Helmert, 2017 ]Eriksson, S.; R ¨oger,
G.; and Helmert, M. 2017. Unsolvability certiﬁcates for
classical planning. In ICAPS .
[Eriksson, R ¨oger, and Helmert, 2018 ]Eriksson, S.; R ¨oger,
G.; and Helmert, M. 2018. A proof system for unsolv-
able planning tasks. In ICAPS .
[Fikes and Nilsson, 1971 ]Fikes, R. E., and Nilsson, N. J.
1971. Strips: A new approach to the application of the-
orem proving to problem solving. Artiﬁcial intelligence
2(3-4):189–208.
[Finucane, Jing, and Kress-Gazit, 2010 ]Finucane, C.; Jing,
G.; and Kress-Gazit, H. 2010. Ltlmop: Experimenting
with language, temporal logic and robot control. In IROS ,
1988–1993. IEEE.
[Geißer, Keller, and Mattm ¨uller, 2016 ]Geißer, F.; Keller, T.;
and Mattm ¨uller, R. 2016. Abstractions for planning with
state-dependent action costs. In ICAPS .
[G¨obelbecker et al., 2010 ]G¨obelbecker, M.; Keller, T.; Eye-
rich, P.; Brenner, M.; and Nebel, B. 2010. Coming up with
good excuses: What to do when no plan can be found. In
ICAPS .
[Grumberg and Veith, 2008 ]Grumberg, O., and Veith, H.
2008. 25 years of model checking: history, achievements,
perspectives , volume 5000. Springer.[Hoffmann, Porteous, and Sebastia, 2004 ]Hoffmann, J.;
Porteous, J.; and Sebastia, L. 2004. Ordered landmarks in
planning. JAIR 22:215–278.
[International Planning Competition, 2011 ]International
Planning Competition. 2011. IPC Competition Domains.
https://goo.gl/i35bxc .
[Kambhampati, Knoblock, and Yang, 1995 ]Kambhampati,
S.; Knoblock, C. A.; and Yang, Q. 1995. Planning as
reﬁnement search: A uniﬁed framework for evaluating
design tradeoffs in partial-order planning. Artiﬁcial
Intelligence 76(1):167–238.
[Keyder, Richter, and Helmert, 2010 ]Keyder, E.; Richter,
S.; and Helmert, M. 2010. Sound and complete landmarks
for and/or graphs. In ECAI .
[Kolobov, Weld, and others, 2010 ]Kolobov, A.; Weld, D.;
et al. 2010. Sixthsense: Fast and reliable recognition of
dead ends in mdps. In AAAI .
[Miller, 2018 ]Miller, T. 2018. Explanation in artiﬁcial in-
telligence: Insights from the social sciences. Artiﬁcial In-
telligence .
[Myers, 1996 ]Myers, K. L. 1996. Advisable planning sys-
tems. Advanced Planning Technology 206–209.
[Nau et al., 2001 ]Nau, D.; Cao, Y .; Lotem, A.; and Munoz-
Avila, H. 2001. The shop planning system. AI Magazine
22(3):91.
[Raman and Kress-Gazit, 2013 ]Raman, V ., and Kress-
Gazit, H. 2013. Towards minimal explanations of
unsynthesizability for high-level robot behaviors. In
IROS , 757–762. IEEE.
[Ramırez and Geffner, 2010 ]Ramırez, M., and Geffner, H.
2010. Probabilistic plan recognition using off-the-shelf
classical planners. In AAAI , 1121–1126.
[Richter, Helmert, and Westphal, 2008 ]Richter, S.; Helmert,
M.; and Westphal, M. 2008. Landmarks revisited. In
AAAI , volume 8, 975–982.
[Simon and Newell, 1971 ]Simon, H. A., and Newell, A.
1971. Human problem solving: The state of the theory
in 1970. American Psychologist 26(2):145.
[Sreedharan, Srivastava, and Kambhampati, 2018 ]
Sreedharan, S.; Srivastava, S.; and Kambhampati, S.
2018. Hierarchical expertise-level modeling for user
speciﬁc contrastive explanations. In IJCAI .
[Steinmetz and Hoffmann, 2017 ]Steinmetz, M., and Hoff-
mann, J. 2017. Search and learn: On dead-end detectors,
the traps they set, and trap learning. In IJCAI , 4398–4404.
[Tenorth et al., 2010 ]Tenorth, M.; Nyga, D.; Beetz, M.; et al.
2010. Understanding and executing instructions for every-
day manipulation tasks from the world wide web. In ICRA .
[Unsolvability International Planning Competition, 2016 ]
Unsolvability International Planning Competi-
tion. 2016. IPC Competition Domains. https:
//unsolve-ipc.eng.unimelb.edu.au/ .

Sarath Sreedharan
Siddharth Srivastava
David Smith
Subbarao Kambhampati


Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient.

Robust Multi-Agent Reinforcement Learning
via Minimax Deep Deterministic Policy Gradient
Shihui LiyYi WuzXinyue Cui§Honghua Dong§Fei FangyStuart Russellz
yCarnegie Mellon University {shihuil,feifang}@cmu.edu
zUniversity of California, Berkeley {jxwuyi,russell}@eecs.berkeley.edu
§Tsinghua University {cuixy14,dhh14}@mails.tsinghua.edu.cn
Abstract
Despite the recent advances of deep reinforcement learning
(DRL), agents trained by DRL tend to be brittle and sensitive
to the training environment, especially in the multi-agent sce-
narios. In the multi-agent setting, a DRL agent’s policy can
easily get stuck in a poor local optima w.r.t. its training part-
ners – the learned policy may be only locally optimal to other
agents’ current policies. In this paper, we focus on the problem
of training robust DRL agents with continuous actions in the
multi-agent learning setting so that the trained agents can still
generalize when its opponents’ policies alter. To tackle this
problem, we proposed a new algorithm, MiniMax Multi-agent
Deep Deterministic Policy Gradient (M3DDPG) with the fol-
lowing contributions: (1) we introduce a minimax extension of
the popular multi-agent deep deterministic policy gradient al-
gorithm (MADDPG), for robust policy learning; (2) since the
continuous action space leads to computational intractability
in our minimax learning objective, we propose Multi-Agent Ad-
versarial Learning (MAAL) to efﬁciently solve our proposed
formulation. We empirically evaluate our M3DDPG algorithm
in four mixed cooperative and competitive multi-agent envi-
ronments and the agents trained by our method signiﬁcantly
outperforms existing baselines.
1. Introduction
Most real-world problems involve interactions between mul-
tiple agents and the complexity of problem increases sig-
niﬁcantly when the agents co-evolve together. Thanks to
the recent advances of deep reinforcement learing (DRL)
on single agent scenarios, which led to successes in play-
ing Atari game (Mnih et al .2015), playing go (Silver
et al .2016) and robotics control (Levine et al .2016), it
has been a rising trend to adapt single agent DRL algo-
rithms to multi-agent learning scenarios and many works
have shown great successes on a variety of problems, in-
cluding automatic discovery of communication and lan-
guage (Sukhbaatar, Fergus, and others 2016; Mordatch
and Abbeel 2017), multiplayer games (Peng et al .2017a;
OpenAI 2018), trafﬁc control (Wu et al .2017) and the analy-
sis of social dilemmas (Leibo et al. 2017).
The critical challenge when adapting classical single agent
DRL algorithms to multi-agent setting is the training instabil-
Copyright c
2019, Association for the Advancement of Artiﬁcial
Intelligence (www.aaai.org). All rights reserved.ity issue: as training progresses, each agent’s policy is chang-
ing and therefore the environment becomes non-stationary
from the perspective of any individual agent (in a way that is
not explainable by changes in the agent’s own policy). This
non-stationary problem can cause signiﬁcant problems when
directly applying the single agent DRL algorithms, for exam-
ple, the variance of the policy gradient can be exponentially
large when the number of agents increases (Lowe et al .2017).
To handle this instability issue, recent works, such as the
counterfactual multi-agent policy gradients (Foerster et al .
2017) and the Multi-Agent Deep Deterministic Policy Gra-
dient (MADDPG) (Lowe et al .2017), proposed to utilized a
centralized critic within the actor-critic learning framework
to reduce the variance of policy gradient.
Despite the fact that using a centralized critic stabilizes
training, the learned policies can still be brittle and sensi-
tive to its training partners and converge to a poor local
mode. This is particularly severe for competitive environ-
ments: when the opponents alter their policies during testing,
the performance of the learned policies can be drastically
worse (Lazaridou, Peysakhovich, and Baroni 2016). Hence,
arobust policy becomes desirable in multi-agent setting: a
well-trained agent should be able behave well in testing when
competing against opponents even with strategies different
from its training partners.
In this work, we focus on robust multi-agent reinforcement
learning with continuous action spaces and propose a novel
algorithm, MiniMax Multi-agent Deep Deterministic Policy
Gradient (M3DDPG) . M3DDPG is a minimax extension1of
the classical MADDPG algorithm (Lowe et al .2017). Its core
idea is that during training, we force each agent to behave
well even when its training opponents response in the worst
way.
Our major contributions are summarized as follow:
We introduce the minimax approach to robust multi-agent
DRL and propose a novel minimax learning objective
based on the MADDPG algorithm;
In order to efﬁciently optimize the minimax learning objec-
tive, we propose an end-to-end learning approach, Multi-
agent Adversarial Learning (MAAL) , which is inspired by
1In fact, we are dealing with gains, i.e., maximizing each agent’s
accumulative reward, so the “minimax” here is essentially “max-
imin”. We keep the term “minimax” to be consistent with literature.
theadversarial training (Goodfellow, Shlens, and Szegedy
2014) technique2.
We empirically evaluate our proposed M3DDPG algorithm
on four mixed cooperative and competitive environments
and the agents trained by M3DDPG outperform baseline
policies on all these environments.
In the rest of the paper, we will ﬁrstly present related works
in section 2. Notations and standard algorithms are described
in section 3. Our main algorithm, M3DDPG, is introduced in
section 4. Experimental results are in section 5.
2. Related Work
Multi-agent reinforcement learning (Littman 1994) has been
a long-standing ﬁeld in AI (Hu, Wellman, and others 1998;
Busoniu, Babuska, and De Schutter 2008). Recent works in
DRL use deep neural networks to approximately represent
policy and value functions. Inspired by the success of DRL
in single-agent settings, many DRL-based multi-agent learn-
ing algorithms have been proposed. Forester et al. (2016b)
and He et al. (2016) extended the deep Q-learning to multi-
agent setting; Peng et al. (2017a) proposed a centralized pol-
icy learning algorithm based on actor-critic policy gradient;
Forester et al. (2016a) developed a decentralized multi-agent
policy gradient algorithm with centralized baseline; Lowe et
al. (2017) extended DDPG to multi-agent setting with a cen-
tralized Q function; Wei et al. (2018) and Grau-Moya (2018)
proposed multi-agent variants of the soft-Q-learning algo-
rithm (Haarnoja et al .2017); Yang et al. (2018) focused on
multi-agent reinforcement learning on a very large population
of agents. Our M3DDPG algorithm is built on top of MAD-
DPG and inherits the decentralized policy and centralized
critic framework.
Minimax is a fundamental concept in game theory and
can be applied to general decision-making under uncertainty,
prescribing a strategy that minimizes the possible loss for
a worst case scenario (Osborne and others 2004). Minimax
was ﬁrstly introduced to multi-agent reinforcement learning
as minimax Q-learning by Littman (1994). More recently,
some works combine the minimax framework and the DRL
techniques to ﬁnd Nash equilibrium in two player zero-sum
games (Foerster et al .2018; Pérolat et al .2016; Grau-Moya,
Leibfried, and Bou-Ammar 2018). In our work, we utilize
the minimax idea for the purpose of robust policy learning.
Robust reinforcement learning was originally introduced
by Morimoto et al. (2005) considering the generalization
ability of the learned policy in the single-agent setting. This
problem is also studied recently with deep neural networks,
such as adding random noise to input (Tobin et al .2017) or
dynamics (Peng et al .2017b) during training. Besides adding
random noise, some other works implicitly adopt the mini-
max idea by utilizing the “worst noise” (Pinto et al .2017;
Mandlekar et al .2017). These works force the learned pol-
icy to work well even under the worst case perturbations
and are typically under the name of “adversarial reinforce-
ment learning”, despite the fact that the original adversarial
2The connection between MAAL and adversarial training will
be discussed in details at the end of section 4.reinforcement learning problem was introduced in the set-
ting of multi-agent learning (Uther and Veloso 1997). In our
M3DDPG algorithm, we focus on the problem of learning
polices that is robust to opponents with different strategies.
Within the minimax framework, ﬁnding the worst case
scenario is a critical component. Lanctot et al. (2017) pro-
posed an iterative approach that alternatively computes the
best response policy while ﬁxes the other. Gao et al. (Gao,
Mueller, and Hayward 2018) replace “mean” in the tempo-
ral difference learning rule with “minimum”. In our work,
we proposed MAAL, which is a general, efﬁcient and fully
end-to-end learning approach. MAAL is motivated by ad-
versarial training (Goodfellow, Shlens, and Szegedy 2014)
and suitable for arbitrary number of agents. The core idea
of MAAL is approximating the minimization in our min-
max objective by a single gradient descent step. The idea of
one-step-gradient approximation was also explored in meta-
learning (Finn, Abbeel, and Levine 2017).
3. Background and Preliminary
In this section, we describe our problem setting and the stan-
dard algorithms. Most of the deﬁnitions and notations follow
the original MADDPG paper (Lowe et al. 2017).
Markov Games
We consider a multi-agent extension of Markov decision pro-
cesses (MDPs) called partially observable Markov games
(Littman 1994). A Markov game for Nagents is deﬁned by
a set of statesSdescribing the possible conﬁgurations of all
agents, a set of actions A1;:::;ANand a set of observations
O1;:::;ONfor each agent. To choose actions, each agent
iuses a stochastic policy i:OiAi7![0;1]param-
eterized byi, which produces the next state according to
the state transition function T:SA 1:::AN7!S .
Each agentiobtains rewards as a function of the state and
agent’s action ri:SAi7!R, and receives a private
observation correlated with the state oi:S7!Oi. The ini-
tial states are determined by a distribution :S7! [0;1].
Each agentiaims to maximize its own total expected return
Ri=PT
t=0
trt
iwhere
is a discount factor and Tis the
time horizon.
To minimize notation, in the following discussion we will
often omitfrom the subscript of .
Q-Learning and Deep Q-Networks (DQN)
Q-Learning and DQN (Mnih et al .2015) are popular methods
in reinforcement learning and have been previously applied
to multi-agent settings (Foerster et al .2016a; Tesauro 2004).
Q-Learning makes use of an action-value function for policy
asQ(s;a) =E[Rjst=s;at=a]. This Q function
can be recursively rewritten as Q(s;a) =Es0[r(s;a) +

Ea0[Q(s0;a0)]]. DQN learns the action-value function
Qcorresponding to the optimal policy by minimizing the
loss:
L() =Es;a;r;s0[(Q(s;aj)y)2]; (1)
wherey=r+
max
a0Q(s0;a0):
Qis a target Q function whose parameters are periodically
updated with the most recent , which helps stabilize learning.
Another crucial component of stabilizing DQN is the use of
an experience replay buffer Dcontaining tuples (s;a;r;s0).
Q-learning algorithm is most suitable for DRL agents with
discrete action spaces.
Policy Gradient (PG) Algorithms
Policy gradient methods is another popular choice for a va-
riety of RL tasks. Let denote discounted state visitation
distribution for a policy . The main idea of PG is to directly
adjust the parameters of the policy in order to maximize the
objectiveJ() =Es;a[R]by taking steps in the direc-
tion ofrJ(). Using the Q function deﬁned previously, the
gradient of the policy can be written as (Sutton et al .2000):
rJ() =Es;a[rlog(ajs)Q(s;a)];(2)
wherepis the state distribution. The policy gradient theorem
has given rise to several practical algorithms, which often
differ in how they estimate Q. For example, one can simply
use a sample return Rt=PT
i=t
itri, which leads to the
REINFORCE algorithm (Williams 1992). Alternatively, one
could learn an approximation of the true action-value function
Q(s;a)called the critic and leads to a variety of actor-critic
algorithms (Sutton and Barto 1998).
Deterministic Policy Gradient (DPG) Algorithms
DPG algorithms extends the policy gradient algorithm to
deterministic policies :S7!A (Silver et al .2014). In
particular, under certain conditions we can write the gradient
of the objective J() =Es[R(s;a)]as:
rJ() =EsD[r(s)raQ(s;a)ja=(s)]; (3)
whereDis the replay buffer. Since this theorem relies on
raQ(s;a), it requires the action space A(and thus the
policy) be continuous.
Deep deterministic policy gradient (DDPG) (Lillicrap et
al.2015) is a variant of DPG where the policy and critic
Qare approximated with deep neural networks. DDPG
is an off-policy algorithm, and samples trajectories from
a replay buffer of experiences that are stored throughout
training. DDPG also makes use of a target network, as in
DQN (Mnih et al. 2015).
Multi-Agent Deep Deterministic Policy Gradient
Directly applying single-agent RL algorithms to the multi-
agent setting by treating other agents as part of the envi-
ronment is problematic as the environment appears non-
stationary from the view of any one agent, violating Markov
assumptions required for convergence. Particularly, this non-
stationary issue is more severe in the case of DRL with neural
networks as function approximators. The core idea of the
MADDPG algorithm (Lowe et al .2017) is learning a central-
ized Q function for each agent which conditions on global
information to alleviate the non-stationary problem and stabi-
lize training.
More concretely, consider a game with Nagents with
policies parameterized by =f1;:::;Ng, and let=f1;:::;Ngbe the set of all agents’ policies. Then we can
write the gradient of the expected return for agent iwith
policyi,J(i) =E[Ri]as:
riJ(i) =
Ex;aD[rii(oi)raiQ
i(x;a1;:::;aN)jai=i(oi)];
(4)
HereQ
i(x;a1;:::;aN)is acentralized action-value func-
tionthat takes as input the actions of all agents, a1;:::;aN,
in addition to some state information x(i.e., x=
(o1;:::;oN)) , and outputs the Q-value for agent i. Let
x0denote the next state from xafter taking actions
a1;:::;aN. The experience replay buffer Dcontains the tu-
ples(x;x0;a1;:::;aN;r1;:::;rN), recording experiences
of all agents. The centralized action-value function Q
iis
updated as:
L(i) =Ex;a;r;x0[(Q
i(x;a1;:::;aN)y)2]; (5)
y=ri+
Q0
i(x0;a0
1;:::;a0
N)
a0
j=0
j(oj);
where0=f0
1;:::;0
Ngis the set of target policies with
delayed parameters 0
i.
Note that the centralized Q function is only used during
training. During decentralized execution, each policy i
only takes local information oito produce an action.
4. Minimax Multi-Agent Deep Deterministic
Policy Gradient (M3DDPG)
In this section, we introduce our proposed new algorithm,
Minimax Multi-agent Deep Deterministic Policy Gradient
(M3DDPG) , which is built on top of the MADDPG algo-
rithm and particularly designed to improve the robustness
of learned policies. Our M3DDPG algorithm contains two
major novel components:
Minimax Optimization Motivated by the minimax concept
in game theory, we introduce minimax optimization into
the learning objective;
Multi-Agent Adversarial Learning The continuous action
space results in computational intractability issue when
optimizing our proposed minimax objective. Hence, we
propose Multi-Agent49 Adversarial Learning (MAAL) to
solve this optimization problem.
Minimax Optimization
In multi-agent RL, the agents’ policies can be very sensitive
to their learning partner’s policy. Particularly in competitive
environments, the learned policies can be brittle when the
opponents alter their strategies. For the purpose of learning
robust policies, we propose to update policies considering
theworst situation : during training, we optimize the accu-
mulative reward for each agent iunder the assumption that
all other agents acts adversarially. This yields the minimax
learning objective maxiJM(i)where
JM(i) =Es[Ri]
= min
at
j6=iEs"TX
t=0
tri(st;at
1;:::;at
N)
at
i=(ot
i)#
(6)
=Es0"
min
a0
j6=iQ
M;i(s0;a0
1;:::;a0
N)
a0
i=(o0
i)#
:(7)
Critically, in Eq. 6, state st+1at timet+ 1depends not only
on the dynamics and the action i(ot
i)but also on all the
previous adversarial actions at0
j6=iwitht0t. In Eq. 7, we
derive the modiﬁed Q function Q
M(s;a1;:::;aN), which is
naturally centralized and can be rewritten in a recursive form
Q
M;i(s;a1;:::;aN) =ri(s;a1;:::;aN)+

Es0"
min
a0
j6=iQ
M;i(s0;a0
1;:::;a0
N)
a0
i=i(s0)#
:(8)
Importantly, Q
M(s;a1;:::;aN)conditions on the current
statesas well as the current actions a1;:::;aNand repre-
sents the current reward plus the discounted worst case future
return starting from the next state ,s0. This deﬁnition brings
the beneﬁts that we can naturally apply off-policy temporal
difference learning later to derive the update rule for Q
M.
Note that for each agent i, none of the adversarial actions
depend on its parameter i, so we can directly apply the deter-
ministic policy gradient theorem to compute riJM(i)and
use off-policy temporal difference to update the Q function.
Thanks to the centralized Q function in MADDPG (Eq. 4),
which takes in the actions from all the agents, our derivation
naturally applies and is perfectly aligned with the MADDPG
formulation (Eq. 4) by injecting a minimization over other
agents’ actions as follows:
riJM(i) =
ExD2
4rii(oi)raiQ
M;i(x;a?
1;:::;ai;:::a?
N)
ai=i(oi)
a?
j6=i= arg min aj6=iQ
M;i(x;a1;:::;aN)3
5;
(9)
whereDdenotes the replay buffer and xdenotes the state
information.
Correspondingly, we obtain the new Q function update rule
by adding another minimization to Eq. 5 when computing
the target Q value:
L(i) =Ex;a;r;x0D[(Q
M;i(x;a1;:::;aN)y)2];(10)
y=ri+
Q0
M;i(x0;a0?
1;:::;a0
i;:::;a0?
N)
a0
i=0
i(oi);
a0?
j6=i= arg min
a0
j6=iQ0
M;i(x0;a0
1;:::;a0
N);
where0
idenotes the target policy of agent iwith delayed
parameters0
i, andQ0
M;idenotes the target Q network for
agenti. Combining Eq. 9 and Eq. 10 yields our proposed
minimax learning framework.Multi-Agent Adversarial Learning
The critical challenge in our proposed minimax learning
framework is how to handle the embedded minimization in
Eq. 9 and Eq. 10. Due to the continuous action space as well
as the non-linearity of Q function, directly optimizing the
minimization problem is computationally intractable. A naive
approximate solution can be performing an inner-loop gradi-
ent descent whenever performing an update step of Eq. 9 or
Eq. 10, but this is too computationally expensive for practical
use. Here we introduce an efﬁcient and end-to-end solution,
multi-agent adversarial learning (MAAL). The main ideas
of MAAL can be summarized in two steps: (1) approximate
the non-linear Q function by a locally linear function; (2)
replace the inner-loop minimization with a 1-step gradient
descent. Note the core idea of MAAL, locally linearizing the
Q function, is adapted from the recent adversarial training
technique originally developed for supervised learning. We
will discuss the connection between adversarial training and
MAAL in the end of this section.
For conciseness, we ﬁrst consider Eq. 10 and rewrite it
into the following form with auxiliary variables :
y=ri+
Q0
M;i(x0;a0?
1;:::;a0
i;:::;a0?
N) (11)
a0
k=0
k(ok);81kN
a0?
j=a0
j+j;8j6=i
j6=i= arg min
j6=iQ0
M;i(x0;a0
1+1;:::;a0
i;:::;a0
N+N):
Eq. 11 can be interpreted as we are now seeking for
a set of perturbations such that the perturbed actions
a0?decrease Q value the most. By linearizing the Q func-
tion atQ
M;i(x;a0
1;:::;a0
N), the desired perturbation j
can be locally approximated by the gradient direction at
Q
M;i(x;a0
1;:::;a0
N)w.r.t.a0
j. Then we use this local to de-
rive an approximation ^jto the worst case perturbation by
taking a small gradient step:
8j6=i;^j=rajQ0
M;i(x0;a0
1;:::;aj;:::;a0
N);(12)
whereis a tunable coefﬁcient representing the perturbation
rate. It can be also interpreted as the step size of the gradient
descent step: when is too small, the local approximation er-
ror will be small but due to the small perturbation, the learned
policy can be far from the optimal solution of the minimax
objective we proposed; when is too large, the approxima-
tion error may incur too much trouble for the overall learning
process and the agents may fail to learn good policies.
We can apply this technique to Eq. 9 as well and eventually
derive the following formulation:
riJ(i) =
Ex;aD2
664rii(oi)raiQ
M;i(x;a?
1;:::;ai;:::a?
N)
ai=i(oi)
a?
j=aj+ ^j;8j6=i
^j=jrajQ
M;i(x;a1;:::;aN)3
775;
(13)
Algorithm 1: Minimax Multi-Agent Deep Deterministic Policy Gradient (M3DDPG) for Nagents
forepisode = 1toMdo
Initialize a random process Nfor action exploration, and receive initial state information x
fort= 1to max-episode-length do
for each agent i, select action ai=i(oi) +Ntw.r.t. the current policy and exploration
Execute actions a= (a1;:::;aN)and observe reward rand new state information x0
Store (x;a;r; x0)in replay bufferD, and set x x0
foragenti= 1toNdo
Sample a random minibatch of Ssamples (xk;ak;rk;x0k)fromD
Setyk=rk
i+
Q0
M;i(x0k;a0
1;:::;a0
N)ja0
i=0
i(ok
i);a0
j6=i=0
j(ok
j)+^0
jwith^0
jdeﬁned in Eq. 14
Update critic by minimizing the loss L(i) =1
SP
k
ykQ
M;i(xk;ak
1;:::;ak
N)2
Update actor using the sampled policy gradient with ^jdeﬁned in Eq. 13:
riJ1
SX
krii(ok
i)raiQ
M;i(xk;a?
1;:::;ai;:::;a?
N)
ai=i(ok
i);a?
j6=i=ak
j+^j
end for
Update target network parameters for each agent i:0
i i+ (1)0
i
end for
end for
and
L(i) =Ex;a;r;x0[(Q
M;i(x;a1;:::;aN)y)2];(14)
y=ri+
Q0
M;i(x0;a0?
1;:::;a0
i;:::;a0?
N)
a0
k=0
k(ok);81kN
a0?
j=a0
j+ ^0
j;8j6=i
^0
j=jra0
jQ0
M;i(x;a0
1;:::;a0
N);
where1;:::;Nare additional parameters. MAAL only
requires one additional gradient computation, and can be
executed in a fully end-to-end fashion. Finally, combining
Eq. 13 and Eq. 14 completes MAAL. The overall algorithm,
M3DDPG, is summarized as Algo. 1.
Discussion
Connection to Adversarial Training Adversarial train-
ingis a robust training approach for deep neural networks on
supervised learning (Goodfellow et al .2014). The core idea
is to force the classiﬁer to predict correctly even when given
adversarial examples , which are obtained by adding a small
adversarial perturbation to the original input data such that the
classiﬁcation loss can be decreased the most. Formally, sup-
pose the classiﬁcation loss function is L() =Ex;y[f(x;y)]
with input data xand labely. Adversarial training aims to
optimize the following adversarial loss instead
Ladv() = Ex;y[f(x+?;y)] (15)
?= arg max
kkf(x+;y):
The core technique to efﬁciently optimize Ladv()is to locally
linearize the loss function at f(x;y)and approximate ?by
the scaled gradient.
Thanks to the centralized Q function, which takes the ac-
tions from all the agents as part of the input, we are able toeasily inject the minimax optimization (Eq. 11) and represent
it in a similar way to adversarial training (Eq. 15) so that
we can adopt the similar technique to effectively solve our
minimax optimization in a fully end-to-end fashion.
Connection to Single Agent Robust RL M3DDPG with
MAAL can be also viewed as the special case of robust
reinforcement learning (RRL) (Morimoto and Doya 2005)
in the single agent setting, which aims to bridge the gap
between training in simulation and testing in the real world
by adding adversarial perturbations to the transition dynamics
during training. Here, we consider the multi-agent setting and
add worst case perturbations to actions of opponent agents
during training. Note that in the perspective of a single agent,
perturbations on opponents’ actions can be also considered
as a special adversarial noise on the dynamics.
Choice ofIn the extreme case of = 0, M3DDPG de-
generates to the original MADDPG algorithm while as
increases, the policy learning tends to be more robust but
the optimization becomes harder. In practice, using a ﬁxed
throughout training can lead to very unstable learning
behavior due to the changing scale of the gradients. The
original adversarial training paper (Goodfellow, Shlens, and
Szegedy 2014) suggests to compute with a ﬁxed norm,
namelyg=rxf(x;y),^=g
kgk, wherexdenotes the in-
put data to the classiﬁer and ydenotes the label. Accordingly,
in our M3DDPG algorithm, we can adaptively compute the
perturbation ^jby
g=rajQ
M;i(x;a1;:::;aN);^j=jg
kgk:(16)
Eq. 16 generally works ﬁne in practice but in some hard
multi-agent learning environments, unstable training behav-
ior can be still observed. We suspect that it is because the
changing norm of actions in these situations. Different from
the supervised learning setting where the norm of the input
dataxis typically stable, in reinforcement learning the norm
of actions can drastically change even in a single episode.
Therefore, it is possible to see cases that even a perturbation
with a small ﬁxed norm overwhelms the action aj, which
may potentially lead to computational stability issue. There-
fore, we also introduce the following alternative for adaptive
perturbation computation:
g=rajQ
M;i(x;a1;:::;aN);^j=jkajkg
kgk:(17)
Lastly, note that in a mixed cooperative and competitive
environment, ideally we only need to add adversarial pertur-
bations to competitors. But empirically we observe that also
adding (smaller) perturbations to collaborators can further
improve the quality of learned policies.
5. Experiments
We adopt the same particle-world environments as the MAD-
DPG paper (Lowe et al .2017) as well as the training con-
ﬁgurations. is selected from a grid search over 0:1,0:01
and0:001. For testing, we generate a ﬁxed set of 2500 envi-
ronment conﬁgurations (i.e., landmarks and birthplaces) and
evaluate on this ﬁxed set for a fair comparison.
speaker
listener“green”agent 1
agent 3landmarklandmark
landmark  p
vc
agent 2predator 1
preypredator 2
predator 3agent 1agent 2
agent 3agent 1agent 2
adversary
?
speaker
listener“green”agent 1
agent 3landmarklandmark
landmark  p
vc
agent 2predator 1
preypredator 2
predator 3agent 1agent 2
agent 3agent 1agent 2
adversary
?
Figure 1: Illustrations of some environments we consider, in-
cluding Physical Deception (left) and Predator-Prey (right).
Environments
The particle world environment consists of Ncooperative
agents,Madversarial agents and Llandmarks in a two-
dimensional world with continuous space. We focus on the
four mixed cooperative and competitive scenarios to best
examine the effectiveness of our minimax formulation.
Covert communication This is an adversarial communi-
cation environment, where a speaker agent (‘Alice’) must
communicate a message to a listener agent (‘Bob’) ( N= 2),
who must reconstruct the message at the other end. However,
an adversarial agent (‘Eve’) ( M= 1) is also observing the
channel, and wants to reconstruct the message — Alice and
Bob are penalized based on Eve’s reconstruction, and thus
Alice must encode her message using a randomly generated
key, known only to Alice and Bob.
00.30.60.9
MA vs MA MA vs Minimax Minimax vs MA Minimax vs MinimaxNormalized Agent ScoreComparison between MADDPG (MA) and M3DDPG (Minimax )
Covert communication Keep-away Physical deception Predator-preyFigure 2: Comparison between M3DDPG (Minimax) and
classical MADDPG (MA) on the four mixed competitive
environments. Each bar cluster shows the 0-1 normalized
score for a set of competing policies in different roles (agent
vs adversary), where a higher score is better for the agent. In
all cases, M3DDPG outperforms MADDPG when directly
pitted against it.
Keep-away This scenario consists of L= 1 target land-
mark,N= 2 cooperative agents and M= 1 adversarial
agent. Cooperating agents need to reach the landmark and
keep the adversarial agent away from the landmark by push-
ing it while the adversarial agent must stay at the landmark
to occupy it.
Physical deception Here,N= 2agents cooperate to reach
a single target landmark from a total of L= 2 landmarks.
They are rewarded based on the minimum distance of any
agent to the target (so only one agent needs to reach the target
landmark). However, a lone adversary ( M= 1) also desires
to reach the target landmark; the catch is that the adversary
does not know which of the landmarks is the correct one.
Thus the cooperating agents, who are penalized based on the
adversary distance to the target, learn to spread out and cover
all landmarks so as to deceive the adversary.
Predator-prey In this variant of the classic predator-prey
game,N= 3slower cooperating agents must chase the faster
adversary (M= 1) around a randomly generated environ-
ment withL= 2large landmarks impeding the way. Each
time the cooperative agents collide with an adversary, the
agents are rewarded while the adversary is penalized.
Comparison to MADDPG
To evaluate the quality of learned policies trained by dif-
ferent algorithms in competitive scenarios, we measure the
performance of agents trained by our M3DDPG and agents
by classical MADDPG in the roles of both normal agent and
adversary in each environment.
The results are demonstrated in Figure 2, where we mea-
sure the rewards of the normal agents in different scenarios
and normalize them to 0-1. We notice that in all the envi-
ronments, the highest score is achieved when the M3DDPG
agents play as the normal agents against the MADDPG ad-
versary (Minimax vs MA); while the lowest score is when the
MADDPG agents act as normal agents against the M3DDPG
adversary (MA vs Minimax). This indicates that policies
Figure 3: Performances of M3DDPG (Minimax, red) and MADDPG (MA, blue) under the worst situation , i.e., against the
disruptive adversaries , on convert communication, keep-away, physical deception and predator-pray from left to right. The y-axis
denotes the reward of normal agents (ﬁxed) and x-axis denotes the training episodes performed of the disruptive adversaries.
Higher reward implies a more robust policy. Agents trained by M3DDPG (Minimax) perform better on all the scenarios.
trained by M3DDPG have much higher quality than original
MADDPG.
Evaluation with Disruptive Adversaries
Instead of M3DDPG and MADDPG directly competing
against each other, now we consider their performances in the
worst possible situations through their individual competi-
tions against disruptive adversaries . We construct disruptive
adversaries by (1) ﬁxing the trained normal agents (M3DDPG
or MADDPG); (2) setting the reward of the disruptive adver-
sary as the negative value of normal agents’ reward (so that
the problem becomes zero-sum); and then (3) using DDPG
to train the disruptive adversary from scratch under the zero-
sum reward setting.
In the particle world environment, the competitive scenar-
ios are generally not zero-sum, which implies that according
to the default reward function, the adversaries may have dif-
ferent purposes rather than directly disrupting the normal
agents. So, in order to evaluate the effectiveness our minimax
optimization in the worst situation, we convert every prob-
lems into a zero-sum form and compare the performances of
our M3DDPG agents as well as the MADDPG agents against
this artiﬁcially constructed adversaries. Moreover, since each
of our four environments has only 1 adversary, after ﬁxing
the normal agents, the learning problem degenerates to the
single agent setting and classical DDPG is sufﬁcient to stably
train a disruptive adversary.
The results are shown in Figure 3, where we plot the re-
ward of the ﬁxed normal agents of different algorithms as
the training of the disruptive adversaries progresses until con-
vergence. Note that due to the different environment designs,
the difﬁculty for the disruptive agents to break the strategy of
normal agents varies: for example, in convert communication ,
since the private key is not accessible to the adversary agent,
breaking the encrypted communication will be very hard;
while in physical deception , since we do not allow communi-
cation and ﬁx the normal agents, a smart enough adversary
may easily infer the target landmark by observing the initial
behavior of the two cooperative agents. Nevertheless, despite
these intrinsic properties, the M3DDPG agents (Minimax)
achieves higher reward in all the scenarios, which implies
better robustness even in the worst situation.6. Conclusion
In this paper, we propose a novel algorithm, minimax mulit-
agent deep deterministic policy gradient (M3DDPG), for
robust multi-agent reinforcement learning, which leverages
the minimax concept and introduces a minimax learning ob-
jective. To efﬁciently optimize the minimax objective, we pro-
pose MAAL, which approximates the inner-loop minimiza-
tion by a single gradient descent step. Empirically, M3DDPG
outperforms the benchmark methods on four mixed coopera-
tive and competitive scenarios.
Nevertheless, due to the single step gradient approximation
in MAAL, which is efﬁcient in computation, an M3DDPG
agent can only explore locally worst situation during the
evolving process at training, which can still lead to unsatisfy-
ing behavior when testing opponents have drastically differ-
ent strategies. It will be an interesting direction to re-examine
the robustness-efﬁciency trade-off in MAAL and further im-
prove policy learning by placing more computations on the
minimax optimization. We leave this as our future work.
References
[Busoniu, Babuska, and De Schutter 2008] Busoniu, L.;
Babuska, R.; and De Schutter, B. 2008. A comprehensive
survey of multiagent reinforcement learning. IEEE Transac-
tions on Systems Man and Cybernetics Part C Applications
and Reviews 38(2):156.
[Finn, Abbeel, and Levine 2017] Finn, C.; Abbeel, P.; and
Levine, S. 2017. Model-agnostic meta-learning for fast adap-
tation of deep networks. arXiv preprint arXiv:1703.03400 .
[Foerster et al. 2016a] Foerster, J. N.; Assael, Y . M.; de Fre-
itas, N.; and Whiteson, S. 2016a. Learning to communi-
cate with deep multi-agent reinforcement learning. CoRR
abs/1605.06676.
[Foerster et al. 2016b] Foerster, J. N.; Assael1, Y . M.; de Fre-
itas, N.; and Whiteson, S. 2016b. Learning to Commu-
nicate to Solve Riddles with Deep Distributed Recurrent
Q-Networks.
[Foerster et al. 2017] Foerster, J.; Farquhar, G.; Afouras, T.;
Nardelli, N.; and Whiteson, S. 2017. Counterfactual multi-
agent policy gradients. arXiv preprint arXiv:1705.08926 .
[Foerster et al. 2018] Foerster, J.; Chen, R. Y .; Al-Shedivat,
M.; Whiteson, S.; Abbeel, P.; and Mordatch, I. 2018. Learn-
ing with opponent-learning awareness. In Proceedings of the
17th International Conference on Autonomous Agents and
MultiAgent Systems , 122–130. International Foundation for
Autonomous Agents and Multiagent Systems.
[Gao, Mueller, and Hayward 2018] Gao, C.; Mueller, M.;
and Hayward, R. 2018. Adversarial policy gradient for
alternating markov games.
[Goodfellow et al. 2014] Goodfellow, I.; Pouget-Abadie, J.;
Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville,
A.; and Bengio, Y . 2014. Generative adversarial nets. In
Advances in neural information processing systems , 2672–
2680.
[Goodfellow, Shlens, and Szegedy 2014] Goodfellow, I. J.;
Shlens, J.; and Szegedy, C. 2014. Explaining and harnessing
adversarial examples. arXiv preprint arXiv:1412.6572 .
[Grau-Moya, Leibfried, and Bou-Ammar 2018] Grau-Moya,
J.; Leibfried, F.; and Bou-Ammar, H. 2018. Balancing
two-player stochastic games with soft q-learning. arXiv
preprint arXiv:1802.03216 .
[Haarnoja et al. 2017] Haarnoja, T.; Tang, H.; Abbeel, P.; and
Levine, S. 2017. Reinforcement learning with deep energy-
based policies. arXiv preprint arXiv:1702.08165 .
[He et al. 2016] He, H.; Boyd-Graber, J.; Kwok, K.; and
Daumé III, H. 2016. Opponent modeling in deep rein-
forcement learning. In International Conference on Machine
Learning , 1804–1813.
[Hu, Wellman, and others 1998] Hu, J.; Wellman, M. P.; et al.
1998. Multiagent reinforcement learning: theoretical frame-
work and an algorithm. In ICML , volume 98, 242–250. Cite-
seer.
[Lanctot et al. 2017] Lanctot, M.; Zambaldi, V .; Gruslys, A.;
Lazaridou, A.; Perolat, J.; Silver, D.; Graepel, T.; et al. 2017.
A uniﬁed game-theoretic approach to multiagent reinforce-
ment learning. In Advances in Neural Information Processing
Systems , 4190–4203.
[Lazaridou, Peysakhovich, and Baroni 2016] Lazaridou, A.;
Peysakhovich, A.; and Baroni, M. 2016. Multi-agent co-
operation and the emergence of (natural) language. arXiv
preprint arXiv:1612.07182 .
[Leibo et al. 2017] Leibo, J. Z.; Zambaldi, V . F.; Lanctot,
M.; Marecki, J.; and Graepel, T. 2017. Multi-agent rein-
forcement learning in sequential social dilemmas. CoRR
abs/1702.03037.
[Levine et al. 2016] Levine, S.; Finn, C.; Darrell, T.; and
Abbeel, P. 2016. End-to-end training of deep visuomo-
tor policies. The Journal of Machine Learning Research
17(1):1334–1373.
[Lillicrap et al. 2015] Lillicrap, T. P.; Hunt, J. J.; Pritzel, A.;
Heess, N.; Erez, T.; Tassa, Y .; Silver, D.; and Wierstra, D.
2015. Continuous control with deep reinforcement learning.
arXiv preprint arXiv:1509.02971 .
[Littman 1994] Littman, M. L. 1994. Markov games as a
framework for multi-agent reinforcement learning. In Pro-
ceedings of the eleventh international conference on machine
learning , volume 157, 157–163.
[Lowe et al. 2017] Lowe, R.; Wu, Y .; Tamar, A.; Harb, J.;
Abbeel, O. P.; and Mordatch, I. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. In
Advances in Neural Information Processing Systems , 6379–
6390.
[Mandlekar et al. 2017] Mandlekar, A.; Zhu, Y .; Garg, A.;
Fei-Fei, L.; and Savarese, S. 2017. Adversarially robust
policy learning: Active construction of physically-plausible
perturbations. In Intelligent Robots and Systems (IROS), 2017
IEEE/RSJ International Conference on , 3932–3939. IEEE.
[Mnih et al. 2015] Mnih, V .; Kavukcuoglu, K.; Silver, D.;
Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Ried-
miller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015.
Human-level control through deep reinforcement learning.
Nature 518(7540):529–533.
[Mordatch and Abbeel 2017] Mordatch, I., and Abbeel, P.
2017. Emergence of grounded compositional language in
multi-agent populations. arXiv preprint arXiv:1703.04908 .
[Morimoto and Doya 2005] Morimoto, J., and Doya, K. 2005.
Robust reinforcement learning. Neural computation
17(2):335–359.
[OpenAI 2018] OpenAI. 2018. OpenAI Five.
https://blog.openai.com/openai-ﬁve/. Accessed: 2018-
09-03.
[Osborne and others 2004] Osborne, M. J., et al. 2004. An
introduction to game theory , volume 3. Oxford university
press New York.
[Peng et al. 2017a] Peng, P.; Yuan, Q.; Wen, Y .; Yang, Y .;
Tang, Z.; Long, H.; and Wang, J. 2017a. Multiagent
bidirectionally-coordinated nets for learning to play starcraft
combat games. CoRR abs/1703.10069.
[Peng et al. 2017b] Peng, X. B.; Andrychowicz, M.; Zaremba,
W.; and Abbeel, P. 2017b. Sim-to-real transfer of robotic
control with dynamics randomization. arXiv preprint
arXiv:1710.06537 .
[Pérolat et al. 2016] Pérolat, J.; Strub, F.; Piot, B.; and
Pietquin, O. 2016. Learning nash equilibrium for general-
sum markov games from batch data. arXiv preprint
arXiv:1606.08718 .
[Pinto et al. 2017] Pinto, L.; Davidson, J.; Sukthankar, R.; and
Gupta, A. 2017. Robust adversarial reinforcement learning.
arXiv preprint arXiv:1703.02702 .
[Silver et al. 2014] Silver, D.; Lever, G.; Heess, N.; Degris, T.;
Wierstra, D.; and Riedmiller, M. 2014. Deterministic policy
gradient algorithms. In Proceedings of the 31st International
Conference on Machine Learning , 387–395.
[Silver et al. 2016] Silver, D.; Huang, A.; Maddison, C. J.;
Guez, A.; Sifre, L.; Van Den Driessche, G.; Schrittwieser, J.;
Antonoglou, I.; Panneershelvam, V .; Lanctot, M.; et al. 2016.
Mastering the game of go with deep neural networks and tree
search. Nature 529(7587):484–489.
[Sukhbaatar, Fergus, and others 2016] Sukhbaatar, S.; Fer-
gus, R.; et al. 2016. Learning multiagent communication
with backpropagation. In Advances in Neural Information
Processing Systems , 2244–2252.
[Sutton and Barto 1998] Sutton, R. S., and Barto, A. G. 1998.
Reinforcement learning: An introduction , volume 1. MIT
press Cambridge.
[Sutton et al. 2000] Sutton, R. S.; McAllester, D. A.; Singh,
S. P.; and Mansour, Y . 2000. Policy gradient methods for
reinforcement learning with function approximation. In Ad-
vances in neural information processing systems , 1057–1063.
[Tesauro 2004] Tesauro, G. 2004. Extending q-learning to
general adaptive multi-agent systems. In Advances in neural
information processing systems , 871–878.
[Tobin et al. 2017] Tobin, J.; Fong, R.; Ray, A.; Schneider, J.;
Zaremba, W.; and Abbeel, P. 2017. Domain randomization
for transferring deep neural networks from simulation to the
real world. In Intelligent Robots and Systems (IROS), 2017
IEEE/RSJ International Conference on , 23–30. IEEE.
[Uther and Veloso 1997] Uther, W. T., and Veloso, M. M.
1997. Generalizing adversarial reinforcement learning. In
Proceedings of the AAAI Fall Symposium on Model Directed
Autonomous Systems , 206. Citeseer.
[Wei et al. 2018] Wei, E.; Wicke, D.; Freelan, D.; and Luke,
S. 2018. Multiagent soft q-learning. arXiv preprint
arXiv:1804.09817 .
[Williams 1992] Williams, R. J. 1992. Simple statistical
gradient-following algorithms for connectionist reinforce-
ment learning. Machine learning 8(3-4):229–256.
[Wu et al. 2017] Wu, C.; Kreidieh, A.; Parvate, K.; Vinitsky,
E.; and Bayen, A. M. 2017. Flow: Architecture and bench-
marking for reinforcement learning in trafﬁc control. arXiv
preprint arXiv:1710.05465 .
[Yang et al. 2018] Yang, Y .; Luo, R.; Li, M.; Zhou, M.; Zhang,
W.; and Wang, J. 2018. Mean ﬁeld multi-agent reinforcement
learning. arXiv preprint arXiv:1802.05438 .

Shihui Li
Yi Wu
Xinyue Cui
Honghua Dong
Fei Fang
Stuart Russell