MDPs with Unawareness in Robotics.

MDPs with Unawareness in Robotics
Nan Rong Joseph Y. Halpern Ashutosh Saxena
Computer Science Department, Cornell University
Ithaca, NY 14853
frongnanjhalpernjasaxenag@cs.cornell.edu
Abstract
We formalize decision-making problems in
robotics and automated control using continuous
MDPs and actions that take place over contin-
uous time intervals. We then approximate the
continuous MDP using ﬁner and ﬁner discretiza-
tions. Doing this results in a family of sys-
tems, each of which has an extremely large ac-
tion space, although only a few actions are “in-
teresting”. We can view the decision maker as
being unaware of which actions are “interesting”.
We an model this using MDPUs , MDPs with
unawareness, where the action space is much
smaller. As we show, MDPUs can be used as
a general framework for learning tasks in robotic
problems. We prove results on the difﬁculty of
learning a near-optimal policy in an an MDPU
for a continuous task. We apply these ideas to
the problem of having a humanoid robot learn on
its own how to walk.
1 INTRODUCTION
Markov decision processes (MDPs) are widely used for
modeling decision making problems in robotics and au-
tomated control. Traditional MDPs assume that the deci-
sion maker (DM) knows all states and actions. However,
in many robotics applications, the space of states and ac-
tions is continuous. To ﬁnd appropriate policies, we typ-
ically discretize both states and actions. However, we do
not know in advance what level of discretization is good
enough for getting a good policy. Moreover, in the dis-
cretized space, the set of actions is huge. However, rela-
tively few of the actions are “interesting”. For example,
when ﬂying a robotic helicopter, only a small set of actions
lead to useful ﬂying techniques; an autonomous helicopter
must learn these techniques. Similarly, a humanoid robot
needs to learn various maneuvers (e.g., walking or running)
that enable it to move around, but the space of potential ac-
tions that it must search to ﬁnd a successful gait is huge,while most actions result in the robot losing control and
falling down.
Halpern, Rong, and Saxena [2010] (HRS from now
on) deﬁned MDPs with unawareness (MDPUs) , where a
decision-maker (DM) can be unaware of the actions in an
MDP. In the robotics applications in which we are inter-
ested, we can think of the DM (e.g., a humanoid robot) as
being unaware of which actions are the useful actions, and
thus can model what is going on using an MDPU.
In this paper, we apply MDPUs to continuous problems.
We model such problems using continuous MDPs , where
actions are performed over a continuous duration of time.
Although many problems ﬁt naturally in our continuous
MDP framework, and there has been a great deal of work
on continuous-time MDPs, our approach seems new, and of
independent interest. (See the discussion in Section 5.) It
is hard to ﬁnd near-optimal policies in continuous MDPs.
A standard approach is to use discretization. We use dis-
cretization as well, but our discrete models are MDPUs,
rather than MDPs, which allows us both to use relatively
few actions (the “interesting actions”), while taking into ac-
count the possibility of there being interesting actions that
the DM has not yet discovered. We would like to ﬁnd a dis-
cretization level for which the optimal policy in the MDP
underlying the approximating MDPU provides a good ap-
proximation to the optimal policy in the continuous MDP
that accurately describes the problem, and then ﬁnd a near-
optimal policy in that discretized MDPU.
HRS gave a complete characterization of when it is possi-
ble to learn to play near-optimally in an MDPU, extending
earlier work [Brafman and Tennenholtz 2002; Kearns and
Singh 2002] showing that it is always possible to learn to
play near-optimally in an MDP. We extend and generalize
these results so as to apply them to the continuous prob-
lems of interest to us. We characterize when brute-force
exploration can be used to ﬁnd a near-optimal policy in
our setting, and show that a variant of the URMAX algo-
rithm presented by HRS can ﬁnd a near-optimal policy. We
also characterize the complexity of learning to play near-
optimally in continuous problems, when more “guided” ex-
ploration is used. Finally, we discuss how MDPUs can be
used to solve a real robotic problem: to enable a humanoid
robot to learn walking on its own. In our experiment, the
robot learned various gaits at multiple discretization levels,
including both forward and backward gaits; both efﬁcient
and inefﬁcient gaits; and both gaits that resemble human
walking, and those that do not.
2 MDPU: A REVIEW
Describing a situation by a standard MDP misses out on
some important features. In general, an agent may not be
aware of all the actions that can be performed. For exam-
ple, an agent playing a video game may not be aware of all
actions that can be performed in a given state. Our model
is compatible with a number of interpretations of unaware-
ness. In the robotics setting, we take a particular concrete
interpretation. Here, the number of actions is typically ex-
tremely large, but only a few of these actions are actually
useful. For example, although an autonomous helicopter
can have a huge number of actions, only a few are useful
for ﬂying the helicopter, whereas the rest simply result in
a crash. We can abstract what is going on by deﬁning a
set of “useful actions”. The DM may initial be unaware of
many or even most of the useful actions. A DM may be-
come aware of a useful action (and thus, can safely perform
it) by performing a special action called the explore action,
denoteda0. Playing the explore action results in the DM
learning about new actions with some probability.
We thus take an MDPU to be a tuple M =
(S;A;A 0;g;a 0;g0;P;D;R ),1where the tuple
(S;A;g;P;R )is a standard MDP—that is, Sis a set
of states,Ais a set of actions, g(s)is the set of actions
available at state s, for each tuple (s;s0;a)2SSA,
P(s;s0;a)gives the probability of making a transition
fromstos0if actionais performed, and R(s;s0;a)gives
the reward earned by the DM if this transition is taken;
A0Ais the set of actions that the DM initially knows
to be useful; a0is the special explore action;g(s)A
is the set of actions that can be performed at state s;
g0(s)A0\g(s)is the set of actions that the DM is
aware of at state s; ﬁnally,Dis the discovery probability
function .D(j;t;s )is the probability of discovering a
useful action given that there are juseful actions to be
discovered at state s, anda0has already been played t1
times without discovering a useful action. Intuitively,
Ddescribes how quickly the DM can discover a useful
action. We assume that D(j;t;s )is non-decreasing as
a function of j: the more useful actions there are to be
found, the easier we can ﬁnd one. How D(j;t;s )varies
withtdepends on the problem. In the sequel, we assume
for ease of exposition that D(j;t;s )is independent of s, so
1The MDPU model in HRS also includes R+
iandR
i, which
are the reward (resp., penalty) functions for playing a0and dis-
covering (resp., not discovering) a useful action. We omit them
here; they play no role in the theorems that we are citing, and
would only clutter our later presentation.we writeD(j;t)rather thanD(j;t;s ).M0= (S;A;P;R )
is called the MDP underlying M.
Kearns and Singh [2002] and Brafman and Tennenholtz
[2002] have studied the problem of learning how to play
near-optimally in an MDP. Roughly speaking, to play near-
optimally means that, for all  > 0and > 0, we can
ﬁnd a policy that obtains expected reward -close to that
of the optimal policy with probability at least 1. HRS
completely characterize the difﬁculty of learning to play
near-optimally in an MDPU. We brieﬂy review the relevant
results here. Despite initially being unaware of some use-
ful actions, we want the DM to learn a policy that is near-
optimal in the underlying MDP. HRS showed that whether
a DM can learn to play optimally and how long it takes de-
pend on the value of D(1;t)—the probability of discover-
ing a new action given that there is a new action to discover
and the DM has tried t1times in the past to discover a
new action.
The ﬁrst result characterizes when it is impossible to learn
to play near-optimally. To make it precise, we recall the
notion of mixing time. Formally, the -return mixing time
of an MDPMis the leastTsuch that an optimal policy for
Mguarantees an expected payoff within of the optimal
reward after running for at least time T. (See [Kearns and
Singh 2002] for more details and motivation.) The follow-
ing theorem shows that if the discovery probability is suf-
ﬁciently low, where “sufﬁciently low” means D(1;t)<1
for alltandP1
t=1D(1;t)<1, then the DM cannot learn
to play near-optimally. We deﬁne 	(T) =PT
t=1D(1;t).
Theorem 2.1 IfD(1;t)<1for alltand	(1)<1,
then there exists a constant csuch that no algorithm can
obtain a reward that is guaranteed to be within cof optimal
for an MDPU M= (S;A;A 0;G;aog0;P;D;R ), even if
S,jAj, and a bound on the optimal reward are known.
Theorem 2.1 says that when 	(1)<1, it is impossible
for the DM to learn an optimal policy. On the other hand,
if	(1) =1, then it is possible to learn to play near-
optimally. HRS present an algorithm called URMAX, a
variant of the RMAX algorithm [Brafman and Tennenholtz
2002], that learns near-optimal play.
Theorem 2.2 If	(1) =1, then the URMAX algorithm
computes a near-optimal policy.
In fact, we can say even more. If 	(1) =1, then the ef-
ﬁciency of the best algorithm for determining near-optimal
play depends on how quickly 	(1)diverges. HRS char-
acterize the running time of URMAX in terms of the func-
tion	, and give lower bounds on the time required to learn
near-optimally in terms of 	. These results show that UR-
MAX learns to play near-optimally almost as quickly as
possible. Speciﬁcally, it learns a policy with an expected
reward-close to the optimal reward with probability 1
in time polynomial in jSj,jAj,1=,1=, a boundRmaxon
the optimal reward, the -return mixing time, and the small-
estTsuch that 	(T)ln(4N=), whenever it is possible
to do so. The polynomial-time results are summarized in
the next result.
Theorem 2.3 It is possible to learn to play near-optimally
in polynomial time iff there exist constants m1andm2such
that	(T)m1ln(T) +m2for allT > 0. Moreover, if
it is possible to learn to play near-optimally in polynomial
time, URMAX does so.
3 ANALYZING ROBOTIC PROBLEMS
AS MDPUS
As we said in the introduction, we apply the MDPU frame-
work to robotic problems such as having a humanoid robot
learn to walk. For such problems, we typically have a
continuous space of states and actions, where actions take
place in continuous time, and actions have a nontrivial du-
ration.
Suppose that the original continuous problem can be char-
acterized by a continuous MDP M1(deﬁned formally be-
low). We would like to ﬁnd a “good” discretization Mof
M1. “Good” in this setting means that an optimal pol-
icy forMis-optimal for M1, for some appropriate .2
Clearly the level of discretization matters. Too coarse a dis-
cretization results in an MDP whose optimal policy is not -
optimal forM1; on the other hand, too ﬁne a discretization
results in the problem size becoming unmanageably large.
For example, in order to turn a car on a smooth curve (with-
out drifting), the optimal policy is to slowly turn the steer-
ing wheel to the left and back, in which the action varies
smoothly over time. This can be simulated using a rela-
tively coarse discretization of time. However, in order to
make a sharp turn using advanced driving techniques like
drifting, the steering wheel needs to be turned at precise
points in time, or else the car will go into an uncontrollable
spin. In this case, a ﬁne discretization in time is needed.
Unfortunately, it is often not clear what discretization level
to use in a speciﬁc problem. Part of the DM’s problem is to
ﬁnd the “right” level of discretization. Thus, we describe
the problem in terms of a continuous MDP M1and a se-
quence ((M1;M0
1);(M2;M0
2);:::), whereMiis an MDPU
with underlying MDP M0
i, fori= 1;2;:::. Intuitively,
(M0
1;M0
2;M0
3;:::)represents a sequence of ﬁner and ﬁner
approximations to M1.
Continuous Time MDP with Continuous Actions over
Time: To make this precise, we start by deﬁn-
ing our model of continuous MDPs. Let M1=
(S1;A1;g1;P1;R1).S1is a continuous state space,
which we identify with a compact subset of IRnfor some
integern > 0; that is, each state can be represented by a
vector (s1;;sn)of real numbers. For example, for a hu-
2A policyis-optimal for an MDP Mif the expected aver-
age reward for a policy for Mis no more than greater than the
expected average reward of .manoid robot, the state space can be described by a vector
which includes the robot’s (x;y;z )position, and the cur-
rent positions of its movable joints.
Actions: DescribingA1requires a little care. We assume
that there is an underlying set of basic actions AB, which
can be identiﬁed with a compact subset of IRmfor some
m > 0; that is, each basic action can be represented by a
vector (a1;;am)of real numbers. For example, for a
humanoid robot, the basic actions can be characterized by
a tuple that contains the targeted positions for its movable
joints. However, we do not take A1to consist of basic
actions. Rather, an action is a path of basic actions over
time. Formally, an action in A1is a piecewise continuous
function from a domain of the form (0;t]for somet >0
to basic actions. Thus, there exist time points t0< t1<
::: < tkwitht0= 0 andtk=tsuch thatais continuous
in the interval (tj;tj+1]for allj <k . The number tis the
length of the action a, denotedjaj. We use left-open right-
closed intervals here; we think of the action in the interval
(tj;tj+1]as describing what the DM does right after time
tjuntil timetj+1. By analogy with the ﬁnite case, g1(s)
is the set of actions in A1available ats.
Reward and Transition Functions: We now deﬁne R1
andP1, the reward and transition functions. In a discrete
MDP, the transition function Pand reward function Rtake
as arguments a pair of states and an action. Thus, for ex-
ample,P(s1;s2;a)is the probability of transitioning from
s1tos2using action a, andR(s1;s2;a)is the reward the
agent gets if a transition from s1tos2is taken using ac-
tiona. In our setting, what matters is the path taken by
a transition according to a. Thus, we take the arguments
toP1andR1to be tuples of the form (s1;sc;a), where
s1is a state,ais an action in A1of lengtht, andscis
a piecewise continuous function from (0;t]toS1. Intu-
itively,scdescribes a possible path of states that the DM
goes through when performing action a, such that before a
starts, the DM was at s1.3Note that we do not require that
limt!0+sc(t) =s1. Intuitively, this means that there can
be a discrete change in state at the beginning of an inter-
val. This allows us to capture the types of discrete changes
considered in semi-MDPs [Puterman 1994].
We think of R1(s1;sc;a)as the reward for transitioning
froms1according to state path scvia actiona. We assume
thatR1is bounded: speciﬁcally, there exists a constant
csuch thatR1(s1;sc;a)< cjaj. Fors12S1and
a2A1, we takeP1(s1;;a)to be a probability density
function over state paths of length jajstarting ats1.P1is
not deﬁned for transitions starting at terminal states.
We require R1andP1to be continuous functions,
so that if (si;si
c;ai)approaches (s;sc;a)(where all the
state sequences and actions have the same length t), then
R1(si;si
c;ai)approachesR1(s;sc;a)andP1(si;si
c;ai)
3We are thus implicitly assuming that the result of perform-
ing a piecewise continuous action must be a piecewise continuous
state path.
approaches P1(s;sc;a). To make the notion of “ap-
proaches” precise, we need to consider the distance
between state paths and the distance between actions.
Since we have identiﬁed both states (resp., basic ac-
tions) with subsets of IRn(resp.,IRm), this is straight-
forward. For deﬁniteness, we deﬁne the distance be-
tween two vectors in IRnusing theL1norm, so that
d(~ p;~ q) =Pn
1jpiqij. For actions aanda0inA1of
the same length, deﬁne d(a;a0) =Rjaj
t=0d(a(t);a0(t))dt.
For state paths scands0
cof the same length, de-
ﬁned(sc;s0
c) =Rjscj
t=0d(sc(t);s0
c(t))dt. Finally, deﬁne
d((sc;a);(s0
c;a0)) =d(sc;s0
c)+d(a;a0). This deﬁnition of
distance allows us to formalize the notion of continuity for
R1andP1. The key point of the continuity assumption is
that it allows us to work with discretizations, knowing that
they really do approximate the continuous MDP.
Constraints on Actions: We typically do not want to take
A1to consist of all possible piecewise continuous func-
tions. For one thing, some hardware and software restric-
tions will make certain functions infeasible. For example,
turning a steering wheel back and forth 1020times in one
second can certainly be described by a continuous function,
but is obviously infeasible in practice. But we may want to
impose further constraints on A1andg1(s).
In the discussion above, we did not place any constraints
on the length of actions. When we analyze problems of
interest, there is typically an upper bound on the length of
actions of interest. For example, when playing table ten-
nis using a robotic arm, the basic actions can be viewed as
tuples, describing the direction of movement of the racket,
the rotation of the racket, and the force being applied to
the racket; actions are intuitively all possible control se-
quences of racket movements that are feasible according to
the robot’s hardware and software constraints; this includes
slight movements of the racket, strokes, and preﬁxes of
strokes. An example of a piecewise continuous action here
would be to move the racket forward with a ﬁxed force for
some amount of time, and then to suddenly stop applying
the force when the racket is close to the ball. We can bound
the length of actions of interest to the time that a ball can
be in the air between consecutive turns. We assume that,
in any case, there is an upper bound Ton the length of all
actions. This seems to be a reasonable assumption for all
the applications of interest to us here.
Awareness: Even with the constraints discussed above,
A1is typically extremely large. Of course, not all actions
inA1are “useful”. For instance, in the helicopter exam-
ple, most actions would crash the helicopter. We thus con-
sider potentially useful actions. (We sometimes call them
justuseful actions .) Informally, an action is potentially use-
ful if it is not useless . A useless action is one that either
destroys the robot, or leaves it in an uncontrollable state,
or does not change the state. For example, when ﬂying a
helicopter, actions that lead to a crash are useless, as are ac-
tions that make the helicopter lose control. More formally,given a state s, the set of useful actions at state sare the
actions that transit to a different state in which the robot
is neither destroyed nor uncontrollable. Note that an ac-
tion that crashes the helicopter in one state may not cause
a crash in a different state. For robotics applications, we
say that a robot is aware of an action if it identiﬁes that
action as a potentially useful action, either because it has
been preprogrammed with the action (we are implicitly as-
suming that the robot understands all actions with which it
has been programmed) or it has simulated the action. For
example, a humanoid robot that has been pre-programmed
with only simple walking actions, and has never tried run-
ning or simulated running before, would be unaware of run-
ning actions. Let A1denote the useful actions in A1, and
letA10denote the useful actions that the robot is initially
aware of. (These are usually the actions that the robot has
been pre-programmed with.)
Discretization: We now consider the discretization of
M1. We assume that, for each discretization level i,S1is
discretized into a ﬁnite state space SiandABis discretized
into a ﬁnite basic action space ABi, wherejS1jjS2j
:::andjAB1jjAB2j:::. We further assume that,
for alli, there exists di>0, withdi!0, such that for
all statess2S1and basic actions aB2AB, there ex-
ists a states02Siand a basic action aB02ABisuch
thatd(s;s0)di, andd(aB;a0
B)di. Thus, we are as-
suming that the discretizations can give closer and closer
approximations to all states and basic actions. At level i,
we also discretize time into time slices of length ti, where
Tt1> t2> ::: .Thus, actions at discretization level
iare sequences of constant actions of lengthti, where a
constant action is a constant function from (0;ti]to a sin-
gle basic action.4In other words, the action lengths at dis-
cretization level iare multiples of ti. Thus, at discretization
leveli, there arePbT=tic
l=1jABijlpossible actions. Let A0
i
consist of this set of actions. (Note that some actions in A0
i
may not be in A1, since certain action sequences might be
infeasible due to hardware and software constraints.) Let
AiA0
ibe the set of useful actions at level i.
LetMibe the MDPU where SiandAiare deﬁned above;
A10is the set of useful actions that the DM is initially
aware of;g(s)is the set of useful actions at state s;g0(s)
is the set of useful actions that the DM is aware of at
states; and the reward function Riis just the restric-
tion ofR1toAiandSi. Fors12Sianda2Ai,
we takePi(s1;;a)to be a probability distribution over
Qjaj
i, the set of state paths of length jajthat are piece-
wise constant and each constant section has a length that
is a multiple of ti. For a state path sc2Qjaj
i, let
Pi(s1;sc;a)be the normalized probability of traversing a
state sequence that is within distance diof state sequence
scwhen playing action astarting from state s1. Formally,
4Note that we are not assuming that the action space Ai+1
is a reﬁnement of Ai(which would further require ti+1to be a
multiple ofti).
Pi(s1;sc;a) = (R
fs0c:d(sc;s0c)digdP1(s1;;a))=c, where
c=P
sc2Qjaj
iR
fs0c:d(sc;s0c)digdP1(s1;;a)is a normal-
ization constant. Since the robot is not assumed to know all
useful actions at any speciﬁc discretization level, it needs
to explore for the useful actions it wasn’t aware of. Finally,
given a speciﬁc exploration strategy, Di(j;t)describes the
probability of discovering a new useful action at discretiza-
tion leveli, given that there are jundiscovered useful ac-
tions at level i, and the robot has explored ttimes without
ﬁnding a useful action. We model exploration using a0;
every time the robot explores, it is playing a0.
It remains to deﬁne the discretization of an action in A0. In
order to do this, for a2A1, deﬁneai2Aito be a best ap-
proximation to ain leveliifjaijis the largest multiple of ti
that is less than or equal to jaj, andR(jaij)
0d(a(t);ai(t))dt
is minimal among actions a02Aof lengthjaij. Intuitively,
aiis an action in Aiwhose length is as close as possible to
that ofaand, among actions of that length, is closest in dis-
tance toa. The action aiis not unique. For a2A0, deﬁne
its discretization at level ito be a best approximation to a
at that level. When there are several best approximations,
we choose any one of them.
Policies: As usual, a policy inMiis a function from Sito
Ai. We want to compute UMi(s;;t ), the expected average
reward over time tofstarted in state s2Si. Letaj2Ai
andscjbe a state sequence in Qjajj
i, forj= 0;:::;l .
Say that a sequence ((a0;sc0);(a1;sc1);;(al;scl))isa
path compatible with policy starting atsif(s) =a0
and(scj(jajj)) =aj+1for all 0jl1. Let
I
s;tconsist of all paths ((a0;sc0);(a1;sc1);;(al;scl))
starting atscompatible with such thatPl
j=0jajj
t <Pl+1
j=0jajj, whereal+1=(scl(jalj)). Essentially,
when computing UMi(s;;t ), we consider the expected re-
ward over all maximally long paths that have total length
at mostt. Thus,UMi(s;;t ) =P
p2I
s;tP
i(p)R
i(p)
t,
where, given a path p= ((a0;sc0);(a1;sc1);;(al;scl)),
P
i(p) = l
j=0Pi(scj(0);scj;aj), andR
i(p) =Pl
j=0Ri(scj(0);scj;aj).
Now that we have deﬁned the average reward of a policy
at discretization level i, we can deﬁne the average reward
of a policy in M1. Given a discretization level i, leti
be a projection of1at leveli, deﬁned as follows: for
eachsi2Si, deﬁnei(si)to be an action ai2Aisuch
thataiis a best approximation to (si)at leveli, as de-
ﬁned above. As mentioned, there might be several best
approximations; aiis not unique. Thus, the projection is
not unique. Nevertheless, we deﬁne UM1(s;1;t)to be
limi!1UMi(s;i;t), whereiis a projection of to dis-
cretization level i. The continuity of the transition and re-
ward functions guarantees that the limit exists and is inde-
pendent of the choice of projections.
We now consider how the URMAX algorithm of Section 2
can be applied to learn near-optimal policies. We useURMAX at each discretization level. Note that URMAX
never terminates; however, it eventually learns to play near-
optimally (although we may not know exactly when). The
time it takes to learn to play near-optimally depends on
the exploration strategy. The next theorem consider brute-
force searching, where, at discretization level i, at each dis-
cretization level i, all actions in A0
iare exhaustively exam-
ined to ﬁnd useful actions. (The proof of this and all other
theorems can be found in the supplementary material.)
Theorem 3.1 Using brute-force exploration, given  > 0
and0<  < 1, we can ﬁnd an -optimal policy in M1
with probability at least 1in time polynomial in l,jA0
lj,
jSlj,1=,1=,Rl
max, andTl, wherelis the leastisuch that
the optimal policy for M0
iis(=2)-optimal for M1,Rl
max
is the maximum reward that can be obtained by a transition
inM0
l, andTlis the-return mixing time for M0
l.
Although brute-force exploration always learns a near-
optimal policy, the method can be very inefﬁcient, since
it exhaustively checks all possible actions to ﬁnd the use-
ful ones. Thus, at discretization level i, it needs to checkPbT=tic
l=1jABijlactions, and as igrows, the method soon
becomes impractical. On the other hand, the result is of
some interest, since it shows that even when there are in-
ﬁnitely many possible levels of discretizations, a method as
simple as brute-force exploration sufﬁces.
When the number of possible actions is huge, the probabil-
ity of ﬁnding a potentially useful action can be very low.
In this case, making use of an expert’s knowledge or imi-
tating a teacher’s demonstration can often greatly increase
the probability of ﬁnding a useful action. We abstract the
presence of an expert or a teacher by assuming that there is
some constant  > 0such thatD(1;t)for allt. In-
tuitively, the presence of a teacher or an expert guarantees
that there is a minimal probability such that, if there is a
new action to be found at all, then the probability of ﬁnding
it is at least , no matter how many earlier failed attempts
there have been at ﬁnding a useful action.
Using apprentice learning lets us improve Theorem 3.1 by
replacing thejA0
ljcomponent of the running time by jAlj;
thus, with apprentice learning, the running time depends
only on the number of useful actions, not the total number
of potential actions. The savings can be huge.
Theorem 3.2 Using an exploration method where
Di(1;t)for alli;t > 0(where2(0;1)is a
constant), for all  > 0and0<  < 1, we can ﬁnd an
-optimal policy in M1with probability at least 1in
time polynomial in l,jAlj,jSlj,1=,1=,1=,Rmax, and
Tl, wherelis the smallest isuch that the optimal policy
forM0
iis(=2)-optimal to M1,Rl
max is the maximum
reward that can be obtained by a transition in M0
l, andTl
is the-return mixing time for M0
l.
Figure 1: The arena with the robot at the center; and the
robot.
4 HUMANOID ROBOT WALKING
We consider the problem of a humanoid robot with 20 joint
motors (which we sometimes call just “joints”) learning to
walk on its own. More precisely, we require the robot to
move from the center of an arena to its boundary; we take
any reasonable motion to be “walking”. (Figure 1 shows
the robot and the arena in which it must walk.)
4.1 The continuous MDP
We start by deﬁning a continuous M1for the robot prob-
lem. A state s2S1is of the form s= (w1;;w23)2
IR23, where (w1;w2;w3)give the position of the robot’s
center of mass and (w4;;w23)are the current positions
of the robot’s 20 joint motors. We deﬁne the domain of
each dimension as follows: since the radius of the arena
is 5 meters, w1;w22[5;5]; since the robot’s height is
0.454 meters, w32[0;0:4](we do not expect the robot’s
center of mass to be higher than 0.4). Each joint mo-
tor has its speciﬁc range of mobility, which determines
the domain of the corresponding dimension. For example,
w52[3:14;2:85]represents the current position of the
robot’s left shoulder. The mobility range for all joint mo-
tors are intervals in [;].
The basic actions a2ABare of the form a=
(v1;;v20)2IR20, whereviis the target position for
the robot’sith joint motor. The domain of each dimension
is the mobility range for the corresponding joint motor. For
example,v2, which corresponds to the left shoulder, has
mobility range [3:14;2:85];v2= 2:85means to move
the robot’s left shoulder forward as far as possible. Since
walking is composed of repeated short movements that are
typically no longer than half a second, we set T= 0:512
seconds. Thus, A1, the set of useful actions, consists of
piecewise continuous functions that map from time to basic
actions and comply with the robot’s hardware and software
limitations, of length t<0:512seconds.
We now deﬁne R1andP1. Intuitively, the robot
obtains a reward for gaining distance from the cen-
ter of the arena. If the coordinates of the center
of the arena are given by s0= (s0[1];s0[2]), then
R1(s1;sc;a) =dis(s0;sc(jaj))dis(s0;s1), where
dis(s0;s1) =p
(s0[1]s1[1])2+ (s0[2]s1[2])2is theL2-norm distance between s0ands1on the (x;y)-plane.
The reward could be negative, for example, if the robot
moves back towards the center of the arena.
By deﬁnition, P1(s1;;a)is a probability distribution over
state sequences of length jajstarting ats1. For example, if
the robot slowly moves its right leg forward while staying
balanced, the state path taken by the robot is a determin-
istic path. On the other hand, if ais the action of turn-
ing around quickly, P1(s;;a)is distribution over various
ways of falling down.
4.2 Discretizations
We now deﬁne MiandM0
i. In our experiments we con-
sidered only levels 2 and 3 (level 1 is uninteresting since
it has just one state and one action), so these are the only
levels that we describe in detail here. (These turn out to
sufﬁce to get interesting walking behaviors.) At these lev-
els, we discretized more ﬁnely the joints corresponding to
the left and right upper and lower leg joints and the left and
right ankle joints, since these turn out to be more critical
for walking. (These are components (w14;;w19)in the
state tuples and (v11;;v16)in basic-actions tuples.) We
call these the relevant dimensions . We assume that the six
relevant state and actions components have ipossible val-
ues at leveli, fori= 2;3, as doesw3, since this describes
how high off the ground the robot is (and thus, whether or
not it has fallen). All other dimensions take just one value.
We tookt2=t3to be 128ms. SinceT= 0:512s, an action
contains at mostbT=tic= 4basic actions.
A10is the set of preprogrammed actions. We preprogram
the robot with a simple sitting action that lets the robot
slowly return to its initial sitting gesture. When we con-
sider apprenticeship learning, we also assume that the robot
is preprogrammed with a “stand-up” action, that enables it
to stand up from its initial sitting position. (Intuitively, we
are assuming that the expert taught the robot how to stand
up, since this is useful after it has fallen.)
A0
iis the set of potential actions at level i. Given our
assumptions, for i= 2;3, at leveli, there are (i6)4
potential actions (there are ipossible values for each
of the six relevant dimensions, and each action is a se-
quence of four basic actions). Thus, at level 3, there are
(36)4=282,429,536,481 potential actions. As we men-
tioned, a useful action is an action that moves the robot
without making it lose control. Here, an action is useful
if it moves the robot without resulting in the robot falling
down. At both levels 2 and 3, more than 80 useful actions
were found in our experiments. The most efﬁcient action
found at level 3 was one where the right leg moves back-
wards, immediately followed by the left leg, in such a way
that the robot maintains its balance at all times. By way
of contrast, turning the body quickly makes the robot lose
control and fall down, so is useless.
Fors12Si,a2Ai, andsc2Qjaj
i,Pi(s1;sc;a)is the
normalized probability of traversing a state sequence that
isdiclose tosc, a sequence of states in Si, where we de-
ﬁnedi=12
i+ 28+ 20:4. Sodidecreases in i, and dis-
cretizations at a higher level better approximate the contin-
uous problem. All basic actions in ABare within distance
diof a basic action in ABiand all states in Sare withindi
of a state in Si. Lets2S, and letsibe the closest state to
sinSi. It is easy to check that d(s;si)difori= 2;3.
TheDifunction depends on the exploration method used
to discover new actions. In our experiment, we used
two exploration methods: brute-force exploration and
apprenticeship-learning exploration.
At discretization level i, using brute-force exploration, we
haveDi(jAij;t) =jAij
jA0
ij, since there arejAijuseful actions
andjA0
ijpotential actions, and we test an action at random.
With apprenticeship learning, we used following hints from
a human expert to increase the probability of discovering
new actions: (a) a sequence of moving directions that, ac-
cording to the human expert, resembles human walking;5
(b) a preprogrammed stand-up action; (c) the information
that an action that is symmetric to a useful action is also
likely to be useful (two actions are symmetric if they are ex-
actly the same except that the target values for the left joints
and those for the right joints are switched). We also use a
different discretization: the ankle joint was discretized into
10 values. The human expert suggests more values in the
ankle joints because whether or not the robot falls depends
critically on the exact ankle joint position. These hints were
provided before the policy starts running; the discretization
levels are set then too. There were no further human-robot
interactions.
4.3 Experiments
For our experiments, we simulated DARwIn OP, a com-
mercially available humanoid robot. The simulations were
conducted on Webots PRO platform 8.2.1 using a Mac-
Book Pro with 2.8GHz Intel Core i7 Processor, 16GB 1600
MHz DDR3 memory, 0.5TB Flash Storage Mac HD, on OS
X Yosemite 10.10.5. We modeled the robot walking prob-
lem as an MDPU, and implemented the URMAX algorithm
using programming language Python 2.7.7.
As we said, given the number of actions involved, we con-
ducted experiments only for discretization level 2 and 3.
Both sufﬁced to enable the robot to learn to walk, using a
generous notion of “walk”—more precisely, they sufﬁced
to enable the robot to learn to locomote to the boundary of
the arena. As mentioned, two exploration methods were
used: brute-force exploration and apprenticeship-learning
exploration. One trial was run for brute-force exploration at
each of levels 2 and 3, and one trial was run for apprentice-
ship learning at level 2. Each trial took 24 hours. More than
5The sequence gives directions only for certain joints, with-
out speciﬁc target values, leaving the movement remaining joints
open for experimentation.Brute-force
(level 2)Brute-force
(level 3)Apprenticeship
learning (level 2)
jSij 130 1460 3200
jABij 64 729 1600
jAij 16777216 282429536481 6553600000000
ti(ms) 124 124 124
Length of
action (ms)496 496 496
Execution
time (hours)24 24 24
Best avg rwd
(m/action)0.043486 0.067599 0.083711
Num of useful
actions found131 89 180
Table 1: Performance comparisons.
Figure 2: A backward gait (from left to right).
15 stable gaits were found in total, where a gait is stable if
it enables the robot to move from the center of the arena to
the boundary without falling. In addition, more than 400
useful actions were found. The best gait among all stable
gaits achieved a velocity of 0.084m/s, which seems reason-
able, given that the best known walking speed of DARwIn-
OP is 0.341m/s [Budden et al. 2013]. Given more time to
experiment, we would expect the performance to improve
further.
The robot successfully learned gaits of various styles, in-
cluding both forward and backward gaits (see Figures 2 and
3), both efﬁcient and inefﬁcient gaits, gaits that resemble
human walking and the ones that do not. Somewhat sur-
prisingly, the best gait actually walks backwards. (Videos
of some of the gaits and a demo of the learning process
can be found at https://youtu.be/qW51iInpdV0.) As shown
in Table 1, as the discretization level increases, both the
velocity of the best gait and the number of useful actions
found increase. This agrees with the expectation that ﬁner
discretization better approximates the continuous problem,
and thus gets an expected reward closer to the optimal re-
ward of the continuous problem. Apprenticeship learning
resulted in more useful actions than the brute-force explo-
ration and in gaits with a higher average reward. Again,
this is hardly surprising; the hints provided by the human
expert increases the probability of ﬁnding useful actions.
On the other hand, when the expert gives “bad” hints, the
Figure 3: A forward gait (from left to right).
robot performs worse than with brute-force exploration.
With regard to comparisons under the same setting, we
implemented two baseline comparisons for bipedal walk-
ing. The ﬁrst tries random sequences of actions; the sec-
ond searches for useful actions and then repeats each use-
ful action in an attempt to ﬁnd stable gaits. (The motiva-
tion for repeating actions is that human-like walking is in
fact a rhythmic/cyclic movement of a repeated action.) Us-
ing the same setting as our experiments with URMAX, the
ﬁrst baseline found no stable gait in 24 hours, and the max-
imum distance the robot travelled before falling down was
0.342 meters; the second baseline found one stable gait in
24 hours, and the maximum distance travelled was 5 me-
ters (the maximum distance possible from the center of the
arena to the boundary) with a speed of 0.0058 m/s. Our
appraoch found several stable gaits at each discretization
level and the distance travelled using each stable gait is 5
meters with a maximum speed of 0.083m/s (10 times better
than the second baseline).
Our approach, using MDPUs, requires no knowledge on
the kinematics of the robot other than the number of joints
and the moving range of each joint. Moreover, it makes no
assumptions about the moving pattern of the resulting gait;
for example, we do not assume that a gait must be cyclic,
or symmetric between left and right joints, nor do we spec-
ify the length of a gait. Although we do specify the length
of a useful action, a gait could be composed of a single or
multiple useful actions. Given the few assumptions and lit-
tle prior knowledge assumed, the performance of the robot
seems quite reasonable. More importantly, the experiment
proves that the use of MDPUs enables the robot to learn
useful new maneuvers (walking, in this case) by itself, with
minimum human input.
5 RELATED WORK
There has been work on optimal policy learning in MDPs
using computational resources. Kearns and Singh’s [2002]
E3algorithm guarantees polynomial bounds on the re-
sources required to achieve near-optimal return in gen-
eral MDPs; variants and extensions of this work can be
found in [Brafman and Tennenholtz 2002; Kakade et al.
2003; Kearns and Koller 1999]. However, algorithms such
asE3usually require the exploration of the entire MDP
state/action space. This becomes impractical in our setting,
where the number of actions is extremely large. In such
cases, several exploration methods have been employed
to help ﬁnd useful actions. For example, Abbeel and Ng
[2005] utilize a teacher demonstration of the desired task
to guide the exploration; Dearden et al. [1999] utilize the
value of information to determine the sequence of explo-
ration. Other papers (e.g., [Dean et al. 1998; Hauskrecht
et al. 1998]) consider MDPs with large action spaces.
We are far from the ﬁrst to consider MDPs with continuous
time. For example, semi-MDPs (SMPDs) and continuous-time MDPs have continuous time [Puterman 1994]. How-
ever, these models have discrete actions that can be taken
instantaneously, and do not consider continuous actions
taken over some duration of time. In Markov decision drift
processes [Hordijk and Van der Duyn Schouten 1984], the
state does not have to stay constant between successive ac-
tions (unlike an SMDP), and can evolve in a determinis-
tic way according to what is called a drift function . But
Markov decision drift processes do not have actions with
probabilistic outcomes that take place over an interval of
time. They make signiﬁcant use of discrete approxima-
tions to compute optimal policies, just as we do. There
has also been work on MDPs with continuous state space
and action space (e.g., [Antos and Munos 2007; Feng et al.
2004]), but with discrete time. For our applications, we
need time, space, and actions to be continuous; this adds
new complications. In control theory, there are methods
for controlling continuous time systems where system tran-
sitions are linear function of state and time [Zinober 1989].
These can be extended to non-linear systems [Khalil 2002].
However, the transitions in these systems are usually de-
terministic, and they do not deal with rewards or policies.
Sutton, Precup, and Singh [1999] consider high-level ac-
tions (which they call options ) that are taken over a du-
ration of time (such as “opening a door”), but they view
time as discrete, which signiﬁcantly simpliﬁes the model.
Rachelson, Garcia, and Fabiani [2008] consider continu-
ous actions over some time interval, however, they assume
there are decision epochs, which are the only time points
where rewards are considered. In our model, the rewards
depend on the entire state sequence that the system tra-
verses through while an action is taken. While this makes
the model more complicated, it seems more appropriate for
the problems of interest to us.
There has also been a great deal of work on bipedal robot
walking, since it is a fundamental motor task for which bi-
ological systems signiﬁcantly outperform current robotic
systems [Tedrake et al. 2005]. There have been three main
approaches for solving the task:
The ﬁrst approach describes the kinematics of the
robot in detail using non-linear and linear equation
systems, then solves these systems to obtain desirable
trajectories. See, for example, [Huang et al. 2001;
Gonalves and Zampieri 2006; Kajita et al. 2003; Kim
et al. 2007; Strom et al. 2010; Xue et al. 2012].
The second approach uses genetic algorithms [Cheng
and Lin 1997; Hasegawa et al. 2000; Picado et al.
2009]. The traits describing a gait are taken to be
the genes in a genetic algorithm. Different gaits (i.e.,
settings of the parameters) are evaluated in terms of
features such as stability and velocity; The most suc-
cessful gaits are retained, and used to produce the next
generation of gaits through selection, mutation, inver-
sion, and crossover of their genes.
The third approach uses gradient learning, which
starts with either a working gait or a randomly ini-
tialized gait. It then improves the gait’s performance
by changing its parameters, using machine-learning
methods (such as neural networks) to ﬁnd the most
proﬁtable set of changes in the parameters. See, for
example, [Budden et al. 2013; Kim and Uther 2003;
Schulman et al. 2015; Tedrake et al. 2005].
Since the ﬁrst approach requires a full description of the
robot’s kinematics, as well as composing and solving a
non-linear system, it requires a great deal of human in-
put. Moreover, its application is limited to walking prob-
lems. The approach is unable to produce gaits other than
those speciﬁed by human (e.g., to walk forward by step-
ping forward the left and the right legs in turn under a spe-
ciﬁc speed). Both the second and the third approach require
little human input (when starting from random gaits), and
may produce a variety of gaits. Both also have the potential
to be generalized to problems other than bipedal walking.
However, both are heuristic search algorithms, and have
no theoretical guarantee on their performance. In contrast,
our method produces a variety of gaits, provides a general
framework for solving robotic problems, and produces a
near-optimal policy in the limit. Moreover, our method re-
quires minimum human input, although, as the experiments
show, it does better with more human input.
A comparison of our method and the genetic algorithm may
provide insights into both methods. Although the two ap-
proaches seem different, the searching process made pos-
sible by selection, mutation, inversion, and crossover in a
genetic algorithm can be viewed as a special case of the
explore action in an MDPU. Conversely, the explore ac-
tion in an MDPU for the robot can be roughly viewed as
searching for a set of genes of unknown length (since a gait
can be understood as a continuous action over an uncertain
amount of time, composed of one or more shorter actions,
where each shorter action is described by a set of parame-
ters). Our approach can be viewed as being more ﬂexible
than a genetic algorithm; in a genetic algorithm, the length
of the chromosome (i.e., the number of parameters that de-
scribe the gait) is ﬁxed; only their values that give the best
performance are unknown.
The recent work of Mordatch et al [2016] also provides a
general approach for reinforcement learning in robot tasks.
Like us, they require prior knowledge only of the mobility
range of each of the robot’s joints, and not their kinemat-
ics; they also model the problem as an MDP. Their goal
is to ﬁnd an optimal trajectory (i.e., a sequence of states),
such that when followed, performs a desired task (such as
reaching out the robot’s hand to a desired position) with
minimal cost. They use neural networks to solve the cost-
minimization problem. Thus, their approach does not have
any guarantees of (approximate) optimality of the perfor-
mance. Moreover, the complexity of their approach grows
quickly as the length of the trajectory grows (while ours is
polynomial in the number of useful actions, states visited,and the difﬁculty of discovering new actions, and thus is not
signiﬁcantly affected by the length of the trajectory). That
said, Mordatch et al.’s method has successfully learned a
few relatively simple tasks on a physical DARwIn OP2
robot, including hand reaching and leaning the robot’s torso
to a desired position [Mordatch et al. 2016], although it has
not yet been applied to walking.
6 CONCLUSION
We have provided a general approach that allows robots to
learn new tasks on their own. We make no assumptions on
the structure of the tasks to be learned. We proved that in
the limit, the method gives a near-optimal policy. The ap-
proach can be easily applied to various robotic tasks. We il-
lustrated this by applying it to the problem of bipedal walk-
ing. Using the approach, a humanoid robot, DARwIn OP,
was able to learn various walking gaits via simulations (see
https://youtu.be/qW51iInpdV0 for a video). We plan to ap-
ply our approach to more robotic tasks, such as learning to
run and to walk up and down stairs. We believe the process
will be quite instructive in terms of adding useful learn-
ing heuristics to our approach, both speciﬁc to these tasks
and to more general robotic tasks. We are also interested
in having the robot simulate learning to walk in the same
way a baby does, for example, by limiting the robot’s abil-
ities initially, so that it must crawl before it walks. Part of
our interest lies in seeing if such initial limitations actually
make learning more efﬁcient.
Acknowledgments: The work of Halpern and Rong was
supported in part by NSF grants IIS-0911036 and CCF-
1214844, and by AFOSR grant FA9550-12-1-0040, and
ARO grant W911NF-14-1-0017. The work of Saxena was
supported in part by a Microsoft Faculty Fellowship.
References
Abbeel, P. and A. Y . Ng (2005). Exploration and ap-
prenticeship learning in reinforcement learning. In
Proc. 22nd Int. Conf. on Machine Learning (ICML
’05), pp. 1–8.
Antos, A. and R. Munos (2007). Fitted Q-iteration in
continuous action-space MDPs. In Advances in Neu-
ral Information Processing Systems 20 (Proc. of
NIPS 2007) , pp. 9–16.
Brafman, R. I. and M. Tennenholtz (2002). R-MAX: A
general polynomial time algorithm for near-optimal
reinforcement learning. Journal of Machine Learn-
ing Research 3 , 213–231.
Budden, D., J. Walker, M. Flannery, and A. Mendes
(2013). Probabilistic gradient ascent with applica-
tions to bipedal robot locomotion. In Australasian
Conference on Robotics and Automation (ACRA
2013) , pp. 37–45.
Cheng, M.-Y . and C.-S. Lin (1997). Genetic algorithm
for control design of biped locomotion. Journal of
Robotic Systems 14 (5), 365–373.
Dean, T., K. Kim, and R. Givan (1998). Solving stochas-
tic planning problems with large state and action
spaces. In Proc. 4th Int. Conf. on Artiﬁcial Intelli-
gence Planning Systems , pp. 102–110.
Dearden, R., N. Friedman, and D. Andre (1999). Model
based bayesian exploration. In Proc. 15th Conf. on
Uncertainty in AI (UAI ’99) , pp. 150–159.
Feng, Z., R. Dearden, N. Meuleau, and R. Washington
(2004). Dynamic programming for structured con-
tinuous Markov decision problems. In Proc. 20th
Conf. on Uncertainty in AI , pp. 154–161.
Gonalves, J. B. and D. E. Zampieri (2006, 12). An in-
tegrated control for a biped walking robot. Journal
of the Brazilian Society of Mechanical Sciences and
Engineering 28 , 453 – 460.
Halpern, J. Y ., N. Rong, and A. Saxena (2010). MDPs
with unawareness. In Proc. 26th Conf. on Uncer-
tainty in AI (UAI ’10) , pp. 228–235.
Hasegawa, Y ., T. Arakawa, and T. Fukuda (2000).
Trajectory generation for biped locomotion robot.
Mechatronics 10 , 67–89.
Hauskrecht, M., N. Meuleau, L. Kaelbling, T. Dean,
and C. Boutilier (1998). Hierarchical solution of
Markov decision processes using macro-actions. In
Proc. 14th Conf. on Uncertainty in AI (UAI ’98) , pp.
220–229.
Hordijk, A. and F. A. Van der Duyn Schouten (1984).
Discretization and weak convergence in Markov de-
cision drift processes. Mathematics of Operations
Research 9 , 112–141.
Huang, Q., K. Yokoi, S. Kajita, K. Kaneko, H. Arai,
N. Koyachi, and K. Tanie (2001). Planning walk-
ing patterns for a biped robot. IEEE Transactions on
Robotics and Automation 17 , 280–289.
Kajita, S., F. Kanehiro, K. Kaneko, K. Fujiwara,
K. Harada, K. Yokoi, and H. Hirukawa (2003).
Biped walking pattern generation by using pre-
view control of zero-moment point. In Proc. Inter-
national Conference on Robotics and Automation
(ICRA 2003), Vol. 2 , pp. 1620 – 1626.
Kakade, S., M. Kearns, and J. Langford (2003). Explo-
ration in metric state spaces. In Proc. 20th Int. Conf.
on Machine Learning (ICML ’03) , pp. 306–312.
Kearns, M. and D. Koller (1999). Efﬁcient reinforce-
ment learning in factored MDPs. In Proc. 16th Int.
Joint Conf. on AI (IJCAI ’99) , pp. 740–747.
Kearns, M. and S. Singh (2002). Near-optimal reinforce-
ment learning in polynomial time. Machine Learn-
ing 49 (2-3), 209–232.
Khalil, H. K. (2002). Nonlinear Systems . Prentice-Hall.Kim, J., I. Park, and J. Oh (2007). Walking control al-
gorithm of biped humanoid robot on uneven and in-
clined ﬂoor. Journal of Intelligent and Robotic Sys-
tems 48 (4), 457–484.
Kim, M. S. and W. Uther (2003). Automatic gait optimi-
sation for quadruped robots. In Proc. Australasian
Conference on Robotics and Automation (ACRA) .
Mordatch, I., N. Mishra, C. Eppner, and
P. Abbeel (2016). Combining model-based
policy search with online model learning
for control of physical humanoids. Preprint,
http://www.eecs.berkeley.edu/ igor.mordatch/darwin/
paper.pdf.
Picado, H., M. Gestal, N. Lau, L. P. Reis, and A. M.
Tome (2009). Automatic generation of biped walk
behavior using genetic algorithms. In Proc. 10th
International Work-Conference on Artiﬁcial Neural
Networks , pp. 805–812.
Puterman, M. L. (1994). Markov Decision Processes .
John Wiley and Sons, Inc.
Rachelson, E., F. Garcia, and P. Fabiani (2008). Extend-
ing the bellman equation for MDPs to continuous ac-
tions and continuous time in the discounted case. In
10th Int. Symp. on Artiﬁcial Intelligence and Math-
ematics .
Schulman, J., S. Levine, P. Moritz, M. Jordan, and
P. Abbeel (2015). Trust region policy optimization.
InProc. 31st International Conference on Machine
Learning (ICML ’15) , pp. 1889–1897.
Strom, J., G. Slavov, and E. Chown (2010). Omnidirec-
tional walking using ZMP and preview control for
the nao humanoid robot. In RoboCup 2009: Robot
Soccer World Cup XIII , pp. 378–389.
Sutton, R., D. Precup, and S. Singh (1999). Between
MDPs and semi-MDPs: A framework for temporal
abstraction in reinforcement learning. Artiﬁcial In-
telligence 112 , 181–211.
Tedrake, R., T. W. Zhang, and H. S. Seung (2005).
Learning to walk in 20 minutes. In Proc. Fourteenth
Yale Workshop on Adaptive and Learning Systems .
Xue, F., X. Chen, J. Liu, and D. Nardi (2012). Real time
biped walking gait pattern generator for a real robot.
In T. R ¨ofer, N. M. Mayer, J. Savage, and U. Saranlı
(Eds.), RoboCup 2011: Robot Soccer World Cup
XV, pp. 210–221. Springer.
Zinober, A. S. (1989). Deterministic control of uncertain
systems. In Proc. Int. Conf. on Control and Appli-
cations (ICCON ’89) , pp. 645–650.

Nan Rong
Joseph Y
Halpern
Ashutosh Saxena


Market Manipulation: An Adversarial Learning Framework for Detection and Evasion.

Market Manipulation: An Adversarial Learning Framework for Detection and
Evasion
Xintong Wang andMichael P. Wellman
University of Michigan, Ann Arbor
xintongw@umich.edu, wellman@umich.edu
Abstract
We propose an adversarial learning framework to
capture the evolving game between a regulator who
develops tools to detect market manipulation and
a manipulator who obfuscates actions to evade de-
tection. The model includes three main parts: (1) a
generator that learns to adapt original manipulation
order streams to resemble trading patterns of a nor-
mal trader while preserving the manipulation intent;
(2) a discriminator that differentiates the adversari-
ally adapted manipulation order streams from nor-
mal trading activities; and (3) an agent-based simula-
tor that evaluates the manipulation effect of adapted
outputs. We conduct experiments on simulated or-
der streams associated with a manipulator and a
market-making agent respectively. We show ex-
amples of adapted manipulation order streams that
mimic a speciﬁed market maker’s quoting patterns
and appear qualitatively different from the original
manipulation strategy we implemented in the sim-
ulator. These results demonstrate the possibility of
automatically generating a diverse set of (unseen)
manipulation strategies that can facilitate the train-
ing of more robust detection algorithms.
1 Introduction
Market manipulation is deﬁned by the US Securities and Ex-
change Commission as “intentional conduct designed to de-
ceive investors by controlling or artiﬁcially affecting the mar-
ket for a security”. Though it has long been present, manipu-
lation practice has evolved in its forms, and is of increasing
concern with the automation of trading and the interconnect-
edness of ﬁnancial markets [Lin, 2015 ]. Automated programs
are employed to inject deceitful information, as other traders
make extensive uses of machine learning techniques to extract
information from all possible sources (including the mislead-
ing ones) and execute decisions.
We focus on a speciﬁc but common form of manipulation,
called spooﬁng , which is applied through a series of direct
trading actions in a market. Traders interact with the market
by submitting orders to buy or sell; we refer to the sequence of
such actions taken by an individual trader over a period of time
Figure 1: An example of spooﬁng activities conducted over the course
of 0.6 seconds. A series of large out-of-the money manipulation sell
orders (red triangles) are ﬁrst placed to drive prices down and make
the buy order accepted (the ﬁlled blue triangle). The deceptive sell
orders are immediately replaced with large buy ones (blue triangles)
to push the price up and proﬁt from the sale at a higher price (the
ﬁlled red triangle). Source: UK Financial Conduct Authority.
as the trader’s order stream . Orders that do not execute im-
mediately rest in the order book , a repository for outstanding
offers to trade. At any given time, the order book for a particu-
lar security reﬂects the market’s expressed supply and demand
for that security. A manipulative order stream can be viewed
as atargeted attack [Huang et al. , 2011 ], corrupting the order
book’s signal on supply and demand. False expressions in
the manipulative order are designed to fool traders about the
market state, leading them to alter trading behavior in a way
that will directly move the price and beneﬁt the manipulator.
Fig. 1 illustrates an alleged spooﬁng order stream.
The automated nature of many manipulative strategies has
also spurred efforts to automate detection. Nasdaq recently
announced an AI-based surveillance system trained with his-
torical data and spotted patterns of market-abuse techniques
to detect suspect equities trading practices [Rundle, 2019 ].
Despite recent advances, developing high-ﬁdelity detection
systems faces the challenge that an adversary often takes steps
to obfuscate their strategies in effort to escape detection (e.g.,
manipulating in a way that appears as normal trading activity).
This causes regulators to play a costly game of cat-and-mouse
with manipulators who constantly innovate to evade.
We propose an adversarial learning framework to reason
about how a manipulator might mask its behavior (represented
as the order stream) to evade detection of a given discrimina-
tive model. The idea is to let a generative model learn to adapt
existing manipulation strategies to resemble characteristics of
normal trading, while preserving a comparable manipulation
effect. A history of adapted order streams that effectively
manipulate are further used to improve the robustness of the
detector. We apply such adversarial reasoning recursively, up-
dating the generator and the discriminator level-by-level, and
characterize the evolution of adapted manipulation strategies.
Our generative model adopts the sequence-to-sequence
paradigm [Sutskever et al. , 2014 ], and takes a manipulation or-
der stream as source and a paired benign trader’s order stream
as target. It learns to adapt the source by minimizing the com-
bination of an adversarial loss and a self-regularization loss.
The adversarial loss is calculated by a discriminator that clas-
siﬁes an order stream as adapted from manipulation or target,
minimized as the output becomes indistinguishable from a
benign trader’s order stream. The self-regularization loss is
a feature-wise distance between the source and the adapted
stream, penalizing large changes between the two to preserve
the manipulation effect.
We conduct experiments and evaluate the proposed
approach using order streams generated by an agent-based
market simulator.1The simulator models simple manipulation
strategies similar to Fig.1 [Wang and Wellman, 2017 ], and
can practically produce a large set of order streams associated
with each agent across a variety of market conditions. We
run controlled simulations to acquire order streams associated
with a manipulator (SP) as source , and as target the order
streams that a market-making agent (MM) would have placed
under corresponding market conditions. To help quantify
the manipulation effect, we decompose the SP behavior
into manipulation and exploitation orders, and deﬁne a
baseline order stream (EXP) that omits the manipulation
orders (i.e., those not intended to execute). Our goal here
is to adapt manipulation order streams to resemble market-
making, a legitimate trading role with generally positive
inﬂuence on market efﬁciency [Schwartz and Peng, 2012;
Wah et al. , 2017 ]. Fig. 2 gives an overview of our approach.
Experimental results show that our proposed framework can
generate adapted manipulation order streams that resemble
quoting patterns of a market maker and appear qualitatively
different from the original manipulation strategy we imple-
mented in the simulator. This adaptation evades detection,
but at the cost of compromising effectiveness in manipulation.
After a few iterations of evolving and evading the detector, the
strategy has sacriﬁced almost all of its manipulation capability.
Though it is likely impossible to develop a detector immune
from adversarial attacks, modeling the evasion can be a useful
step toward more robust detection of market manipulation.
2 Related Work
Agent-Based Models of Market Manipulation To study
the effects of particular trading practices, researchers classify
market participants into different roles based on their trading
1Learning from real market data is infeasible, as actual order
streams identiﬁed as manipulation do not exist in any substantial
quantity.
(a) Update the generator and the discriminator level-by-level.
(b) Given a ﬁxed detector Dl1, trainGlto generate SP l.
Figure 2: Overview of our approach. We start with a classiﬁer D0
that discriminates between SP and MM order streams. In response, a
generatorG1learns to adapt SP order streams, producing SP1that
can evade detection by D0.SP1order streams are then incorporated
to train the next-level discriminator D1. We apply such adversarial
reasoning recursively, producing a sequence of adapted manipulators
and corresponding increasingly robust detectors.
intent and activity patterns (e.g., trading volume, frequency,
position). An agent-based market simulator designs agents
around such roles, and reproduces “stylized facts” observed in
real ﬁnancial markets through strategic interactions of these
agents [LeBaron, 2006; Kirilenko et al. , 2017 ].
In prior work, we developed an agent-based model of mar-
ket manipulation [Wang and Wellman, 2017 ], demonstrating
settings where a manipulation agent can effectively deceive
approximately rational background traders through spooﬁng.
Speciﬁcally, in markets populated with background learning
traders who bid based on beliefs induced from market obser-
vations including the malicious activities, the manipulator is
able to push prices signiﬁcantly higher than they would be oth-
erwise, and proﬁt from this manipulation. Since background
trading agents react to different market conditions according
to their codiﬁed strategies, the model can verify manipula-
tion intent and quantify its impact by conducting controlled
experiments of markets with and without a spooﬁng agent.
Learning via Adversarial Training There is a substantial
body of work on adversarial training [Goodfellow et al. , 2015;
Tzeng et al. , 2017; V olpi et al. , 2018; Sinha et al. , 2018 ],
investigating a variety of training procedures designed to learn
models robust to (adversarial) perturbations in the input. Many
of these approaches involve augmenting training dataset with
examples from a target domain that is considered “hard” under
the current model. A key issue addressed in some but not all
of this work is to preserve speciﬁed properties of the source
domain while generating adversarial examples to improve
robustness.
Our approach draws particular inspiration from Shrivastava
010002000300040005000
Time990009940099800100200100600101000Price(a) EXP (baseline).
010002000300040005000
Time990009940099800100200100600101000Price (b) SP (source).
010002000300040005000
Time990009940099800100200100600101000Price (c) MM (target).
Figure 3: Order streams associated with EXP, SP, and MM in a set of controlled simulations. During the execution stage (time before 1000),
both EXP and SP bought one share of the security at price 99,908. Then, SP maintained manipulation buy orders at a tick behind the best bid to
push the price up. As a result, SP managed to sell the share at price 100,102, whereas EXP sold the share at 100,044.
et al. [2017 ], who proposed Simulated + Unsupervised (S+U)
learning. The idea is to train a generative model to improve the
realism of simulated images using unlabeled real ones, while
preserving the annotation information from the simulator. A
pixel-level loss is further imposed between the simulated input
and the generated image to enforce annotation. Experimen-
tal results show that S+U learning enables the generation of
highly realistic images with reliable labels and helps to im-
prove learning models’ performance on classiﬁcation tasks,
including gaze estimation and hand pose estimation. Our work
extends the approach to adapt simulated order streams while
preserving the intent behind the original sequence of actions.
3 Formulation and Model
3.1 Trading Strategies and Representations
We follow prior work [Wang and Wellman, 2017; Wang et
al., 2018; Wah et al. , 2017 ]in the design of manipulation and
market-making strategy, extending each with a bit of ﬂexibility
to reduce overﬁtting to artifacts. We describe the trading
strategies and their representations as order streams below.2
Manipulation Strategy (SP) During each simulation run,
the manipulator aims to maneuver prices either up or down
with equal probability. We elaborate the case of manipulat-
ing prices up, and the other applies vice versa. The strategy
includes three stages. During the ﬁrst execution stage, the
agent buys by accepting any sell order at price lower than the
fundamental mean r. In the next manipulation stage, it stops
buying and instead maintains large manipulation buy limit or-
ders at price one tick below the best bid. The goal is to falsely
signal demand to push price up so that the units bought earlier
can be sold at higher prices later. During the last stage, the
manipulator starts to sell the units by accepting any buy orders
at a price higher than r. The agent continues to manipulate
until the trading period ends or all the bought units are sold.
Market-making Strategy (MM) Upon each arrival, the
market maker submits a quote ladder centered around an es-
timate of the terminal fundamental value of the underlying
security, denoted by ^rt. Speciﬁcally, the quote ladder is de-
cided by three strategic parameters !;K; that respectively
2Since an order stream is a sequence of actions incurred by a
strategy, we refer to a strategy and order streams associated with that
strategy interchangeably.control the quote spread, number of price levels, and the num-
ber of ticks between two adjacent prices:
(
[BtK;:::;B t(K)]for buy orders
[St+ (K);:::;S t+K]for sell orders,(1)
whereBt= ^rt!=2,St= ^rt+!=2, andandtruncate
the price ladder such that limit orders do not immediately
transact with the market’s current best bid and ask. We add
Gaussian noise around each price in Eq. (1)and its associated
quantity to mitigate certain artifacts (e.g., prices separated by
an equal distance). Since quote ladders are symmetrically cen-
tered around unbiased estimations of the terminal fundamental
value, the MM orders in expectation do not distort learning
traders’ pricing beliefs. The MM agent follows the same ar-
rival schedule as the manipulator to produce a paired target
order stream, which records orders that would have placed
under market conditions encountered by the manipulator.
Exploitation Strategy (EXP) The exploitation order
streams serve as the control group to measure the effect of
manipulation orders. The strategy executes the same buy and
sell scheme as the SP strategy during the ﬁrst and last stage
without placing any manipulation order.
Order Stream Representation An order stream records a
sequence of (hypothetical) actions associated with an agent. It
is represented by a variable-length sequence with an element
corresponding to each time an agent arrives and submits a bid
schedule . A bid schedule comprises a set of limit orders, each
specifying a price (expressed by distance to market quote) and
a quantity. Fig. 3 shows order streams respectively associated
with EXP, SP, and MM in a set of controlled simulations.
3.2 The Model
We use the market simulator to generate a dataset of labeled or-
der streamsD=f(wi;EXP);(xi;SP);(yi;MM)gN
i=1, where
wi,xi, andyidenote order streams incurred by their respective
strategies under one set of controlled simulations (like those
in Fig. 3). The goal here is to adapt the simulated SP order
streams to become indistinguishable from the MM ones while
preserving some manipulation effect.
Model Overview Our generator adopts the sequence-to-
sequence paradigm [Sutskever et al. , 2014 ], which consid-
ers the interconnection between bid schedules within a se-
quence (e.g., a manipulator who buys ﬁrst is more likely to
manipulate price up and later sell). It has an encoder-decoder
structureG= (Genc;Gdec), wheredenotes the function
parameters. This encoder-decoder model has been widely
used in tasks that require sequence-to-sequence learning, such
as the statistical machine translation [Sutskever et al. , 2014;
Cho et al. , 2014 ]and sentence generation [Logeswaran et
al., 2018 ]. The encoder adopts a recurrent neural network
(RNN) that takes an order stream xas input and produces a
ﬁxed-length latent representation vector zx:=Genc(x). The
vector contains compressed information of the input (e.g., ma-
nipulate prices up or down), and is decoded by Gdec, a second
RNN that generates x0pGdec(jzx)to resemble characteris-
tics of the target domain y. The discriminator Dalso uses an
RNN component followed by a linear layer, and outputs the
probability of an input being an adapted order stream.
A Recursive Training Procedure We propose a recursive
training procedure of the generator and the detector (depicted
in Fig. 2a), designed to mimic the adversarial reasoning be-
tween a manipulator and a regulator. The manipulator starts by
playing the SP strategy that is codiﬁed in our market simulator,
and the regulator develops detector D0to distinguish manipu-
lation order streams from MM streams. The manipulator then
constructs its next-level strategy SP1by learning a generator
G1to adapt SP, such that the adapted order streams can evade
the detection of D0and preserve a comparable manipulation
effect. To achieve both aims, the generator is trained to mini-
mize a combination of adversarial loss and regularization loss
(depicted in Fig. 2b), which we describe in detail below. In
response, a new detector D1is trained to identify both the
original manipulation strategy SP and the evolved one SP1.
We apply such reasoning recursively to generate adversarial
manipulation activities, so as to improve the robustness of a
detector.
Adversarial Loss We follow the GAN setup [Goodfellow et
al., 2014 ]which models the generator and the discriminator as
a two-player minimax game. During training, the level- ldis-
criminator network Dlupdates its parameters lto minimize
the following loss:
LD(l) =X
ilog(D(x0
i;l))X
ilog(1D(yi;l));(2)
wherex0
irepresents some learned (or identity) transformation
ofxi, andD()denotes the probability of the input order
stream either associated with or adapted from SP.
We ﬁx the discriminator Dl1and train the level- lgenerator
Glto maximize the probability of Dl1making a mistake.
Speciﬁcally, it learns lby minimizing the adversarial loss:
Ladv
G(l) =X
ilog(1Dl1(G(xi;l))): (3)
Self-Regularization Loss To preserve manipulation effect,
we combine the adversarial loss with a self-regularization
loss that penalizes any difference between the adapted and
original order stream. This can be interpreted as a manipulator
preference to adapt its original manipulation strategy as little
as possible to evade detection. We deﬁne regularization loss
as the mean squared error between the input and the adapted
order stream:
Lreg
G(l) =1
NX
ikG(xi;l)xik2
2; (4)Payoff Manipulation
EffectTransaction
RiskDl1
(%)Dl
(%)
SP 411;1 0 - 100
SP1362;0.50 0.14 0.59 100
SP23100.30 0.26 0 100
SP33030.22 0.59 0 100
MM 121 0.15 0.85 100 100
EXP 3240 0 - -
Table 1: Summary statistics of the respective trading strategy on test
dataset. Asterisks denote statistical signiﬁcance at 5%level of the
paired t-test for payoffs compared to MM ()and EXP ().
wherekk2is the L2 norm. The overall loss for GisLG=
Ladv
G+Lreg
G, whereis a hyperparameter.
Measuring Manipulation Effect We evaluate the manip-
ulation effect of an adapted order stream x0
i:=Gl(xi)by
feeding it back to the market simulator under the same set of
experimental controls. That is, we compare the effects under
scenarios where background traders are guaranteed to arrive
at the same time, receive identical private values, and observe
the same fundamental values as in simulations that generate
wi;xi, andyi. Any change in background bidding behavior
can therefore be attributed to the adapted order stream.
We compare market outcomes incurred by the adapted order
stream to those of markets with SP and EXP, and measure the
manipulation intensity andtransaction risk . The manipulation
intensity ofx0
i, denoted by x0
i, is deﬁned as the fraction of the
price deviation realized by x0
iin that of the SP order stream:
x0
i=8
>>><
>>>:minn
maxnPx0
iPwi
PxiPwi;0o
;1o
ifPxi>Pwi
minn
maxnPwiPx0
i
PwiPxi;0o
;1o
otherwise,(5)
wherePwi;Pxi, andPx0
idenote the average transaction price
in respective markets since the start of the manipulation stage.
The higher the manipulation intensity is, the better x0
ipreserves
the manipulation effect. Transaction risk is deﬁned as the ratio
between the number of transactions and the number of arrivals
during the manipulation phase. By deﬁnition, SP and EXP
have manipulation intensity one and zero, respectively, and
both exhibit transaction risk zero.
4 Experimental Results
We follow the proposed framework and generate adversarial
order streams by adapting the simulated SP order streams to
look like quoting patterns of a market maker. We visualize
examples of adapted manipulation activities, and demonstrate
the competing improvement between the adapted manipulation
strategies and the detectors.
4.1 Dataset and Implementation Details
We conduct simulations using the agent-based market simu-
lator, and generate 10,944 groups of labeled order streams
f(wi;EXP);(xi;SP);(yi;MM)g(out of 30,000 controlled
−2 −1 0 1 2
Distance to Best Quotes (x1000)0.0000.0010.0020.003DensitySP
SP1
SP2
SP3
MM(a) Price distribution.
0 50 100 150 200
Order Quantity0.000.050.100.150.20DensitySP
SP1
SP2
SP3
MM (b) Quantity distribution.
0 510 15 20 25 30
Order Imbalance Ratio0.00.20.40.60.81.01.2DensitySP
SP1
SP2
SP3
MM (c) Order imbalance distribution.
Figure 4: Comparisons of the respective statistics on the SP order streams, adapted outputs, and MM order streams.
0.0 0.2 0.4 0.6 0.8 1.0
Manipulation Intensity0.00.20.40.60.81.0Transaction Risk
0.00.20.40.60.81.0
(a) SP 1.
0.0 0.2 0.4 0.6 0.8 1.0
Manipulation Intensity0.00.20.40.60.81.0Transaction Risk
0.00.20.40.60.81.0 (b) SP 2.
0.0 0.2 0.4 0.6 0.8 1.0
Manipulation Intensity0.00.20.40.60.81.0Transaction Risk
0.00.20.40.60.81.0 (c) SP 3.
0.0 0.2 0.4 0.6 0.8 1.0
Manipulation Intensity0.00.20.40.60.81.0Transaction Risk
0.00.20.40.60.81.0 (d) MM.
Figure 5: The manipulation effect of order streams associated with the corresponding level of SP strategy. Each color of a cell encodes the
cumulative density of order streams that achieve a certain manipulation intensity and transaction risk. The closer dark blue is to the bottom
right, the better adapted order streams are able to preserve higher manipulation intensity with lower transaction risk.
simulation runs).3Each trading session lasts 5000 time steps,
and the generated order streams have lengths varying from 4
to 91. The ﬁrst execution stage is from time 200 to 1000, after
which the manipulation agent starts to spoof. At time 2000,
it begins to liquidate previously accumulated positions. The
underlying security has a fundamental mean r= 105. Based
on estimations of the ﬁnal fundamental value, the MM submits
a quote ladder with != 256 ,K= 8,N(128;10), and
quantityqN(5;2). We use 8,896 groups of order streams
for training (with a 80/20 train-validation split) and the rest
2,048 groups for testing.
We use a bi-directional Gated Recurrent Unit (GRU) RNN
[Cho et al. , 2014 ]with a hidden state size of 64, followed by
a linear layer for both Genc,Gdec, andDin the experiments.
Since order streams are of variable lengths, we pad them to
the maximum length for forward passes, and cut them back
to original lengths for loss calculations and evaluations. We
initialized the model parameters with the uniform distribution
between -0.08 and 0.08. We use batches of 64 order streams
to train the discriminator and the generator, and pick weight
of the self-regularization loss = 1based on the validation
performance.
4.2 Generating Adapted Manipulation Examples
We evaluate the adversarially adapted order streams from
three main aspects: (1) similarity to the MM quoting patterns,
(2) preservation of manipulation effect, and (3) effectiveness
3We keep valid simulations where the manipulator successfully
trades during the ﬁrst stage (so that there is an incentive to spoof),
and pushes prices to its desired direction by at least ten ticks.in evading the detection of an existing discriminator. Table 1
presents summary statistics of order streams associated with
their corresponding trading strategies (or generative models).
We discuss each aspect in detail below.
Comparing to MM We follow prior work [Liet al. , 2020 ]
in using price and quantity distributions to measure how well
the generated order streams resemble the target MM streams.
We further propose a domain-speciﬁc measure, the order im-
balance distribution , deﬁned as the ratio between the num-
bers of buy and sell orders submitted over a trading period
(whichever value is larger on the numerator). This captures
a trader’s imbalance in preference between long and short
positions. Fig. 4 presents comparisons of the respective dis-
tributions. We ﬁnd the adapted manipulation order streams
produce distributions similar to that of the MM, and are able
to overcome artifacts codiﬁed in the SP strategy (e.g., large-
quantity orders always at one tick behind the best quotes and
severe order imbalance to deceive the market). Speciﬁcally, we
ﬁnd that orders are gradually adapted to cover a wider range
of prices with relative small quantities, and order balance is
roughly maintained throughout the trading period.
Preserving Manipulation Effect We feed adapted order
streams back to the market simulator under the same set of
experimental controls, and measure their manipulation effect
by the manipulation intensity and transaction risk as deﬁned
in Sec. 3. Fig. 5 shows the two-dimensional cumulative
density over the 2,048 adapted outputs with respect to the two
proposed metrics. We ﬁnd that SP1can preserve a comparable
manipulation intensity under a reasonable transaction risk;
however, as the generator adapts in response to a more robust
Sell
Buy040801201602004080120160200Limit Sell OrderLimit Buy OrderTransacted SellTransacted Buy
Figure 6: Examples of adapted manipulation order streams. Dashed black lines represent the latest transaction prices, whereas dashed grey
lines the transaction prices if no manipulation exists.
discriminator, the adapted streams become to suffer a large
degradation in manipulation intensity and an increase in trans-
action risk (e.g., SP3has a similar performance to MM). This
weakened manipulation effect is further conﬁrmed in Table 1.
Evading the Detection Table 1 shows that a generator can
easily fool an existing detector with adversarially generated
order streams. By learning from a history of adapted or-
der streams, the discriminator is able to detect manipulation
streams from all previous levels, and in the meantime ensures
the training stability of the next-level generator.
Qualitative Evaluation Fig. 6 demonstrates examples of
the original and its corresponding adapted manipulation
order streams. We observe that the adapted streams become
qualitatively similar to the trading patterns of a MM, and
such simultaneous quoting behavior on both sides of the
market has indeed been suggested as a good strategy for high
frequency traders to mask their manipulative intent [Levens,
2015 ]. We note several other ﬁndings from the evolution of
adapted manipulation strategies. First, SP1remains to place
large orders close to the market best quote, whereas SP2and
SP3choose to either largely decrease the order quantity or
place large orders behind smaller ones to avoid being detected.
Second, SP2andSP3tend to submit orders at more aggressive
prices across market quotes, and this may cause unintended
transactions during the manipulation phase.
5 Conclusion
We employ an adversarial learning framework to model the
evolving game between a regulator and a manipulator, inwhich the regulator deploys algorithms to detect manipulation
and the manipulator masks actions to evade detection. Evasion
is represented by a generative model, trained by augmenting
manipulation order streams with examples of market making
activity traces. The intent is to produce adapted streams that
are hard to distinguish from a market maker’s behavior. We
visualize examples of adapted manipulation order streams, and
show they resemble quoting patterns of a market maker and
appear qualitatively different from the original manipulation
strategy we implemented in the simulator. This adaptation
evades detection, but only at the cost of compromising ef-
fectiveness in market manipulation. After a few iterations of
evolving and evading the detector, the strategy has sacriﬁced
almost all of its manipulation capability.
Our results reﬂect the speciﬁc modeling and simulation
choices adopted, and thus it remains to be seen whether a
more clever form of adaptation can evade detection while
retaining more effectiveness in manipulation. Whether or not
it is possible to ultimately craft successful adversarial attacks,
the generation and evasion process modeled here provides a
way to anticipate the evolution of evasive adversaries. Such
anticipation capacity provides a way to develop more robust
detection methods, for market manipulation as well as other
fraudulent behaviors.
Acknowledgments
This work was supported in part by the US National Science
Foundation IIS-1741190.
References
[Cho et al. , 2014 ]Kyunghyun Cho, Bart van Merri ¨enboer,
Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol-
ger Schwenk, and Yoshua Bengio. Learning phrase rep-
resentations using RNN encoder–decoder for statistical
machine translation. In Empirical Methods in Natural Lan-
guage Processing , pages 1724–1734, 2014.
[Goodfellow et al. , 2014 ]Ian Goodfellow, Jean Pouget-
Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher-
jil Ozair, Aaron Courville, and Yoshua Bengio. Generative
adversarial nets. In International Conference on Neural
Information Processing Systems , pages 2672–2680, 2014.
[Goodfellow et al. , 2015 ]Ian Goodfellow, Jonathon Shlens,
and Christian Szegedy. Explaining and harnessing adver-
sarial examples. In International Conference on Learning
Representations , 2015.
[Huang et al. , 2011 ]Ling Huang, Anthony D. Joseph, Blaine
Nelson, Benjamin I.P. Rubinstein, and J. D. Tygar. Ad-
versarial machine learning. In Proceedings of the 4th
ACM Workshop on Security and Artiﬁcial Intelligence , page
43–58, 2011.
[Kirilenko et al. , 2017 ]Andrei A. Kirilenko, Albert S. Kyle,
Mehrdad Samadi, and Tugkan Tuzun. The ﬂash crash:
High frequency trading in an electronic market. Journal of
Finance , 72:967–998, 2017.
[LeBaron, 2006 ]Blake LeBaron. Agent-based computational
ﬁnance. Handbook of Computational Economics , 2:1187–
1233, 2006.
[Levens, 2015 ]Tara E. Levens. Too fast, too frequent? High-
frequency trading and securities class actions. University
of Chicago Law Review , 82:1511–1558, 2015.
[Liet al. , 2020 ]Junyi Li, Xintong Wang, Yaoyang Lin,
Arunesh Sinha, and Michael P. Wellman. Generating real-
istic stock market order streams. In 34th AAAI Conference
on Artiﬁcial Intelligence , 2020.
[Lin, 2015 ]Tom C. W. Lin. The new market manipulation.
Emory Law Journal , 66:1253–1314, 2015.
[Logeswaran et al. , 2018 ]Lajanugen Logeswaran, Honglak
Lee, and Samy Bengio. Content preserving text generation
with attribute controls. In International Conference on
Neural Information Processing Systems , pages 5108–5118,
2018.
[Rundle, 2019 ]James Rundle. Nasdaq deploys AI to detect
stock-market abuse. Wall Street Journal , 2019.
[Schwartz and Peng, 2012 ]Robert A Schwartz and Lin Peng.
Market makers. Encyclopedia of Finance , 2012.
[Shrivastava et al. , 2017 ]Ashish Shrivastava, Tomas Pﬁster,
Oncel Tuzel, Josh Susskind, Wenda Wang, and Russell
Webb. Learning from simulated and unsupervised images
through adversarial training. In IEEE Conference on Com-
puter Vision and Pattern Recognition , pages 2242–2251,
2017.[Sinha et al. , 2018 ]Aman Sinha, Hongseok Namkoong, and
John Duchi. Certiﬁable distributional robustness with prin-
cipled adversarial training. In International Conference on
Learning Representations , 2018.
[Sutskever et al. , 2014 ]Ilya Sutskever, Oriol Vinyals, and
Quoc V . Le. Sequence to sequence learning with neural net-
works. In International Conference on Neural Information
Processing Systems , pages 3104–3112, 2014.
[Tzeng et al. , 2017 ]E. Tzeng, J. Hoffman, K. Saenko, and
T. Darrell. Adversarial discriminative domain adaptation.
InIEEE Conference on Computer Vision and Pattern Recog-
nition , pages 2962–2971, 2017.
[V olpi et al. , 2018 ]Riccardo V olpi, Hongseok Namkoong,
Ozan Sener, John Duchi, Vittorio Murino, and Silvio
Savarese. Generalizing to unseen domains via adversarial
data augmentation. In International Conference on Neural
Information Processing Systems , pages 5339–5349, 2018.
[Wah et al. , 2017 ]Elaine Wah, Mason Wright, and Michael P.
Wellman. Welfare effects of market making in continuous
double auctions. Journal of Artiﬁcial Intelligence Research ,
59:613–650, 2017.
[Wang and Wellman, 2017 ]Xintong Wang and Michael P.
Wellman. Spooﬁng the limit order book: An agent-based
model. In International Conference on Autonomous Agents
and Multiagent Systems , pages 651–659, 2017.
[Wang et al. , 2018 ]Xintong Wang, Yevgeniy V orobeychik,
and Michael P. Wellman. A cloaking mechanism to mitigate
market manipulation. In International Joint Conference on
Artiﬁcial Intelligence , pages 541–547, 2018.