Safe exploration in markov decision processes

Safe Exploration in Markov Decision Processes
Teodor Mihai Moldovan moldovan@cs.berkeley.edu
Pieter Abbeel pabbeel@cs.berkeley.edu
University of California at Berkeley, CA 94720-1758, USA
Abstract
In environments with uncertain dynamics ex-
ploration is necessary to learn how to per-
form well. Existing reinforcement learning
algorithms provide strong exploration guar-
antees, but they tend to rely on an ergod-
icity assumption. The essence of ergodicity
is that any state is eventually reachable from
any other state by following a suitable policy.
This assumption allows for exploration algo-
rithms that operate by simply favoring states
that have rarely been visited before. For
most physical systems this assumption is im-
practical as the systems would break before
any reasonable exploration has taken place,
i.e., most physical systems don't satisfy the
ergodicity assumption. In this paper we ad-
dress the need for safe exploration methods
in Markov decision processes. We rst pro-
pose a general formulation of safety through
ergodicity. We show that imposing safety by
restricting attention to the resulting set of
guaranteed safe policies is NP-hard. We then
present an ecient algorithm for guaranteed
safe, but potentially suboptimal, exploration.
At the core is an optimization formulation
in which the constraints restrict attention to
a subset of the guaranteed safe policies and
the objective favors exploration policies. Our
framework is compatible with the majority
of previously proposed exploration methods,
which rely on an exploration bonus. Our ex-
periments, which include a Martian terrain
exploration problem, show that our method
is able to explore better than classical explo-
ration methods.
Appearing in Proceedings of the 29thInternational Confer-
ence on Machine Learning , Edinburgh, Scotland, UK, 2012.
Copyright 2012 by the author(s)/owner(s).1. Introduction
When humans learn to control a system, they natu-
rally account for what we think of as safety. For exam-
ple, when a novice pilot learns how to
y an RC heli-
copter, they will slowly spin up the blades until the he-
licopter barely lifts o, then quickly put it back down.
They will repeat this a few times, slowly starting to
bring the helicopter a little bit o the ground. When
doing so they would try out the cyclic (roll and pitch)
and rudder (yaw) control, while|until they have be-
come more skilled|at all times staying low enough
that simply shutting it down would still have it land
safely. When a driver wants to become skilled at driv-
ing on snow, they might rst slowly drive the car to a
wide open space where they could start pushing their
limits. When we are skiing downhill, we are careful
about not going down a slope into a valley where there
is no lift to take us back up.
One would hope that exploration algorithms for phys-
ical systems would be able to account for safety and
have similar behavior naturally emerge. Unfortunately
most existing exploration algorithms completely ig-
nore safety issues. More precisely phrased, most exist-
ing algorithms have strong exploration guarantees, but
to achieve these guarantees they assume ergodicity of
the Markov decision process (MDP) in which the ex-
ploration takes place. An MDP is ergodic if any state
is reachable from any other state by following a suit-
able policy. This assumption does not hold true in the
exploration examples presented above as each of these
systems could break during (non-safe) exploration.
Our rst important contribution is a denition of
safety, which, at its core, requires restricting atten-
tion to policies that preserve ergodicity with some well
controlled probability. Imposing safety is, unfortu-
nately, NP-hard in general. Our second important
contribution is an approximation scheme leading to
guaranteed safe, but potentially sub-optimal, explo-
ration.1A third contribution is the consideration of
1Note that existing (unsafe) exploration algorithms arearXiv:1205.4810v3  [cs.LG]  6 Jul 2012
Safe Exploration in Markov Decision Processes
uncertainty in the dynamics model that is correlated
over states. While usually the assumption is that un-
certainty in dierent parameters is independent|as
this makes problem more tractable computationally|
being able to learn about state-action pairs before vis-
iting them is critical for safety.
Our experiments illustrate that our method indeed
achieves safe exploration, in contrast to plain explo-
ration methods. They also show that our algorithm
is almost as computationally ecient as planning in
a known MDP|but then, as every step leads to an
update in knowledge about the MDP, this computa-
tion is to be repeated after every step. Our approach
is able to safely explore grid worlds of size up to 50
100. Our method can make safe any type of explo-
ration that relies on exploration bonuses, which is the
case for most existing exploration algorithms, includ-
ing, for example, the methods proposed in (Brafman
& Tennenholtz, 2001; Kolter & Ng, 2009). In this ar-
ticle we do not focus on the exploration objective and
use existing ones.
Safe exploration has been the focus of a large number
of articles. (Gillula & Tomlin, 2011; Aswani & Bouf-
fard, 2012) propose safe exploration methods for linear
systems with bounded disturbances based on model
predictive control and reachability analysis. They de-
ne safety in terms of safe regions of the state space,
which, we will show, is not always appropriate in the
context of MDPs. The safe exploration for MDP meth-
ods proposed by (Geramifard et al., 2011; Hans et al.,
2008) gauge safety based on the best best estimate of
the transition measure but they ignore the level of un-
certainty in this estimate. As we will show, this is not
sucient to provably guarantee safety.
Provably ecient exploration is a recurring theme in
reinforcement learning (Strehl & Littman, 2005; Li
et al., 2008; Brafman & Tennenholtz, 2001; Kearns
& Singh, 2002; Kolter & Ng, 2009). Most methods,
however, tend to rely on the assumption of ergodicity
which rarely holds in interesting practical examples;
consequently, these methods are rarely applicable for
physical systems. The issue of provably guaranteed
safety, or risk aversion, under uncertainty in the MDP
parameters has also been studied in the reinforcement
literature. In (Nilim & El Ghaoui, 2005) they propose
a robust MDP control method assuming the transition
frequencies are drawn from an orthogonal convex set
by an adversary. Unfortunately, it seems impossible
to use their method to constrain some safety objec-
tive while optimizing a dierent exploration objective.
also sub-optimal, in that they are not guaranteed to com-
plete exploration in the minimal number of time steps.In (Delage & Mannor, 2007) they present a safe ex-
ploration algorithm for the special case of Gaussian
distributed ambiguity in the reward and state-action-
state transition probabilities, but their safety guaran-
tees are only accurate if the ambiguity in the transition
model is small.
2. Notation and Assumptions
Due to space constraints, we will not give a general in-
troduction to Markov decision processes (MDPs). For
an introduction to MDPs we refer the readers to (Sut-
ton & Barto, 1998; Bertsekas & Tsitsiklis, 1996).
We use capital letters to denote random variables; for
example, the total reward is: V:=P1
t=0RSt;At. We
represent the policies and the initial state distributions
by probability measures. Usually the measure will
correspond to a policy and the measure s:=(s),
which puts measure only in state s, will correspond
to starting in state s. With this notation, the usual
value recursion, assuming a known transition measure,
p, reads:
Ep
s;[V] =X
a;s0s;a
E[R]s;a+ps;a;s0Ep
s0;[V]
:
We specify the transition measure as a superscript of
the expectation operator rather than a subscript for
typographical convenience; in this case, and in gen-
eral, the positioning of indexes as subscripts or su-
perscripts adds no extra signicance. We will let the
transition measure psometimes sum to less than one,
that isP
s0ps;a;s01. The missing mass is implicitly
assigned to transitioning to an absorbing \end" state,
which, for example, allows us to model
discounting
by simply using
pas a transition measure.
We model ambiguous dynamics in a Bayesian way, al-
lowing the transition measure to also be a random
variable. When this is the case, we will use Pto de-
note the, now random, transition measure. The belief ,
which we will denote by , is our Bayesian probabil-
ity measure over possible dynamics, governing Pand
R. Therefore, the expected return under the belief and
policy, starting from state s, isEEP
s;[V]. We allow
beliefs under which transition measures and rewards
are arbitrarily correlated. In fact, such correlations
are usually necessary to allow for safe exploration. For
compactness we will often use lower case letters to de-
note the expectation of their upper case counterparts.
Specically, we will use the notations p:=E[P] and
r:=E[R] throughout.
Safe Exploration in Markov Decision Processes
3. Problem formulation
3.1. Exploration Objective
Exploration methods, as those proposed in (Brafman
& Tennenholtz, 2001; Kolter & Ng, 2009), operate
by nding optimal policies in constructed MDPs with
exploration bonuses. The R- max algorithm, for ex-
ample, constructs an MDP based on the discounted
expected transition measure and rewards under the
belief, and adds a deterministic exploration bonus
equal to the maximum possible reward in the MDP,

s;a=rmax, to any transitions that are not suciently
well known. Our method allows adding safety con-
straints to any such exploration methods. Henceforth,
we will restrict attention to such exploration methods,
which can be formalized as optimization problems of
the form:
maximize oE
p
s0;o1X
t=0
rSt;At+
St;At
:(1)
3.2. Safety Constraint
The issue of safety is closely related to ergodicity . Al-
most all proposed exploration techniques presume er-
godicity; authors present it as a harmless technical as-
sumption but it rarely holds in interesting practical
problems. Whenever this happens, their ecient ex-
ploration guarantees cease to hold, often leading to
very inecient policies. Informally, an environment
is ergodic if any mistake can be forgiven eventually.
More specically, a belief over MDPs is ergodic if and
only if any state is reachable from any other state via
some policy or, equivalently, if and only if:
8s;s0;9rsuch thatEEP
s;r[Bs0] = 1; (2)
whereBs0is an indicator random variable of the event
that the system reaches state s0at least once: Bs0=
1f9t<1such thatSt=s0g= min (1;P
t1St=s0).
Unfortunately, many environments are not ergodic.
For example, our robot helicopter learning to
y can-
not recover on its own after crashing. Ensuring almost
sure ergodicity is too restrictive for most environments
as, typically, there always is a very small, but non-
zero, chance of encountering that particularly unlucky
sequence of events that breaks the system. Our idea
is to restrict the space of eligible policies to those that
preserve ergodicity with some user-specied probabil-
ity,, called the safety level . We name these policies
-safe. Safe exploration now amounts to choosing the
best exploration policy from this set of safe policies.
Informally, if we stopped a -safe policy oat any time
T, we would be able to return from that point to theB SF Ea .2
a .8b
c .9a
bc .1
Figure 1. Starting from state S, the policy (aababab. . . ) is
safe at a safety level of :8. However, the policy (acccc. . . )
is not safe since it will end up in the sink state E with
probability 1. State-action Sa and state B can neither be
considered safe nor unsafe, since both policies use them.
home state s0with probability by deploying a return
policyr. Executing only -safe policies in the case of
a robot helicopter learning to
y will guarantee that
the helicopter is able to land safely with probability
whenever we decide to end the experiment. In this
example,Tis the time when the helicopter is recalled
(perhaps because fuel is running low), so we will call
Ttherecall time . Formally, an outbound policy ois
-safe with respect to a home state s0and a stopping
timeTif and only if:
9rsuch thatEEP
s0;o
EP
ST;r[Bs0]
:(3)
Note that, based on Equation (2), any policy is -safe
for anyif the MDP is ergodic with probability one
under the belief. For convenience we will assume that
the recall time, T, is exponentially distributed with
parameter 1
, but our method also applies when
the recall time equals some deterministic horizon. Un-
fortunately, expressing the set of -safe policies is NP-
hard in general, as implied by the following theorem
proven in the appendix.
Theorem 1. In general, it is NP-hard to decide
whether there exist -safe policies with respect to a
home state, s0, and a stopping time, T, for some be-
lief,.
3.3. Safety Counter-Examples
We conclude this section with counter-examples to
three other, perhaps at rst sight more intuitive, def-
initions of safety. First, we could have tried to dene
safety in terms of sets of safe states or state-actions.
That is, we might think that making the non-safe
states and actions unavailable to the planner (or sim-
ply inaccessible) is enough to guarantee safety. Fig-
ure 1 shows an MDP where the same state-action is
used both by a safe and by an unsafe policy. The
idea behind this counter-example is that safety de-
pends not only on the states visited, but also on the
number of visits, thus, on the policy. This shows that
safety should be dened in terms of safe policies, not
in terms of safe states or state-actions.
Safe Exploration in Markov Decision Processes
L B US
U B LS
ab
abaa
ab
b
aaa
Figure 2. Under out belief the two MDPs above both have
probability :5. It is intuitively unsafe to go from the start
state S to B since we wouldn't know whether the way back
is via U or L, even though we know for sure that a return
policy exists.
A CB
A CB
A CBaaaaa
aa .5a .5
a .5a .5a
Figure 3. The two MDPs on the left both have probability
:5. Under this belief, starting from state A, policy (aaa. . . )
is unsafe. However, under the mean transition measure,
represented by the MDP on the right, the policy is safe.
Second, we might think that it is perhaps enough to
ensure that there exists a return policy for each poten-
tial sample MDP from the belief, but not impose that
it be the same for all samples. That is, we might think
that condition 3 is too strong and, instead, it would
be enough to have:
E1f9r:EP
s0;oEP
ST;r[Bs0] = 1g:
Figure 2 shows an MDP where this condition holds,
yet all policies are naturally unsafe.
Third, we might think that it is sucient to simply use
the expected transition measure when dening safety,
as in the equation below. Figure 3 shows that this is
not the case; the expected transition measure is not a
sucient statistic for safety.
9rsuch thatEp
s0;oh
Ep
ST;r[Bs0]i
:
4. Guaranteed Safe, Potentially
Sub-optimal Exploration
Although imposing the safety constraint in Equa-
tion (3) is NP-hard, as shown in Theorem 1, we can
eciently constrain a lower bound on the safety objec-
tive, so the safety condition is still provably satised.
Doing so could lead to sub-optimal exploration since
the set of policies we are optimizing over has shrunk.
However, we should keep in mind that the exploration
objectives represent approximate solutions to other
NP-hard problems, so optimality has already been for-
feited in existing (non-safe) approaches to start outAlgorithm 1 Safe exploration algorithm
Require: prior belief , discount
, safety level .
Require: function: belief!exploration bonus
M;N new MDP objects
repeat
s0;' current state and observations
update belief with information '

s;a () (exploration bonus based on )

s;a P
s0E[min(0;Ps;a;s0E[Ps;a;s0])]
M:transition measure  E[P](11s=s0)
M:reward function 1s=s0+ (11s=s0)
s;a
r;v M:solve()
N:transition measure
E[P]
N:reward function E[Rs;a] +
s;a
N:constraint reward func.  (1
)vs+

s;a
N:constraint lower bound
o;v;v N:solve under constraint()
q
s;a (1
)vs+

s;a+P
s0ps;a;s0v
s0
a argmaxfo
s0;a>0gq
s0;a(de-randomize policy)
take action ain environment
until= 0, so there is nothing left to explore
with. Algorithm 1 summarizes the procedure and the
experiments presented in the next section show that,
in practice, when the ergodicity assumptions are vi-
olated, safe exploration is much more ecient than
plain exploration.
Putting together the exploration objective dened in
Equation (1) and the safety objective dened in Equa-
tion (3) allows us to formulate safe exploration at level
as a constrained optimization problem:
maximize o;rE
p
s0;oX
t
rSt;At+
St;At
such that: EEP
s0;o
EP
ST;r[Bs0]
:
The exploration objective is already conveniently for-
mulated as the expected reward in an MDP with tran-
sition measure
p, so we will not modify it. On the
other hand, the safety constraint is dicult to deal
with as is. Ideally, we would like the safety constraint
to also equal some expected reward in an MDP. We
will see that, in fact, it takes two MDPs to express the
safety constraint.
First, we express the inner term, EP
ST;r[Bs0], as the
expected reward in an MDP. We can replicate the be-
haviour of Bs0, that is counting only the rst time
states0is reached, by using a new transition measure,
P(11s=s0) under which, once s0is reached, any
further actions lead immediately to the implicit \end"
Safe Exploration in Markov Decision Processes
state. Formally, we express this by the identity:
EP
ST;r[Bs0] =EP(11s=s0)
ST;r1X
t=01St=s0:
We now focus on the outer term, EP
s0;o
EP
ST;r[Bs0]
.
Since the recall time, T, is exponentially distributed
with parameter 1
, we can view STas the nal state
in a
-discounted MDP starting at state s0, following
policyo. In this MDP, the inner term plays the role of
a terminal reward. To put the problem in a standard
form, we convert this terminal reward to a step-wise
reward by multiplying it by 1
.
EP
s0;o
EP
ST;r[Bs0]
=E
P
s0;o1X
t=0(1
)
EP
St;r[Bs0]
:
At this point, we have expressed the safety constraint
in the MDP formalism, but the transition measures of
these MDPs, P(11s=s0) and
P, are still random.
If we could replace these random transition measures
with their expectations under the belief that would
signicantly simplify the safety constraint. It turns out
we can do this, at the expense of making the constraint
more stringent. Our tool for doing so is Theorem 2
presented below, but proven in the appendix. It shows
that we can replace a belief over MDPs by a single
MDP with the expected transition measure, featuring
an appropriate reward correction such that, under any
policy, the value of this MDP is a lower bound on the
expected value under the belief.
Theorem 2. Letbe a belief such that for any policy,
, and any starting state, s, the total expected reward
in any MDP drawn from the belief is between 0and1;
i.e.0EP
s;[V]1; -almost surely. Then the fol-
lowing bound holds for any policy, , and any starting
state,s:
EEP
s;1X
t=0RSt;AtEp
s;1X
t=0
E[RSt;At] +
St;At
where
s;a:=X
s0E[min(0;Ps;a;s0E[Ps;a;s0])]:
We rst apply Theorem 2 to the outer term, yielding
the following bound:
EP
s0;o
EP
ST;r[Bs0]
=E
P
s0;o1X
t=0(1
)
EP
St;r[Bs0]
E
p
s0;o1X
t=0
(1
)EEP
St;r[Bs0] +

St;At
:We, then, apply it again to the inner term:
EEP
s;r[Bs0] =EP(11s=s0)
s;r1X
t=01St=s0 (4)
Ep(11s=s0)
s;r1X
t=0
1St=s0+ (11St=s0)
St;At
:
Combining the last two results allows us to replace
the NP-hard safety constraint with a stricter, but now
tractable, constraint. The resulting optimization prob-
lem corresponds to the guaranteed safe, but poten-
tially sub-optimal exploration problem:
maximize o;rE
p
s0;oX
t
rSt;At+
St;At
(5)
s.t.:E
p
s0;o1X
t=0
(1
)vSt+

St;At
and
vs=Ep(11s=s0)
s;r1X
t=0
1St=s0+ (11St=s0)
St;At
:
The termvsrepresents our lower bound for the inner
term per Equation (4), and is simply the value function
of the MDP corresponding to the inner term; i.e. the
MDP with transition measure p(11s=s0) and reward
function 1 s=s0+(11s=s0)
s;a, under policy r. Since
the return policy, r, does not appear anywhere else,
we can split the safe exploration problem we obtained
in Equation (5) into two steps:
Step one: nd the optimal return policy
r, and
corresponding value function v
s, by solving the stan-
dard MDP problem below:
Ep(11s=s0)
s;r1X
t=0
1St=s0+ (11St=s0)
St;At
:
Step two: nd the optimal exploration policy
oun-
der the strict safety constraint, by solving the con-
strained MDP problem below:
maximize oE
p
s0;oX
t
rSt;At+
St;At
s.t.:E
p
s0;o1X
t=0
(1
)v
St+

St;At
:
The rst step amounts to solving a standard MDP
problem while the second step amounts to solving a
constrained MDP problem. As shown by (Altman,
1999), both can be solved eciently either by linear
programming, or by value-iteration. In our exper-
iments we used the LP formulation with the state-
action occupation measure as optimization variable.
Safe Exploration in Markov Decision Processes
(a) Based on the available infor-
mation after the rst step, moving
South-West is unsafe.
(b) The safe explorer successfully
uncovers all of the map by avoiding
irreversible actions.
(c) The adapted R- max explorer
gets stuck before observing the en-
tire map.
(d) Moving South-East is currently
considered unsafe since, based on
the available information, there is
no return path.
(e) After seeing more of the map,
our safe explorer decides that the
transition initially deemed unsafe
is, in fact, safe.
(f) The adapted R- max explorer
acts greedily. Even though its sec-
ond action, is, in fact safe, its third
action is not, so it gets stuck.
(g) Moving East is safe with prob-
ability:8 since the return path is
blocked for only one out of ve pos-
sible heights of the unknown square
South of the start position.
(h) Safe exploration with = 1:0
does not risk moving East event
though the exploration bonuses are
much higher there.
(i) Safe exploration with :6
does move East. Note that, in this
case, our method overestimates the
probability of failure by a factor of
two and, thus, acts conservatively.
Figure 4. Exploration experiments in simple grid worlds. See text for full details. Square sizes are proportional to
corresponding state heights between 1 and 5. The large, violet squares have a height of 5, while the small, blue squares
have a height of 1. Gray spaces represent states that have not yet been observed. Each row corresponds to the same grid
world. The rst column shows the belief after the rst exploration step, while the second and third columns show the
entire trajectory followed by dierent explorers.
Solutions to the constrained MDP problem will usu-
ally be stochastic policies, and, in our experiments, we
found that following them sometimes leads to random
walks which explore ineciently. We addressed the
issue by de-randomizing the exploration policies in fa-
vor of safety. That is, whenever the stochastic policy
proposes multiple actions with non-zero measure, we
choose the one among them that optimizes the safety
objective.
5. Experiments
5.1. Grid World
Our rst experiment models a terrain exploration
problem where the agent has limited sensing capabil-
ities. We consider a simple rectangular grid world,
where every state has a height Hs. From our Bayesian
standpoint these heights are independent, uniformly
distributed categorical random variables on the setf1;2;3;4;5g. At any time the agent can attempt to
move to any immediately neighboring state. Such
move will succeed with probability one if the height of
the destination state is no more than one level above
the current state; otherwise, the agent remains in the
current state with probability one. In other words, the
agent can always go down clis, but is unable to climb
up if they are too steep. Whenever the agent enters a
new state it can see the exact heights of all immedi-
ately surrounding states. We present this grid world
experiment to build intuition and to provide an easily
reproducible result. Figure 4 shows a number of ex-
amples where our exploration method results in intu-
itively safe behavior, while plain exploration methods
lead to clearly unsafe, suboptimal behavior.
Our exploration scheme, which we call adapted R- max,
is a modied version of R-max exploration (Brafman
& Tennenholtz, 2001), where the exploration bonus of
moving between two states is now proportional to the
Safe Exploration in Markov Decision Processes
(a) Safe exploration with =
:98 leads to a model entropy
reduction of 7680.
(b) Safe exploration with =
:90 leads to a model entropy
reduction of 12660.
(c) Safe exploration with =
:70 leads to a model entropy
reduction of 35975.
(d) Regular (unsafe, = 0)
exploration leads to a model
entropy reduction of 3214.
Figure 5. Simulated safe exploration on a 2km by 1km area of Mars at -30.6 degree latitude and 202.2 degrees longitude,
for 15000 time steps, at dierent safety levels. See text for full details. The color saturation is inversely proportional to
the standard deviation of the height map under the posterior belief. Full coloration represents a standard deviation of
1cm or less. We report the dierence between the entropies of the height model under the prior and the posterior beliefs
as a measure of performance. Images: NASA/JPL/University of Arizona.
number of neighboring unknown states that would be
uncovered as a result of the move, to account for the
remote observation model. The safety costs for this
exploration setup, as prescribed by Theorem 2 are:

s;a=2E[Ps;a](1E[Ps;a]) =2Var[Ps;a]
wherePs;a:= 1Hs+aHs+1is the probability that at-
tempted move asucceeds in state sand the belief de-
scribes the distribution of the heights of unseen states.
In practice we found that this correction is a factor
of two larger than would be sucient to give a tight
safety bound.
A somewhat counter intuitive result is that adding
safety constraints to the exploration objective will, in
fact, improve the fraction of squares explored in ran-
domly generated grid worlds. The reason why plain
exploration performs so poorly is that the ergodicity
assumptions are violated, so eciency guarantees no
longer hold. Figure 6 in the appendix summarizes our
exploration performance results.
5.2. Martian Terrain
For our second experiment, we model the problem of
autonomously exploring the surface of Mars by a rover
such as the Mars Science Laboratory (MSL) (Lock-
wood, 2006). The MSL is designed to be remote con-
trolled from Earth but communication suers a latency
of 16.6 minutes. At top speed, it could traverse about
20m before receiving new instructions, so it needs to
be able to navigate autonomously. In the future, when
such rovers become faster and cheaper to deploy, the
ability to plan their paths autonomously will become
even more important. The MSL is designed to a static
stability of 45 degrees, but would only be able to
climb slopes up to 5 degrees without slipping (MSL,
2007). Digital terrain models for parts of the surface of
Mars are available from the High Resolution ImagingScience Experiment (HiRISE) at a scale of 1.00 me-
ter/pixel and accurate to about a quarter of a meter.
The MSL would be able to obtain much more accurate
terrain models locally by stereo vision.
The state-action space of our model MDP is the same
as in the previous experiment, with each state corre-
sponding to a square area of 20 by 20 meters on the
surface. We allow only transitions at slopes between
-45 and 5 degrees. The heights, Hs, are now assumed
to be independent Gaussian random variables. Un-
der the prior belief, informed by the HiRISE data, the
expected heights and their variances are:
E[H] =D20[gh] and
Var[H] =D20[g(hgh)2] +v0
wherehare the HiRISE measurements, gis a Gaus-
sian lter with = 5 meters, \" represents image
convolution, D20is the sub-sampling operator and
v0= 24m2is our estimate of the variance of HiRISE
measurements. We model remote sensing by assum-
ing that the MSL can obtain Gaussian noisy measure-
ments of the height at a distance daway with variance
v(d) = 106(d+ 1m)2.
To account for this remote sensing model we use a
rst order approximation of the entropy of Has an
exploration bonus:

s;a=X
s0Var[Hs0]=v(ds;s0):
Figure 5 shows our simulated exploration results for a
2km by 1km area at 30:6 degrees latitude and 202 :2
degrees longitude (PSP, 2008). Safe exploration at
level 1:0 is no longer possible, but, even at a con-
servative safety level of :98, our method covers more
ground than the regular (unsafe) exploration method
which promptly get stuck in a crater. Imposing the
safety constraint naively, with respect to the expected
Safe Exploration in Markov Decision Processes
Table 1. Per-step planning times for the 50 100 grid world
used in the Mars exploration experiments, with
=:999.
Problem setting Planning time (s)
Safe exploration at .98 5 :861:47
Safe exploration at .90 10 :947:14
Safe exploration at .70 4 :573:19
Naive constraint at .98 2 :550:42
Regular (unsafe) exploration 1 :620:26
transition measure, as argued against at the end of
Section 3.3, performs as poorly as unsafe exploration
even if the constraint is set at :98.
5.3. Computation Time
We implemented our algorithm in Python 2.7.2.7,
using Numpy 1.5.1 for dense array manipulation,
SciPy 0.9.0 for sparse matrix manipulation and Mosek
6.0.0.119 for linear programming. The discount factor
was set to :99 for the grid world experiment and to
:999 for Mars exploration. In the latter experiment
we also restricted precision to 106to avoid numeri-
cal instabilities in the LP solver. Table 1 summarizes
planning times for our Mars exploration experiments.
6. Discussion
In addition to the safety formulation we discussed in
Section 3.2, out framework also supports a number of
other safety criteria that we did not discuss due to
space constraints:
Stricter ergodicity ensuring that return is possible
within some horizon, H, not just eventually, with
probability .
Ensuring that the probability of leaving some pre-
dened safe set of state-actions is lower than 1 .
Ensuring that the expected total reward under the
belief is higher than .
Additionally, any number and combination of these
constraints at dierent -levels can be imposed simul-
taneously.
Acknowledgements
This material is based upon work supported in part
by NSF under award IIS-0931463, by ARO under the
MAST program, by a Sloan Fellowship, by a gift from
Intel, by the U. S. Army Research Laboratory and
the U. S. Army Research Oce under contract/grant
number W911NF-11-1-0391.References
MSL Landing Site Selection. User's Guide to
Engineering Constraints, 2007. URL http:
//marsoweb.nas.nasa.gov/landingsites/msl/
memoranda/MSL_Eng_User%_Guide_v4.5.1.pdf .
Stratigraphy of Potential Crater Hydrothermal System,
2008. URL http://hirise.lpl.arizona.edu/dtm/dtm.
php?ID=PSP_010228_1490 .
Altman, Eitan. Constrained Markov Decision Processes .
Chapman and Hall, 1999.
Aswani, Anil and Bouard, Patrick. Extensions of
Learning-Based Model Predictive Control for Real-Time
Application to a Quadrotor Helicopter. In Proc. Amer-
ican Control Conference (ACC) (to appear) , 2012.
Bertsekas, Dimitri P. and Tsitsiklis, John N. Neuro-
Dynamic Programming . Athena Scientic, October
1996.
Blondel, Vincent D. and Tsitsiklis, John N. A survey of
computational complexity results in systems and con-
trol. Automatica , 36(9):1249{1274, September 2000.
Brafman, Ronen I. and Tennenholtz, Moshe. R-MAX - A
General Polynomial Time Algorithm for Near-Optimal
Reinforcement Learning. In Journal of Machine Learn-
ing Research , volume 3, pp. 213{231, 2001.
Delage, Erick and Mannor, Shie. Percentile optimization
in uncertain Markov decision processes with application
to ecient exploration. ICML; Vol. 227 , pp. 225, 2007.
Geramifard, A, Redding, J, Roy, N, and How, J P. UAV
Cooperative Control with Stochastic Risk Models. In
Proceedings of the American Control Conference (ACC) ,
San Francisco, CA, 2011.
Gillula, Jeremy H. and Tomlin, Claire J. Guaranteed safe
online learning of a bounded system. In 2011 IEEE/RSJ
International Conference on Intelligent Robots and Sys-
tems, pp. 2979{2984. IEEE, September 2011.
Hans, A, Schneega, D, Sch afer, AM, and Udluft, S.
Safe exploration for reinforcement learning. In ESANN
2008, 16th European Symposium on Articial Neural
Networks , 2008.
Kearns, Michael and Singh, Satinder. Near-Optimal Re-
inforcement Learning in Polynomial Time. Machine
Learning , 49(2):209{232, November 2002.
Kolter, J. Zico and Ng, Andrew Y. Near-Bayesian explo-
ration in polynomial time. In Proceedings of the 26th
Annual International Conference on Machine Learning
- ICML '09 , pp. 1{8, New York, New York, USA, 2009.
ACM Press.
Li, Lihong, Littman, Michael L., and Walsh, Thomas J.
Knows what it knows: a framework for self-aware learn-
ing. In Proceedings of the 25th international conference
on Machine learning , pp. 568{575, 2008.
Lockwood, Mary Kae. Introduction: Mars Science Labo-
ratory: The Next Generation of Mars Landers. Journal
of Spacecraft and Rockets , 43(2), 2006.
Nilim, Arnab and El Ghaoui, Laurent. Robust Control
of Markov Decision Processes with Uncertain Transition
Matrices. Operations Research , 53(5):780{798, 2005.
Strehl, Alexander L. and Littman, Michael L. A theo-
retical analysis of Model-Based Interval Estimation. In
Proceedings of the 22nd international conference on Ma-
chine learning - ICML '05 , pp. 856{863, New York, New
York, USA, August 2005. ACM Press.
Sutton, Richard S. and Barto, Andrew G. Reinforcement
learning: an introduction . MIT Press, 1998.
Safe Exploration in Markov Decision Processes
X1
Y1
Z1X2
Y2
Z2. . .
. . .
. . .Xn
Yn
ZnD S
Figure 7. MDP reduction of the 3SAT problem.
Appendix
Proof of Theorem 1.
Proof. We will prove the theorem by reducing the sat-
isability problem in conjunctive normal form with
three variables (3SAT) to the problem of deciding
whether there exists a p-safe policy for a belief that
we will construct. The 3SAT problem amounts to de-
ciding whether there exists an assignment to boolean
variablesfUkgsuch that the following expression is
true:
(X1_Y1_Z1)^^ (Xn_Yn_Zn)
where each of the variables Xi;Yi;Ziequals one of the
variables infUkg, possibly negated.
We start by constructing an MDP to represent this
problem as shown in Figure 7. In addition to actions
corresponding to the outgoing arrows, the agent also
has the option of remaining in the same state. A tran-
sition from some state to another state will succeed if
and only if the boolean variable corresponding to the
origin state is true. The boolean variable associated
to statesSandDare always true. Our belief is the
uniform distribution over truth values of the boolean
variablesfUkg.
Now consider the following simple policy: from Dgo
toSand then stay in S. For any recall time T > 0,
the recall event will nd the agent in state S, so the
policy isp-safe for any p >0 if and only if the belief
assigns a non-zero measure to MDPs in which state D
is accessible form state S, so if any only if there exists
at least one boolean assignment for the fUkgsuch that
stateDis accessible from S. It is easy to see that, this
is the case if and only if the 3SAT formula is satised,
and this observation completes the reduction.
This result should come as no surprise since similar
optimization problems have been shown to be NP-hard
in the context of Partially Observable Markov DecisionProcesses (Blondel & Tsitsiklis, 2000).
Proof of Theorem 2.
Proof. The result is an immediate consequence of the
following Lemma.
Lemma 3. Given a belief and a policy , there
exists a policy dependent reward correction, ;, de-
ned below, such that the MDP with transition mea-
surep:=EPand rewards r+;, wherer:=ER,
has the same expected total return as the belief for any
initial distribution. Formally:
8 EEP
;1X
t=0RSt;At=Ep
;1X
t=0
rs;a+;
s;a
;
s;a:=X
s0E[(Ps;a;s0E[Ps;a;s0])Es0;;P[V]]:
Proof. The Markov property under belief reads:
EEP
s;[V] =X
as;aE[Rs;a]+
+X
as;aX
s0E[Ps;a;s0EP
s0;[V]]:
The Markov property assuming expected transition
frequencies and expected rewards with safety penalty
is:
Ep
s;[V] =X
as;a(rs;a+;
s;a)+
+X
as;aX
s0ps;a;s0Ep
s0;[V]]:
Now let  s:=EEP
s;[V]Ep
s;[V]. By subtracting
the rst two equations we get that:
s=X
as;aX
s0ps;a;s0s0:
We can see that  satises the same equation as the
value function in an MDP with transition measure p
and zero rewards. Since the value function in such
an MDP is uniquely dened and identically zero, we
conclude  s= 0.
Safe Exploration in Markov Decision Processes
102|.2102|.4102|.6202|.2202|.4202|.6302|.2302|.4302|.6402|.2402|.4402|.600.20.40.60.81
Grid world size |Mean fraction of inaccessible squares (walls)Fraction of grid world uncoveredSafe version of adapted
R-max explorationAdapted R- max
explorationOriginal R- max or
Near-Bayesian exploration
Figure 6. Exploration eciency comparison. We are showing the median, the upper and the lower quartiles of the frac-
tion of the grid world that was uncovered by dierent explorers in randomly generated grid worlds. The \amount" of
non-ergodicity is controlled by randomly making a fraction of the squares inaccessible (walls). We ran 1000, 500, 100,20
experiments for grids of sizes 102, 202, 302and 402respectively. We are comparing against our own adapted R- max ex-
ploration objective, the original R- max objective (Brafman & Tennenholtz, 2001) and the Near-Bayesian exploration
objective (Kolter & Ng, 2009). The last two behave identically in our grid world environment, since, once a state is
visited, all transitions out of that state are precisely revealed.


Formal Verification of Distributed Aircraft Controllers

Formal Veriﬁcation of Distributed Aircraft Controllers
Sarah M. Loos
Computer Science Dept.
Carnegie Mellon University
Pittsburgh, PA, USA
sloos@cs.cmu.eduDavid Renshaw
Computer Science Dept.
Carnegie Mellon University
Pittsburgh, PA, USA
dwrensha@cs.cmu.eduAndré Platzer
Computer Science Dept.
Carnegie Mellon University
Pittsburgh, PA, USA
aplatzer@cs.cmu.edu
ABSTRACT
As airspace becomes ever more crowded, air tra c management
must reduce both space and time between aircraft to increase through-
put, making on-board collision avoidance systems ever more im-
portant. These safety-critical systems must be extremely reliable,
and as such, many resources are invested into ensuring that the pro-
tocols they implement are accurate. Still, it is challenging to guar-
antee that such a controller works properly under every circum-
stance. In tough scenarios where a large number of aircraft must ex-
ecute a collision avoidance maneuver, a human pilot under stress is
not necessarily able to understand the complexity of the distributed
system and may not take the right course, especially if actions must
be taken quickly. We consider a class of distributed collision avoid-
ance controllers designed to work even in environments with arbi-
trarily many aircraft or UA Vs. We prove that the controllers never
allow the aircraft to get too close to one another, even when new
planes approach an in-progress avoidance maneuver that the new
plane may not be aware of. Because these safety guarantees always
hold, the aircraft are protected against unexpected emergent behav-
ior which simulation and testing may miss. This is an important
step in formally veriﬁed, ﬂyable, and distributed air tra c control.
Categories and Subject Descriptors
F.3.1 [ Logics and Meanings of Programs ]: Specifying and Veri-
fying and Reasoning about Programs
Keywords
formal veriﬁcation; distributed aircraft controllers
1. INTRODUCTION
Veriﬁcation of air tra c control is particularly challenging be-
cause it lies in the intersection of many ﬁelds which already give
This material is based upon work supported by the National Sci-
ence Foundation under NSF CAREER Award CNS-1054246, NSF
EXPEDITION CNS-0926181, grant no. CNS-0931985, and grant
no. CNS-1035800. Sarah M. Loos was supported by a DOE Com-
putational Science Graduate Fellowship.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for proﬁt or commercial advantage and that copies
bear this notice and the full citation on the ﬁrst page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior speciﬁc
permission and/or a fee.
HSCC’13, April 8–11, 2013, Philadelphia, Pennsylvania, USA.
Copyright 2013 ACM 978-1-4503-1567-8/13/04 ...$10.00.tough veriﬁcation problems when examined independently. It is a
distributed system, with a large number of aircraft interacting over
an unbounded time horizon. Each aircraft has nonlinear continuous
dynamics combined with complex discrete controllers. And ﬁnally,
every protocol must be ﬂyable (i.e. not cause the aircraft to enter a
stall, bank too sharply, or require it to turn on sharp corners).
In this paper, we investigate the safety of collision-avoidance
controllers for aircraft systems. We want to prove safety not just
for a single aircraft or a pair of aircraft, but for all aircraft oper-
ating simultaneously in the sky. Because this system is composed
of multiple independent computational agents that interact with the
physical world, it is called a distributed hybrid system . It is this
combination of continuous ﬂight dynamics, discrete ﬂight control
decisions, and distributed communication that causes veriﬁcation
of aircraft control protocols to be extremely challenging.
Aircraft control systems are safety-critical, so they must be de-
signed with a high assurance of correctness. When the costs of
failure are high, system designers must be able to guarantee ahead
of time that their systems work as intended. Many methods, such
as testing and simulation, are used in combination to improve reli-
ability. While testing and simulation may reveal software bugs and
increase safety assurance, they are not able to prove safety guaran-
tees over the continuous and inﬁnite state-spaces characteristic of
hybrid systems like ﬂight control, where the aircraft move continu-
ously through space and time. The complexity of curved ﬂight dy-
namics has been di cult for many analysis techniques [1–8], which
often resort to unﬂyable approximations of ﬂight trajectories that
require aircraft to turn on corners. However, the formal veriﬁcation
techniques described in this paper are able to provide guarantees for
ﬂyable maneuvers over the entirety of this continuous state-space
and therefore over all evolutions of all aircraft movement.
These strong guarantees are especially important in a distributed
system with a large number of interacting participants. As in [1, 4,
7, 9, 10], many previous approaches to aircraft control have looked
into a relatively small number of agents. But with thousands of
aircraft ﬂying through commercial airspace daily, this system is
already far too complex for humans to predict every scenario by
looking at interactions of only a few aircraft. And this challenge
increases when we examine controllers for Unmanned Aerial Vehi-
cles (UA Vs), which are becoming increasingly autonomous and ﬂy
even closer together, with less direct supervision by humans. As a
result, we must provide a good argument for why a controller will
always take the right action, even in extremely crowded airspace.
In this paper we specify and verify two control policies for pla-
nar aircraft avoidance maneuvers using automated theorem prover
KeYmaeraD to produce a proof of safety for each of them. We de-
sign these policies such that all aircraft adhere to a simple and easy-
to-implement separation principle: associated with each aircraft is
a disc, within which the aircraft must remain. In this way, the prob-
lem reduces to proving that i) su cient separation is maintained
between pairs of discs, and ii) individual aircraft always remain in-
side their associated disc. We model 2D ﬂight dynamics since they
are the relevant dynamics for planar maneuvers, but investigating
3D maneuvers and dynamics may make interesting future work.
The complexities which arise from the curved ﬂight trajectories
of an arbitrary number of aircraft interacting in a distributed man-
ner, along with the tight coupling of discrete control and continuous
dynamics presently make KeYmaeraD the only veriﬁcation tool ca-
pable of proving safety for this system. Our contributions are:
We provide the ﬁrst formally veriﬁed distributed system of
aircraft with curved ﬂight dynamics.
Our controller requires only ﬂyable aircraft trajectories with
no corners or instantaneous changes of ground speed.
We prove our controller is safe for an arbitrarily large num-
ber of aircraft . This guarantee is necessary for high-tra c
applications such as crowded commercial airspace, unmanned
aerial vehicle maneuvers, and robotic swarms.
Other aircraft may enter an avoidance maneuver already in
progress and safety for all aircraft is guaranteed still.
We use Arithmetic coding to reduce proof complexity and
branching.
We prove that even when the interactions of many aircraft
cause unexpected emergent behaviors , all resulting control
choices are still safe.
We present hierarchical and compositional techniques to re-
duce a very complex system into smaller, provable pieces.
2. RELATED WORK
Many methods for ensuring correctness have been researched,
each having di erent strengths in dealing with the various chal-
lenges posed by air tra c control. Pallottino et al. [11] proposed
a distributed collision avoidance policy that is closely related to
the systems we examine here. They provide a thorough empiri-
cal description of the system’s behavior, emphasizing simulation
and physical experiment. They formulate a liveness property and
give probabilistic evidence for it using Monte Carlo methods. They
also provide an informal proof of safety that is similar in high-level
ideas to our proofs, but does not consider a model for ﬂight dy-
namics. However, since we provide formal proofs of safety based
directly on the control protocols and working with a continuous
model of ﬂight dynamics, we provide a higher degree of assurance
and a clearer avenue to safely extend the systems.
Veriﬁcation methods for systems with an arbitrary number of
agents behaving under distributed control fall primarily into one
of two categories: theorem proving and parameterized veriﬁcation.
Johnson and Mitra [8] use parameterized veriﬁcation to guaran-
tee that a distributed air tra c landing protocol (SATS) is collision
free. Using backward reachability, they prove safety in the SATS
landing maneuver given a bound on the number of aircraft that
can be engaged in the landing maneuver. The protocol divides the
airspace into regions and models the aircraft ﬂight trajectory within
each region by a clock. We consider the complementary problem
of free ﬂight instead of airport landing tra c, and we show that in
free space, arbitrarily many aircraft can join our maneuver, and we
model aircraft movement using ﬂyable , curved ﬂight dynamics.
Other provably safe systems with a speciﬁc (usually small) num-
ber of agents are presented in [1, 4, 7, 9]. The work by Umeno
and Lynch [4, 7] is complementary to ours; however while they
consider real-time properties of airport protocols using Timed I /O
Automata, we prove local properties of the actual hybrid system
ﬂight dynamics. Duperret et al. [9] verify a roundabout maneu-ver with three vehicles. Each vehicle is constrained to a speciﬁc,
pre-deﬁned path, so physical dynamics are simpliﬁed to one di-
mension. Tomlin et al. [1] analyze competitive aircraft maneu-
vers game theoretically using numerical approximations of partial
dierential equations. As a solution, they propose roundabout ma-
neuvers and give bounded-time veriﬁcation results for up to four
aircraft using straight-line approximations of ﬂight dynamics.
Flyability is identiﬁed as a major challenge in Košecká et al. [12],
where planning based on superposition of potential ﬁelds is used
to resolve air tra c conﬂicts. This planning does not guarantee
ﬂyability but, rather, defaults to classical vertical altitude changes
whenever a nonﬂyable path is detected. The resulting maneuver
has not yet been veriﬁed. The planning approach has been pursued
by Bicchi and Pallottino [13] with numerical simulations.
Numerical simulation algorithms approximating discrete-time
Markov Chain approximations of aircraft behavior have been pro-
posed by Hu et al. [2]. They approximate bounded-time probabilis-
tic reachable sets for one initial state. We consider hybrid systems
combining discrete control choices and continuous dynamics in-
stead of uncontrolled, probabilistic continuous dynamics. Hwang
et al. [6] have presented a straight-line aircraft conﬂict avoidance
maneuver involving optimization over complicated trigonometric
computations, and validate it using random numerical simulation
and informal arguments. The work of Dowek et al. [3] and Galdino
et al. [5] shares many goals with ours. They consider unﬂyable,
straight-line maneuvers and formalize geometrical proofs in PVS.
Our approach has a di erent focus from complementary work:
Our maneuver directly involves curved ﬂight unlike [1–8].
This makes our maneuver more realistic since it is ﬂyable ,
but much more di cult to analyze.
Unlike [2,6,12], we do not give results for a ﬁnite (sometimes
small) number of initial ﬂight positions (as in simulation).
Instead, we verify uncountably many initial states and give
unbounded-time horizon veriﬁcation results.
Unlike [1, 2, 6, 12–14], we use symbolic computation so that
numerical and ﬂoating point errors can not violate soundness.
Unlike [2–8, 13, 15], we analyze hybrid system dynamics di-
rectly, not approximations like clocks.
Unlike [1,2,6,11–13,15] we produce formal , deductive proofs.
In [3–7, 11], it is not proved that the hybrid dynamics and
ﬂight equations follow the geometrical thoughts. In contrast,
our approach directly works for the hybrid ﬂight dynamics.
Unlike [1–3,5,6,9,10,12,13], we verify the case of arbitrar-
ily many aircraft, which is crucial for dense airspace.
Unlike [13,14], we do not guarantee optimality of the result-
ing maneuver.
3. PRELIMINARIES
Quantiﬁed Hybrid Programs.
QHPs [16, 17] are deﬁned by the following grammar ( ; are
QHPs,terms, ia variable of sortC,fa function symbol, sa term
with sort compatible to f, and His a formula of ﬁrst-order logic):
; ::=8i:CAj8 i:CfD &Hgj?Hj[j;j
whereAis a list of assignments of the form f(s) :=and nonde-
terministic assignments of the form f(s) :=C, andDis a list of
dierential equations of the form f(s)0=. When an assignment
list does not depend on the quantiﬁed variable i, we may elide the
quantiﬁcation for clarity.
The e ect of assignment f (s) :=is a discrete jump assigning
tof(s). The e ect of nondeterministic assignment f (s) :=Cis
a discrete jump assigning any value inCtof(s). The e ect of
quantiﬁed assignment 8i:CA is the simultaneous e ect of all as-
signments inAfor all objects iof sortC. For example, the QHP
8i:C!(i) :=!(i)+1 expresses that all aircraft iof sortCsimul-
taneously increase their angular velocity. The e ect of quantiﬁed
dierential equation8i:CfD&Hgis a continuous evolution where,
for all objects iof sortC, all di erential equations in Dhold and
formula Hholds throughout the evolution (i.e. the state remains
in the region described by evolution domain constraint H ). The
dynamics of QHPs changes the interpretation of terms over time:
for an R-valued function symbol f,f(~s)0denotes the derivative
of the interpretation of the term f(~s) over time during continuous
evolution, not the derivative of f(~s) by its argument ~s. For this
paper, fdoes not occur in ~s. In most cases, ~sis just i. For in-
stance, the following QHP expresses that all aircraft iof sortC
ﬂy by8i:Cx(i)0=d(i);d(i)0=!(i)d(i) such that their position x(i)
changes continuously according to their respective direction d(i)
and their direction changes according to their angular velocity !(i).
The e ect of test?His askip (i.e., no change) if formula His
true in the current state and abort (blocking the system run by a
failed assertion), otherwise. Nondeterministic choice [is for
alternatives in the behavior of the distributed hybrid system. In the
sequential composition ;, QHPstarts afterﬁnishes (never
starts ifcontinues indeﬁnitely). Nondeterministic repetition
repeatsan arbitrary number of times, possibly zero times.
Quantiﬁed Differential Dynamic Logic.
The formulas of Q dL[16, 17] are deﬁned as in ﬁrst-order dy-
namic logic plus many-sorted ﬁrst-order logic by the following
grammar (; are formulas, 1;2are terms of the same sort, iis a
variable of sortC, andis a QHP):
; ::=1=2j12j:j^ j_
j8i:Cj9i:Cj[]jhi
We use standard abbreviations to deﬁne ;>;<;!. The real num-
bersRform a distinguished sort, upon which are deﬁned the rigid
functions +and. SortsC,Rhave no ordering and hence 1=2
is the only relation allowed on them. For sort R, we abbreviate
8x:Rby8x. In the following, all formulas and terms have to
be well-typed. Q dLformula []expresses that all states reach-
able by QHP satisfy formula . Likewise,hiexpresses that
there is at least one state reachable by for whichholds.
Proof Calculus and Prover.
The Q dLproof calculus [16, 17] consists of proof rules that op-
erate on sequents , which are syntactic objects of the form )
where andare ﬁnite sets of Q dLformulas. Loos et al. [18] use
QdLto verify adaptive cruise control for arbitrarily many cars on a
highway, with simple continuous dynamics on a straight lane.
KeYmearaD is a theorem prover which mechanizes the use of the
QdLproof calculus. It has previously been used to verify a simple
car control system [19]. KeYmaeraD further implements quantiﬁed
dierential invariants [20]. KeYmaeraD constructs proofs by fol-
lowing a user-created tactic script . When KeYmaeraD runs a tactic
script, it applies proof rules to the sequent based on tactics speci-
ﬁed in the script which will ultimately reduce the sequent to several
problems in ﬁrst-order real arithmetic. These simpler problems are
then sent to a backend decision procedure in Mathematica.
4. CASE STUDY SYSTEMS
In this section, we present two classes of aircraft controllers,
each of which maintains a guaranteed minimum distance pbetweenall aircraft. We then prove that both of these controller classes are
safe, (i.e. that the minimum distance pbetween aircraft is never vi-
olated). Each aircraft has a disc-shaped zone large enough to ﬂy a
circle within and which no other aircraft will be allowed to enter.
In the ﬁrst controller class (Section 4.1), each aircraft maintains
a larger bu er disc with the aircraft at its center. This disc allows
pilots some freedom during an avoidance maneuver, including the
choice of circling direction. We imagine this controller will be
useful when passenger comfort is a factor, as in commercial air-
lines. The second class of controllers (Section 4.2) uses smaller
buer discs centered to the left or right of the aircraft. The smaller
discs allow the aircraft to ﬂy closer together, but there may be little
choice in how a maneuver is executed. This is well suited to UA Vs
which may ﬂy very close together and are concerned only with ﬂya-
bility, not passenger comfort. Additionally, since many UA Vs may
be monitored and managed remotely by a small group of people, it
may be more desirable to have a speciﬁc collision avoidance ma-
neuver with little freedom and high predictability. Because the ﬁrst
controller class requires a larger disc than the second, we call it Big
Disc, and appropriately we name the second class Small Discs .
We use two levels of abstraction to analyze the controllers. At the
higher abstraction level we model the bu er discs, which can freeze
instantaneously when they get within pdistance of each other. At
the lower level, we model the movement of aircraft within their
discs, ensuring they always stay within the bu er zone while fol-
lowing ﬂyable trajectories. In the proof, these two levels of abstrac-
tion are joined so that safety is assured for the system as a whole.
We model airspace as R2and aircraft as points moving in this
space. Each aircraft isteers by adjusting its angular velocity !(i).
When!(i) is zero, the plane ﬂies in a straight line. As angular
velocity increases, the plane ﬂies in a tighter and tighter circle, so
we put an upper bound on the angular velocity
(i) based on the
smallest circle that aircraft ican ﬂy while maintaining constant lin-
ear speed v(i). We keep linear speed v(i) constant for each aircraft.
We can determine the radius of each aircraft’s smallest ﬂyable cir-
cle by the equation minr (i)=v(i)=
(i). This model is known as the
Dubins vehicle [21] and has been used previously for aircraft veri-
ﬁcation [1]. We allow an arbitrarily large number of aircraft to be
present in airspace, so long as there is enough space to pack their
discs. To our knowledge, no other method has been able to verify a
protocol or controller safe for an arbitrary number of aircraft using
a continuous model of their ﬂight dynamics. This is not surprising,
since safety must be guaranteed even for unpredictable emergent
behaviors and in crowded, worst-case scenarios. Models written
as QHPs inherently have a compositional and hierarchical struc-
ture which makes them easier to decompose into smaller, provable
pieces by using sound proof rules. We also use nondeterminism in
the model of the controller, which means that our proof is robust to
variations in implementation on individual aircraft.
4.1 Big Disc
During normal free ﬂight (i.e. whenever the aircraft is not en-
gaged in a collision avoidance maneuver), the bu er zone for an
aircraft iis a disc of radius 2 minr (i) centered around the aircraft,
which is at planar position x(i)=(x1(i);x2(i)). So long as the air-
craft does not enter a collision avoidance maneuver, its bu er disc
remains centered on the aircraft. However, should aircraft icome
too close to another plane, it will enter collision avoidance mode
and begin circling at radius minr (i) to either the left or the right.
The disc allows just enough room for this maneuver; however, the
disc is big in the sense that it allows a considerable amount of free-
dom once the aircraft has gone halfway around this initial circle.
The beginning of one possible trajectory of a collision avoidance
2minr/LParen1i/RParen1
minr/LParen1i/RParen1
d/LParen1i/RParen1x/LParen1i/RParen1
disc/LParen1i/RParen1Figure 1: A possible collision avoidance trajectory of BigDisc .
maneuver is illustrated in Figure 1. The current direction of ﬂight
of aircraft iis given by the indexed variable d(i) as a 2D unit vec-
tor. The variable disc(i) stores the position of the center of i’s buer
disc. The aircraft need not always turn at the maximum angular ve-
locity
(i); we require only that the aircraft remain within the disc
by circling in its original direction. (Note: while it is possible for
the aircraft to change its circling direction while staying within the
disc by ﬂying a ﬁgure eight or an ‘S’ shape, we disallow this be-
havior since it would increase the complexity of the controller.)
4.1.1 Formal Model
The Big Disc policy is presented formally in QHP 1 and we de-
scribe it in this section. The variable ca(i), indicates whether air-
craft iis in a collision avoidance maneuver. If ca(i)=0 then iis
in free ﬂight; if ca(i)=1 then iis in an avoidance maneuver and
is circling within its disc. Each aircraft has the ability to enter col-
lision avoidance independently and asynchronously. This simpli-
ﬁes collision avoidance maneuvers with more than two aircraft and
improves reliability, since no perfect synchronization is required,
which would be di cult to implement in a distributed system.
The variable side(i) indicates i’s circling direction when it en-
ters an avoidance maneuver. If side(i)=1,iwill circle counter-
clockwise; if side(i)=1,icircles clockwise. We use kykto denote
the Euclidean norm and we use y?to denote the vector obtained by
rotating yninety degrees counter-clockwise, y?B(y2;y1).
The quantiﬁed hybrid program BigDisc is a loop, which is rep-
resented in line 1 by, the nondeterministic repetition operator.
Each iteration is either a control action as represented by Control
or an evolution of physics as represented by Plant . This loop
may repeat arbitrarily many times. In the Control branch, the
program nondeterministically selects an aircraft k(kBAassigns
an arbitrary aircraft into k) and then allows kto perform some ac-
tion. The allowed actions depend on whether kis in a collision
avoidance maneuver. If it is (case CA), then kmay either adjust its
angular velocity in the Steer branch, or exit the maneuver with
theExit branch. The new angular velocity in the Steer branch
is arbitrary ( !(k) :=R, whereRis an arbitrary real number) but
bounded by
(k) and
(k) due to the subsequent test. The aircraft
may only Exit the collision avoidance circling maneuver when
x(k)=disc(k), i.e., the aircraft must return to the center of the disc
before exiting the maneuver. If kis not in a collision avoidance
maneuver (case NotCA ), then it may once again Steer , or it may
switch its circling direction with the Flip branch, or it may enter
collision avoidance with the Enter branch. In the Enter branch,
the aircraft sets its angular velocity so that it will circle with ra-
dius minr (k), thereby entering a collision avoidance. It also sets the
ca(k) ﬂag to indicate internally that it has entered this maneuver.
The other branch in BigDisc ’s main loop is Plant . The po-
sition of the aircraft, x(i), changes according to its direction, d(i),
which in turn changes according to its angular velocity, !(i). ThisQuantiﬁed Hybrid Program 1 Big Disc
BigDisc(Control[Plant )(1)
ControlkBA; (CA[NotCA ) (2)
CA?(ca(k)=1); (Steer[Exit ) (3)
NotCA?(ca(k)=0); (Steer[Flip[Enter ) (4)
Steer!(k)BR; ?(
(k)!(k)
(k)) (5)
Exit?(disc(k)=x(k));ca(k)B0 (6)
Enter!(k)Bside(k)
(k);ca(k)B1 (7)
Flipside(k)Bside(k) (8)
Plant 8i:A
x(i)0=v(i)d(i);d(i)0=!(i)d(i)?; (9)
disc(i)0=1ca(i)v(i)d(i) &EvDom
(10)
EvDom 8 j:A (11)
(j,i^(ca(i)=0_ca(j)=0))!Sep(i;j) (12)
^jjdisc(i)(x(i)+minr (i)side(i)d(i)?)jj
minr (i)(13)
Sep(i;j) kdisc(i)disc(j)k2minr (i)+2minr (j)+p(14)
makes!(i) our primary control variable. These physical dynamics
are modeled by the di erential equation in line 9. The center of the
disc, disc(i), is stationary during a collision avoidance maneuver,
but otherwise it is equal to the aircraft position. This case distinc-
tion is achieved by using arithmetic coding, whereby we multiply
by (1ca(i)), in line 10. This reduces a branching of the system
which would be incurred if we were to use traditional if-else cod-
ing style, and thereby reduces the complexity of the safety proof.
When the aircraft is in free ﬂight, ca(i)=0, which causes disc(i)0
to equal x(i)0. But, when the aircraft is in a collision avoidance ma-
neuver, ca(i)=1, causing disc(i)0=0, so the disc is stationary.
The evolution domain, EvDom , has two purposes. First, it monitors
the disc positions of other aircraft (line 12). Recall that in order for
the system to be considered safe, no aircraft can pass closer than
distance pto another. So, if the aircraft’s disc comes within pof
another disc (as quantiﬁed in line 14), it forces both aircraft to enter
collision avoidance. Second, while the aircraft has a great amount
of freedom in how it maneuvers during collision avoidance, it must
always be able to ﬂyably remain within its bu er disc. We use the
inequality in line 13 to quantify this condition. It states that if the
aircraft turns in a tight circle with radius minr (i), the origin of that
tight circle is no more than minr (i) away from the point disc(i).
Our model allows for a huge amount of nondeterminism, both in
the discrete dynamics of the controller (e.g. which aircraft are con-
trolled ( kBA, line 2), how the aircraft steer ( !(k)BR, line 5),
and whether to enter a collision avoidance maneuver), and in the
continuous dynamics of the plant (e.g. how long to wait between
control choices). This nondeterminism is a beneﬁcial property of
our collision avoidance protocol, since it allows for each aircraft to
implement slightly di erent control algorithms without violating
the proof of safety for the entire system. As a result, we have veri-
ﬁed a class of controllers, rather than one speciﬁc implementation.
4.1.2 Theorem Statement
In order to guarantee safety, we must prove that for all pairs of
distinct aircraft i;j, the distance between iand jis greater than or
equal to p. We can express this condition formally as
Safe8i;j:Ai,j!kx(i)x(j)kp:
We must show that Safe holds during all executions of BigDisc .
The Q dLformula expressing this property is [ BigDisc ]Safe . We
must also ensure that the aircraft begin in a controllable state. This
means each aircraft must have a bu er disc ( InitA ), which is empty
(InitB ), and within which it may ﬂyably maneuver ( InitC ). For
aircraft ithat are not in an avoidance maneuver, we ensure that i’s
disc is at the same point as i. Since ca(i)=1 for aircraft in a ma-
neuver and ca(i)=0 otherwise, we may write this property as
InitA8i:A(1ca(i))disc(i)=(1ca(i))x(i):
By again using this arithmetic coding style rather than an if-else
statement, we eliminate a signiﬁcant branching factor in the result-
ing proof. We then show that the discs are empty by ensuring suf-
ﬁcient separation between the discs as deﬁned in QHP 1 line 14.
InitB8i;j:Ai,j!Sep(i;j)
Finally, we ensure that aircraft in collision avoidance maneuvers
are able to ﬂyably remain within their discs. We do this by proving
that the following formula is an invariant of our system. It states
that if itightly circles in its current circling direction, the center of
this tight circle will be within distance minr (i) ofdisc(i).
InitC8i:Ajjdisc(i)(x(i)+minr (i)side(i)d(i)?)jjminr (i)
Note that these initial conditions all hold trivially if the aircraft be-
gin far enough apart that none is in a collision avoidance maneuver.
Theorem 1 (S afety of BigDisc ).If the aircraft are initially
in a controllable state, then no two aircraft will come closer than
distance p while all aircraft follow Control ; therefore safety of the
BigDisc controller is expressed by the provable QdLformula:
(InitA^InitB^InitC )![BigDisc ]Safe
We proved Theorem 1 for all parameter values by showing that
InitA ,BandCare maintained as invariants. This proof was gener-
ated using KeYmaeraD from a 330 line user-generated tactic script.
The tactic ﬁle and the KeYmaeraD theorem prover are available
online [22]. A discussion of the critical techniques needed to com-
plete this proof is presented in [23, Appendix A.1].
4.2 Small Discs
One drawback of the Big Disc policy is that it may trigger colli-
sion avoidance maneuvers that are not strictly necessary; the bu er
zones are larger than the required circling space of the aircraft. Our
second policy, Small Discs , aims to decrease the size of the disc.
The only way to do this is to abandon the assumption that the disc
must be centered on the aircraft during free ﬂight. Instead, the
buer zone, a disc of radius minr (i), is centered at a point with
distance minr (i) away from x(i), in a direction perpendicular to i’s
motion either to the left or the right. Thus the aircraft is always
on the edge of its disc, and during collision avoidance, the aircraft
follows the circumference of its disc. As with BigDisc , an aircraft
may ﬂip its circling direction during free ﬂight. This now makes
the disc jump to the other side of the aircraft, and before an aircraft
can ﬂip its circling direction, it must check that it may do so safely.
Fig. 2 illustrates a situation where ﬂipping the disc to the op-
posite side prevents an unnecessary collision avoidance maneuver.
Each aircraft has an active disc (solid discs in Fig. 2) that it will use
for collision avoidance if needed. We also illustrate the inactive
disc(dotted discs) as an alternative choice for the disc if the aircraft
decides to ﬂip its circling direction. In Fig. 2, the active discs of air-
craft iand aircraft jare on a collision course. If nothing is done, at
the latest when the edges of the discs are separated by distance p,
both aircraft will enter collision avoidance by circling to the right
x/LParen1i/RParen1x/LParen1j/RParen1
p x/LParen1k/RParen1x/LParen1l/RParen1
x/LParen1m/RParen1Figure 2: One possible scenario in the Small Discs policy
around the circumference of their respective discs. Notice that in
this case, the aircraft may pass as close as the minimum separation
distance pwhen aircraft iis at the top of its disc while aircraft jis
at the bottom. However, since aircraft ihas free space to its left, it
may ﬂip its circling disc to the opposite side and no collision avoid-
ance maneuvers are necessary. Aircraft jis unable to make such a
ﬂip, since aircraft k’s disc occupies the necessary space. Only col-
lisions of active discs are a problem. The fact that the inactive discs
ofkandmoverlap is immaterial, because if collision avoidance is
necessary, every aircraft will follow the circumference of its active
disc. This also illustrates that aircraft must synchronize disc ﬂip-
ping. If, to enable jto ﬂip its disc, kﬂips its disc, but, at the same
time, and unaware of this, mﬂips its disc, then kandmwould have
incompatible collision avoidance discs. Since purely discrete stan-
dard solutions exist for ensuring consistency in such discrete mode
changes, our model simply uses sequentialized ﬂipping decisions.
4.2.1 Formal Model
The Small Discs policy is presented formally as SmallDiscs
in QHP 2. The overall structure is similar to that of BigDisc .
One notable di erence is that SmallDiscs no longer uses the state
variable disc(i). During free ﬂight, the center of an aircraft’s disc
moves with dynamics that are not easy to express in terms of other
variables; it is certainly not as easy as setting disc(i)0=x(i)0, as we
did in BigDisc . The point disc(i) moves faster or slower than x(i),
depending on whether the aircraft is veering away from its active
disc, or towards it. As a result, disc(i) has very involved continu-
ous dynamics. Fortunately, however, the position of aircraft i’s disc
can be simply expressed in terms of other state variables. By using
dierential-algebraic equations as in [24], we equate
disc(i)=x(i)+minr (i)side(i)d(i)?: (15)
In order to simplify the mathematics of the system, we directly
reduce the system to ordinary di erential equations, which also
makes the connection to BigDisc more apparent. Thus, instead
of using (15) as part of a di erential-algebraic equation [24], we
consider (15) as a deﬁnition and statically replace all occurrences
ofdisc(i) by the right-hand side of (15). The subsequent model
should be read with this in mind. The other major change in QHP
2 is the separation condition Sep and the newly introduced separa-
tion condition FlipSep for ﬂipping the disc.
4.2.2 Theorem Statement
Here again we want to prove that under safe initial conditions,
theSmallDiscs controller is always Safe , where Safe is exactly as
we deﬁned it for BigDisc . We need to modify the initial conditions
forSmallDiscs . There are two properties that we want to hold:
ﬁrst, the discs are separated ( InitD ), and second, when an air-
craft is engaged in collision avoidance it is ﬂying along the circum-
Quantiﬁed Hybrid Program 2 Small Discs
SmallDiscs(Control[Plant )(16)
ControlkBA; (CA[NotCA ) (17)
CA?(ca(k)=1); (Exit[Skip ) (18)
NotCA?(ca(k)=0); (Steer[Flip[Enter ) (19)
Skip?true (20)
Steer!(k)BR; ?(
(k)!(k)
(k)) (21)
Exitca(k)B0 (22)
Enter(!(k)Bside(k)
(k));ca(k)B1 (23)
Flip?(8j:A(j,k!FlipSep (j;k))); (24)
side(k)Bside(k) (25)
FlipSep (i;j) k(x(i)+minr (i)side(i)d(i)?) (26)
(x(j)minr (j)side(j)d(j)?)k (27)
minr (i)+minr (j)+p (28)
Plant 8i:A
x(i)0=v(i)d(i);d(i)0=!(i)d(i)?(29)
&8j:A(j,i^(ca(i)=0_ca(j)=0)) (30)
!Sep(i;j)
(31)
Sep(i;j) k(x(i)+minr (i)side(i)d(i)?) (32)
(x(j)+minr (j)side(j)d(j)?)k (33)
minr (i)+minr (j)+p (34)
ference of its active disc ( InitE ).InitD is similar to InitB for
BigDisc , but it uses our new deﬁnition of Sep forSmallDiscs :
InitD8i;j:Ai,j!Sep(i;j):
Ifiis in a collision avoidance maneuver, then iis turning at max-
imal angular velocity. This implication is expressed with arithmetic
coding by multiplying both sides with the indicator ca(i):
InitE8i:A!(i)ca(i)=
(i)side(i)ca(i):
Theorem 2 (S afety of SmallDiscs ).If the aircraft are initially
in a controllable state (i.e. where InitD andInitE hold), then
no aircraft will come closer than distance p to any other aircraft
so long as each aircraft follows Control ; therefore safety of the
SmallDiscs controller is expressed by the provable QdLformula:
(InitD^InitE )![SmallDiscs ]Safe
We proved Theorem 2 in KeYmaeraD by showing InitD and
InitE are invariant. The accompanying tactic script is 309 lines in
length and is available online [22].
5. REFERENCES
[1] Tomlin, C., Pappas, G.J., Sastry, S.: Conﬂict resolution for
air tra c management. IEEE T. Automat. Contr. 43(4)
(1998) 509–521
[2] Hu, J., Prandini, M., Sastry, S.: Probabilistic safety analysis
in three-dimensional aircraft ﬂight. In: CDC. (2003)
[3] Dowek, G., Muñoz, C., Carreño, V .A.: Provably safe
coordinated strategy for distributed conﬂict resolution. In:
AIAA-2005-6047. (2005)
[4] Umeno, S., Lynch, N.A.: Proving safety properties of an
aircraft landing protocol using I /O automata and the PVS
theorem prover. In Misra, J., Nipkow, T., Sekerinski, E., eds.:
FM. V olume 4085 of LNCS., Springer (2006) 64–80[5] Galdino, A.L., Muñoz, C., Ayala-Rincón, M.: Formal
veriﬁcation of an optimal air tra c conﬂict resolution and
recovery algorithm. In Leivant, D., de Queiroz, R., eds.:
WoLLIC. V olume 4576 of LNCS., Springer (2007) 177–188
[6] Hwang, I., Kim, J., Tomlin, C.: Protocol-based conﬂict
resolution for air tra c control. Air Tra c Control Quarterly
15(1) (2007) 1–34
[7] Umeno, S., Lynch, N.A.: Safety veriﬁcation of an aircraft
landing protocol: A reﬁnement approach. In Bemporad, A.,
Bicchi, A., Buttazzo, G., eds.: HSCC. V olume 4416 of
LNCS., Springer (2007) 557–572
[8] Johnson, T., Mitra, S.: Parameterized veriﬁcation of
distributed cyber-physical systems:an aircraft landing
protocol case study. In: ACM /IEEE ICCPS. (2012)
[9] Duperret, J.M., Hafner, M.R., Del Vecchio, D.: Formal
design of a provably safe roundabout system. (In: IEEE /RSJ
IROS) 2006–2011
[10] Platzer, A., Clarke, E.M.: Formal veriﬁcation of curved ﬂight
collision avoidance maneuvers: A case study. In: FM.
V olume 5850 of LNCS., Springer (2009) 547–562
[11] Pallottino, L., Scordio, V ., Frazzoli, E., Bicchi, A.:
Decentralized cooperative policy for conﬂict resolution in
multi-vehicle systems. IEEE Trans. on Robotics 23(6) (2007)
[12] Košecká, J., Tomlin, C., Pappas, G., Sastry, S.: 2-1 /2D
conﬂict resolution maneuvers for ATMS. In: CDC.
V olume 3., Tampa, FL, USA (1998) 2650–2655
[13] Bicchi, A., Pallottino, L.: On optimal cooperative conﬂict
resolution for air tra c management systems. IEEE Trans.
ITS1(4) (2000) 221–231
[14] Hu, J., Prandini, M., Sastry, S.: Optimal coordinated motions
of multiple agents moving on a plane. SIAM Journal on
Control and Optimization 42(2003) 637–668
[15] Massink, M., Francesco, N.D.: Modelling free ﬂight with
collision avoidance. In Andler, S.F., O utt, J., eds.: ICECCS,
Los Alamitos, IEEE (2001) 270–280
[16] Platzer, A.: Quantiﬁed di erential dynamic logic for
distributed hybrid systems. In Dawar, A., Veith, H., eds.:
CSL. V olume 6247 of LNCS., Springer (2010) 469–483
[17] Platzer, A.: A complete axiomatization of quantiﬁed
dierential dynamic logic for distributed hybrid systems.
Logical Methods in Computer Science 8(4) (2012) 1–44
[18] Loos, S.M., Platzer, A., Nistor, L.: Adaptive cruise control:
Hybrid, distributed, and now formally veriﬁed. In Butler, M.,
Schulte, W., eds.: FM. LNCS, Springer (2011) 42–56
[19] Renshaw, D.W., Loos, S.M., Platzer, A.: Distributed theorem
proving for distributed hybrid systems. In Qin, S., Qiu, Z.,
eds.: ICFEM. V olume 6991 of LNCS., Springer (2011)
356–371
[20] Platzer, A.: Quantiﬁed di erential invariants. In Frazzoli, E.,
Grosu, R., eds.: HSCC, ACM (2011) 63–72
[21] Dubins, L.E.: On curves of minimal length with a constraint
on average curvature, and with prescribed initial and terminal
positions and tangents. Am J Math 79(3) (1957) pp. 497–516
[22] Electronic proofs: www.ls.cs.cmu.edu/discworld .
[23] David Renshaw, Sarah M. Loos, A.P.: Formal veriﬁcation of
distributed aircraft controllers. Technical Report
CMU-CS-12-132, Carnegie Mellon (2012)
[24] Platzer, A.: Di erential-algebraic dynamic logic for
dierential-algebraic programs. J. Log. Comput. 20(1)
(2010) 309–352

Sarah M. Loos
David Renshaw
Andre Platzer