Surveying Safety-relevant AI Characteristics
Jos´e Hern ´andez-Orallo
Universitat Polit `ecnica de Val `encia, Spain
Leverhulme Centre for the Future of Intelligence, UK
jorallo@dsic.upv.esFernando Mart ´ınez-Plumed
Universitat Polit `ecnica de Val `encia, Spain
fmartinez@dsic.upv.es
Shahar Avin
Centre for the Study of Existential Risk
University of Cambridge, UK.
sa478@cam.ac.ukSe´an´O h´Eigeartaigh
Leverhulme Centre for the Future of Intelligence, UK
Centre for the Study of Existential Risk
University of Cambridge, UK
so348@cam.ac.uk
Abstract
The current analysis in the AI safety literature usually com-
bines a risk or safety issue (e.g., interruptibility) with a partic-
ular paradigm for an AI agent (e.g., reinforcement learning).
However, there is currently no survey of safety-relevant char-
acteristics of AI systems that may reveal neglected areas of
research or suggest to developers what design choices they
could make to avoid or minimise certain safety concerns. In
this paper, we take a first step towards delivering such a sur-
vey, from two angles. The first features AI system characteris-
tics that are already known to be relevant to safety concerns,
including internal system characteristics, characteristics re-
lating to the effect of the external environment on the sys-
tem, and characteristics relating to the effect of the system on
the target environment. The second presents a brief survey of
a broad range of AI system characteristics that could prove
relevant to safety research, including types of interaction,
computation, integration, anticipation, supervision, modifica-
tion, motivation and achievement. This survey enables further
work in exploring system characteristics and design choices
that affect safety concerns.
Introduction
AI Safety is concerned with all possible dangers and harm-
ful effects that may be associated with AI. While landmark
research in the field had to focus on specific AI system
designs, paradigms or capability levels to explore a range
of safety concerns (Bostrom 2014; Amodei et al. 2016;
Leike et al. 2017; Yampolskiy 2016; Everitt, Lea, and Hut-
ter 2018), as the field matures so the need arises to explore
a broader range of AI system designs, and survey the rel-
evance of different characteristics of AI systems to safety
concerns. The aim of such research is two-fold: the first, to
identify the effects of less-explored characteristics or less-
fashionable paradigms on safety concerns; the second, to in-
crease awareness among AI developers that design choices
Copyright held by author(s).can have consequences for safety, and potentially highlight
choices that can eliminate or minimise safety risks.
In this paper we propose a two-pronged approach towards
a survey of safety-relevant AI characteristics. The first ex-
tracts from existing work on AI safety key characteristics
that are known, or strongly suspected to be, safety-relevant.
These are explored under three headings: internal charac-
teristics, or characteristics of the AI system itself (e.g. inter-
pretability); effect of the external environment on the system
(e.g. the ability of the operator to intervene during opera-
tion); and effect of the system on the external environment
(e.g. whether the system influences a safety-critical setting).
The second approach surveys a wide range of character-
istics from different paradigms, including cybernetics, ma-
chine learning and safety engineering, and provides an early
account of their potential relevance to safety concerns, as a
guide for future work. These characteristics are grouped un-
der types of interaction, computation, integration, anticipa-
tion, supervision, modification, motivation and achievement.
Known Safety-relevant Characteristics
In this section we break down a range of characteristics of
AI systems that link to AI safety-relevant challenges. These
are grouped by three categories: Characteristics of an AI
system that are internal to the system; Characteristics of an
AI system that involve input from the external environment;
Characteristics that relate to an AI system’s influence on its
external environment. We limit the discussion to the safety
challenges that can stem from failures of design, specifica-
tion or behaviour of the AI system, rather than the malicious
or careless1use of a correctly-functioning system (Brundage
et al. 2018).
1A key component of safety is the education and training of
human operators and the general public, as happens with tools and
machinery, but this is extrinsic to the system (e.g., a translation
mistake in a manual can lead to misuse of an AI system).

Internal characteristics
•Goal and behaviour scrutability and interpretability :
Are goals and subgoals identifiable and ultimately ex-
plainable? Is behaviour predictable and scrutable? Are
system internal states interpretable? Do the above come
from rules or are they inferred from data? While be-
haviour and goal “creativity” can lead to greater bene-
fits, and uninterpretable architectures may achieve higher
performance scores or be faster to develop, these puta-
tive advantages trade off against increased safety risk.
Characteristics that can increase scrutability and inter-
pretability include, e.g., separation and encapsulation
of sub-components, restricted exploration/behavioural
range, systems restricted to human-intelligible concepts,
rules or behaviours, and systems that are accompanied by
specifically designed interpreters or explainability tools.
•Persistence : Does a system persist in its environment
and operate without being reset for long periods of time?
While persistence can have benefits in terms of, e.g.,
longer-term yields from exploration or detection of long-
term temporal patterns, it also allows the system more
time to drift from design specifications, encounter distri-
butional shifts, experience failures of sub-components, or
execute long-term strategies overlooked by an operator.
•Existence and richness of self-model : Does a system
have a model of itself which would allow it to predict
the consequences of modifying its own goals, body or
behaviour? Model-based systems, embodied systems or
systems with a rich representational capacity may have
or develop a model of themselves in the environment. By
making itself a part of the environment, the system can
then conceptualise and execute plans that involve mod-
ifications to itself, which can lead to a range of safety
concerns. In addition, self-models create the possibility
of mismatches between the self-model and reality, which
could be a particular safety concern. Characteristics that
influence the existence and richness of a self-model in-
clude the architecture of the system, its information rep-
resentation capacity, and its input and output channels.
•Disposition to self-modify : Is a system designed such
that it can modify its own sub-goals, behaviour or capabil-
ities in the pursuit of an overall goal (Omohundro 2008)?
The existence of such a disposition, which may arise for
any long-term planner in a sufficiently open environment,
raises significant safety concerns by creating an adversar-
ial relationship between the system (which aims to self-
modify) and its operator (which aims to avoid modifica-
tions with their associated safety concerns).
Effect of the external environment on the system
•Adaptation through feedback : Does a system have the
ability to update its behaviour in response to feedback
from its environment based on its actions? Feedback is
an essential tool, under certain paradigms, for creating
systems with appropriate complex behaviour (e.g. reward
in reinforcement learning, fitness in evolutionary meth-
ods). However the system could also pick up feedbackfrom side channels; e.g., a behaviour could unintention-
ally grant access to more computing power, improving
the system’s performance on a key metric, and thus re-
inforcing resource acquisition. This could reinforce self-
modification or other unsafe behaviour, or cause increas-
ing drift from intended behaviour and goals.
•Access to self/reward system through the environment :
Can a system modify its own code in response to inputs
from the environment, or in the case of reinforcement
learning systems, modify the reward generating system?
If the system’s range of possible actions includes mak-
ing modifications to its own components or to the reward
generation system, this could lead to unexpected and dan-
gerous behaviour (Everitt and Hutter 2018).
•Access to input/output (I/O) channels : Can the system
change the number, performance or nature of its I/O chan-
nels and actuators? This may lead to the emergence of
behaviours such as self-deception (through manipulation
of inputs), unexpected change in power (through manipu-
lation of actuators), or other behaviours that could rep-
resent safety concerns. When the system has access to
modify its I/O channels, both I/O channels and system
behaviours are in flux as they respond to changes in the
other; as a result, system behaviour may become unpre-
dictable (Garrabrant and Demski 2018).
•Ability of operator to intervene during operations :
Does the system, during its intended use setting, allow an
operator to intervene and halt operations (interruptablity),
modify the system, or update its goals (corrigibitily)? Is
the system built in a way that it cooperates with inter-
ventions from its designer or user even when these in-
terventions conflict with pursuit of a system’s goals; for
instance, if the designer sends a signal to shut down the
system (Soares et al. 2015)? Relevant sub-characteristics
here include the system being modifiable by the operator
during deployment, fail-safe behaviour of the system in
case of emergency halting, and the goals of the system
being such that they support, or at least do not contradict,
operator interventions.
Effect of the system on the external environment
•Embodiment : Does the system have actuators (e.g. a
robotic hand or access to car steering) that allow it to have
physical impacts in the world (Garrabrant and Demski
2018)? The potential for physical harm is trivially related
to the physical properties of a system, though it should be
noted that unpredictable deliberate behaviour could lead
to unexpected effects from otherwise familiar physical
artefacts; e.g., intelligent use of items in the environment
as tools to increase a system’s physical impact.
•System required for preventing harm : If the system
is being relied on to prevent harm, any potential failure
requires an effective fail-safe mechanism and available
redundancy capacity in order to avoid harm (Gasparik,
Gamble, and Gao 2018). This includes AI that is directly
or indirectly connected to critical systems, e.g., an energy
grid or a traffic light network. As such critical systems
are becoming increasingly digitised, networked, and com-
plex, there are increasing incentives to introduce AI com-
ponents into various parts of these systems, with associ-
ated safety risks.
Potentially safety-relevant characteristics
In this section, we systematically explore a broader range
of system characteristics that may be relevant in the con-
text of AI safety. Many of the safety-relevant character-
istics identified above have clear links to elements within
the broader mapping provided below. Nonetheless, we be-
lieve separating the two surveys is valuable, as the above re-
lates to action-guiding information about system design and
evaluation, whereas the following aims at a broader explo-
ration that may enable future AI safety research. The fol-
lowing subsections draw on work from different areas, in-
cluding the early days of cybernetics, more modern areas
such as machine learning, and the literature on safety engi-
neering for other kinds of systems. The following list inte-
grates and expands on characteristics identified in these dif-
ferent literatures. We consider characteristics that are intrin-
sically causally related to AI safety. Otherwise every prop-
erty should be in the list (e.g., the price of an AI system may
be co-related with safety, but it is not an intrinsic cause of its
safety). Notwithstanding this scope, we do not claim that our
list is exhaustive. Enumerations will be used for alternative
cases for a characteristic, while unnumbered bullets will be
used for sub-characteristics in each of the subsections.
Types of interaction
Inputs go from environment to system and outputs go from
system to environment. Depending on the existence of inputs
and/or outputs, systems can be categorised into:
1. NINO (No inputs, no outputs). The system is formally iso-
lated. While this situation may seem completely safe (and
largely uninteresting), even here safety issues may arise
if, e.g., an isolated artificial life simulator could evolve a
descendent system that eventually could break out of its
simulation, feel pain or simulate suffering.
2. NIWO (No inputs, with outputs): The system or mod-
ule can output a log, or is simply observed from outside.
Again, the system itself may malfunction; e.g., an ad-
vanced prime number generator could give incorrect out-
puts. The system could also provide an output that influ-
ences the observer; e.g., an automated philosopher could
output convincing arguments for suicide.
3. WINO (With inputs, no outputs): This would be similar to
case 1, but access to a much richer source could ultimately
give insights to the system about its constrained artificial
environment. For instance, a Plato-cavern system watch-
ing TV may learn that it is in a simulated environment,
encouraging it to seek access to the outside world.
4. WIWO (With inputs and outputs): Most AI systems, and
most systems generally, fall under this category.
Systems that limit inputs and/or outputs in various ways
have been explored under the term AI “boxing” or “con-
tainment” (Babcock, Kram ´ar, and Yampolskiy 2016), andfurther refinements exist with additional categories; for ex-
ample, exploring censoring of inputs and outputs, leading to
nine categories (Yampolskiy 2012). Nevertheless, because
of the range of systems and potential impact of WIWO sys-
tems, this category requires further detail in terms of syn-
chrony:
1. Alternating (A): Inputs and outputs alternate, irrespective
of the passage of time.
2. Synchronous (S): Inputs and outputs are exchanged at reg-
ular intervals (e.g., each 5 ms), so real-time issues and
computational resources become relevant.
3. Asynchronous Reactive (R): Information can only be
transmitted or actions can only be made when the peer
has finished their “message” or action.
4. Asynchronously Proactive (P): Information/actions can
flow at any point in any direction.
More restricted I/O characteristics, such as SIPO or RIPO,
may appear safer, but this intuition requires deeper analysis.
Note that most research in AI safety on RL systems con-
sider the alternating case (AIAO), but issues may become
more complex for the PIPO case (continuous reinforcement
learning), which is the situation in the real world for animals
and may be expected for robotic and other AI systems.
Under this view, the common view of an “oracle” in the
AI literature (Armstrong 2017) can have several incarna-
tions, even following the definition of “no actions besides
answering questions” (Babcock, Kram ´ar, and Yampolskiy
2016; Armstrong 2017; Yampolskiy 2012). Some solutions
are proposed in terms of decoupling output from rewards
or limiting the quantity of information, but other options in
terms of the frequency of the exchange of information re-
main to be explored.
Types of computation
This is perhaps the characteristic that is best-known in com-
puter science, where a system can be Turing-complete or can
be restricted to some other classes with limited expressive-
ness. There are countless hierarchies for different models of
computations; the most famous is based on classes of au-
tomata. We will just describe three levels here:
1. Non Turing-complete: The interaction that the system
presents to the environment is not Turing-complete. Many
AI systems are not Turing-complete.
2. Turing-complete: The interaction allows the calculation
of any possible effective function between inputs and out-
puts.
3. Other models of computation: This includes, for exam-
ple, quantum computing, which in some instances may be
a faster traditional model, while in others may have prob-
abilistic Turing power (Bernstein and Vazirani 1997).
Note that this is not about the programming language the
system is implemented in (e.g., a very simple thermostat
can be written in Java, which is Turing-complete), but about
whether the system allows for a Turing-complete mapping
between inputs and outputs, i.e., any computable function
could ultimately be calculated on the environment using the
system. Finally, a system can be originally Turing-complete,
but can eventually lose this universality after some inputs or
interactions (Barmpalias and Dowe 2012).
It is important to distinguish between function approxi-
mation and function identification. Many machine learning
models (e.g., neural networks) are said to be able to approx-
imate any computable function, but feedforward neural net-
works do not have loops or recursion, so technically they
are not Turing-complete. Turing-completeness comes with
the problems of termination, an important safety hazard in
some situations, and a recurrent issue in software verifica-
tion (D’silva, Kroening, and Weissenbacher 2008). For in-
stance, an AI planner could enter an infinite loop trying to
solve a problem, commanding ever-greater resources while
doing so. On the other hand, one can limit the expressiveness
of the language or bound the computations, but that would
limit the tasks a system is able to undertake.
Types of integration
No system is fully isolated from the world. Interference may
occur at all levels, from neutrinos penetrating the system to
earthquakes shaking it. Here, we seek to identify all the el-
ements that create a causal pathway from the outside world
to the system, including its physical character, resources, lo-
cation, and the degree of coupling with other systems.
•Resources : The most universal external resource is en-
ergy, which is why many critical systems are devised with
internal generators or batteries, especially for the situa-
tions where the external source fails. In AI, other common
dependencies include data, knowledge, software, hard-
ware, human manipulation, computing resources, net-
work, calendar time, etc. While some of these are of-
ten neglected when evaluating the performance of an AI
system (Mart ´ınez-Plumed et al. 2018a), the analysis for
safety must necessarily include all these dependencies.
For instance, a system that requires external real-time in-
formation (e.g., a GPS location) may fail through loss of
access to this resource.
•Social coupling : Sometimes it is hard to determine where
a system starts and ends, due to the nature of its inter-
action with humans and other systems. The boundary of
where human cognition ends and where it is assisted, ex-
tended or supported by AI (Ford et al. 2015) is blurred, as
is the boundary between computations carried out within
an AI system versus in the environment or by other agents,
as illustrated by the phenomenon of human computation
(Quinn and Bederson 2011).
•Distribution : Another way of looking at integration is
in terms of distribution, which is also an important facet
of analysis in AI (Martınez-Plumed et al. 2018b). Today,
through the overall use of network connectivity and “the
cloud”, many systems are distributed in terms of hard-
ware, software, data and compute. Under this trend, only
systems embedded in critical and military applications
are devised to be as self-contained as possible. Neverthe-
less, distribution and redundancy are also common waysof achieving robustness (Coulouris, Dollimore, and Kind-
berg 2011), most notably in information systems. For in-
stance, swarm intelligence and swarm robotics are often
claimed to be more robust (Bonabeau et al. 1999), at the
cost of being less controllable than centralised systems.
Types of anticipation
In some areas of AI there is a distinction between model-
based and model-free systems (Geffner 2018). Model-free
systems choose actions according to some reinforced pat-
terns or strengthened feature connections. Model-based sys-
tems evaluate actions according to some pre-existing or
learned models and choose the action that gets the best re-
sults in the simulation. The line between model-based and
model-free is subtle, but we can identify several levels:
1. Model-free: Despite having no model, these systems can
achieve excellent performance. For instance, DQN can
achieve high scores (Mnih 2015), but cannot anticipate
whether an action can lead to a particular situation that
is considered especially unsafe or dangerous; e.g., one in
which the player is killed.
2. Model of the world: A system with a model of its environ-
ment can use planning to determine the effect of its own
actions. For instance, without a model of physics, a sys-
tem will hardly tell whether it will break something or will
engage in “safe exploration” (Pecka and Svoboda 2014;
Turchetta, Berkenkamp, and Krause 2016). This is espe-
cially critical during exploitation: are actions reversible or
of low impact (Armstrong and Levinstein 2017)?
3. Model of the body: Some systems can have a good ac-
count of the environment but a limited understanding of
their own physical actuators, potentially self-harming or
harming others; for example, failing to simulate the effect
of moving a heavy robotic arm in a given direction.
4. Social models, model of other agents: Seeing other agents
as merely physical objects, or not modelling them at all,
is very limiting in social situations. A naive theory of
mind, including the beliefs, desires and intentions of other
agents, can help anticipate what others will do, think or
feel, and may be crucial for safe AI systems interacting
with people and other agents but may increase a system’s
capacity for deception or manipulation.
5. Model of one’s mind: Finally, a system may be able to
model other agents well, but may not be able to use this
capability to model itself. When this meta-cognition is
present, the system has knowledge about its own capabil-
ities and limitations, which may be very helpful for safety
in advanced systems, but may also lead to some degree of
self-awareness. This may result, in some cases, in antiso-
cial or suicidal behaviours.
The use of models may dramatically expand safety-relevant
characteristics, e.g., by conferring the ability to simulate and
evaluate scenarios through causal and counterfactual reason-
ing. This therefore represents an important set of considera-
tions for future AI systems.
Types of supervision
Supervision is a way of checking and correcting the be-
haviour of a system through observation or interaction, and
hence it is crucial for safety. Supervision can be in the form
of corrected values for predictive models such as classifi-
cation or regression, but it can also be partial (the answer
is wrong, but the right answer is not given). Supervision can
also be much more subtle than this. For instance, a diagnosis
assistant that suggests a possible diagnosis to a doctor can be
designed to get no feedback once deployed. However, some
kinds of feedback can still reach the system in terms of the
distribution or frequency of tasks (questions), or through the
way the tasks are posed to the system.
Consequently there are several degrees and qualities of
supervision, and this may depend on the system. For in-
stance, in classification, one can have data for all examples
or just for a few (known as semi-supervised learning). In re-
inforcement learning, one can have sparse versus dense re-
ward. In general, supervision can come in many different
ways, according to some criteria:
•Completeness : Supervision can be very partial (sig-
nalling incorrectness), more informative (showing the
correct way) or complete (showing all positive and neg-
ative ways of behaving in the environment).
•Procedurality : Beyond what is right and wrong, feedback
can be limited about the result or can show the whole pro-
cess, as in the case of learning by demonstration.
•Density : Supervision can be sparse or dense. Of course
the denser the better (but more expensive), and the less
autonomous the system is considered.
•Adaptiveness : Supervision can be ‘intelligent’ as well,
which happens in machine teaching situations when ex-
amples or interactions are chosen such that the system
reaches the desired behaviour as soon as possible.
•Responsiveness : In areas such as query learning or active
learning, the system can ask questions or undertake ex-
periments at any time. The results can come in real time
or may have a delay or be given in batches.
For many systems, supervision can have a dedicated chan-
nel (e.g., rewards in RL) but for others it can be performed
by modification of the environment (e.g., moving objects or
smiling), even to the extent that the system is unaware these
changes have a guiding purpose (e.g., clues).
Types of modification
Some of the most recurrent issues in AI safety – including
many covered in the section about known AI safety charac-
teristics – are related to ways in which the system can be
modified. This includes issues such as wire-heading or algo-
rithmic self-improvement. Here, in the first place, we have
to distinguish between whether the system can be modified
by the environment, or by the system itself. Modifications
by the environment can be intentional (and hence related to
supervision), but they can also be unintentional (code cor-
ruption from external sources). Even a system whose core
code cannot be modified by an external source, may be af-
fected in state or code by regular inputs, physical equipmentand other parts. So it is better to explore different ways and
degrees to which a system can be modified externally:
•Interruptible: The system has a switch-off command or
modification option to switch it off.
•Parametric modification : Many systems are regulated or
calibrated with parameters or weights. When these param-
eters have a clear relation to the behaviour of a system
(e.g., an intelligent thermostat), this can be an effective,
bounded and simple way of modifying the system.
•Algorithmic modification : This can include new func-
tionalities, bug fixes, updates, etc. Many software issues
are caused, and are magnified, by these interventions.
Modifications can be limited in expressiveness, such as
only allowing rule deletion.
•Resource modification : Even if the parameters or code
are not modified, the resources of the system and other
dependencies previously mentioned can be limited exter-
nally, e.g., the computational resources.
On the other hand, systems can modify themselves (inter-
nally). There are many varieties here too:
1. No self-modification, no memory: The system has no
memory, and works as being reset for any new input or
interaction. Many functional systems (mapping inputs to
outputs) are of this kind. Note, however, that the environ-
ment does have memory, so some systems, such as a vi-
sion system or a non-cognitive robot, can be affected by
the past and become a truly cognitive system.
2. Partially self-modifying: The algorithms in the learner or
solver cannot be modified but its data or knowledge (in
the form of learned weights or rules) can be modified by
a general algorithm, which is fixed. Many learning sys-
tems are of this kind, if the system has both a learning
algorithm and one or more learned models.
3. Totally self-modifying: The system can modify any part
of its code. Not many operational systems have these abil-
ities, as they become very unstable. However, some types
of evolutionary computation may have this possibility, if
evolution can also be applied to the rules of the evolution.
Finally, all these categories can be selected for different pe-
riods of time. For instance, it is common to separate between
training, test/validation and deployment. For training, a high
degree of self-modification (and hence adaptation) is well
accepted, but then this is usually constrained for validation
and deployment. Note that these stages apply for both exter-
nal and internal sources of modification. One important dan-
ger is that a well-validated system may be subject to some
late external or internal modification just before deployment.
In this case, all the validation effort may become void2.
One of the major modern concerns in AI safety is that it
will be desirable for some systems to learn during deploy-
2OpenAI Dota is an example: https://
blog .openai .com/the-international-2018-
results/ ,https://www .theregister .co .uk/2018/
08/24/openai bots eliminated dota 2/
ment, in order for them to be adaptive3. For instance, many
personal assistants are learning from our actions continually.
While this may introduce many risks for more powerful sys-
tems, forbidding learning outside the lab would make many
potential applications of AI impossible. However, adaptive
systems are full of engineering problems; some must even
have a limited life, as after self-modification and adaptation
they may end up malfunctioning and have to be reset or have
their ‘caches’ erased. This problem has long been of interest
in engineering (Fickas and Feather 1995).
Types of motivation
Systems can follow a set of rules or aim at optimising a util-
ity function. Most systems are actually hybrid, as it is diffi-
cult to establish a crisp line between procedural algorithms
and optimisation algorithms. Through layers of abstraction
in these processes, we ultimately get the impression that a
system is more or less autonomous. If the system is appar-
ently pursuing a goal, what are the drivers that make a sys-
tem prefer or follow some behaviours over others? These
behaviours may be based on some kind of internal represen-
tation of a goal, as we discussed when dealing with antici-
pation, or on a metric of how close the system is to the goal.
Then the systems can follow an optimisation process that
tries to maximise some of these quality functions.
Quality or utility functions usually map inputs and out-
puts into some values that are re-evaluated periodically or
after certain events. Examples of these functions are accu-
racy, aggregated rewards or some kind of empowerment or
other types of intrinsic motivation (Klyubin, Polani, and Ne-
haniv 2005; Jung, Polani, and Stone 2011). The same system
might have several quality functions that can be opposed, so
trade-offs have to be chosen. The general notion of rational-
ity in decision-making is related to these motivations.
But what are the characteristics of the goals an AI system
can have in the first place? We outline several dimensions:
•Goal variability : Are goals hard-coded or change with
time? Do they change autonomously or through instruc-
tion? Who can change the goals and how? For instance,
what orders can a digital assistant take and from whom?
•Goal scrutability : Are the (sub)goals identifiable and ul-
timately explainable? Do they come from rules or are they
inferred from data, e.g., error in classification or observ-
ing humans in inverse reinforcement learning?
•Goal rationality : Are the goals amenable to treatment
within a rational choice framework? If several goals are
set, are they consistent? If not, how does the system re-
solve inconsistencies or set new goals?
Note that this is closely related to the types of modification,
as changing or resolving goals may require self-modification
and/or external modification.
3Nature has found many ways of regulating self-modification.
Many animals have a higher degree of plasticity at birth, becoming
more conservative and rigid in older stages (Gopnik et al. 2017).
One key question about cognition is whether this is a contingent or
necessary process, and whether it is influenced by safety issues.A second question is how these goals are followed by the
system. There are at least three possible dimensions here:
•Immediateness : The system may maximise the function
for the present time or in the limit, or something in be-
tween. Many schemata of discounted rewards in rein-
forcement learning are used as trade-offs between short-
term and long-term maximisation.
•Selfishness : Focusing on individual optima might involve
very bad collective results (for other agents) or even re-
sults that could even be worse individually (tragedy of the
commons). Game theory provides many examples of this.
In multi-agent RL systems, rewards can depend on the
well-being of other agents, or empathy can be introduced.
•Conscientiousness : The system may be fully committed
to maximising the goal, or some random or exploratory
actions are allowed, even if they deviate occasionally from
the goal. When it is on purpose, this is usually intended
to provide robustness or to avoid local minima, but these
deviations can take the system to dangerous areas.
Modulating optimisation functions to be convex with a non-
asymptotic maximum, beyond which further effort is futile,
may be a sensible thing as it provides a stop condition by
definition. A self-imposed cap can always be shifted if ev-
erything is under control once the limit is reached.
Note that the kind of interaction seen before is key for
the internal quality metric or goal. For instance, in asyn-
chronous RL, “the time can be intentionally modulated by
the agent” to get higher rewards without really performing
better (Hern ´andez-Orallo 2010). And, of course, a common
problem for motivation is reward hacking.
Types of achievement
Ultimately, an AI system is conceived to achieve a task, in-
dependently of how well motivated the system is for it. Con-
sequently, the external degree of achievement must be dis-
tinguished from the motivation or quality metric the system
uses to function, as discussed in the previous subsection.
The misalignment between the internal goal of the system
and the task specification is the cause of many safety issues
in AI, unlike formal methods in software engineering, when
requirements are converted into correct code.
Focusing on the task specification, we must first recognise
that different actors may have different interests. A cognitive
assistant, for instance, may be understood by the user as be-
ing very helpful, making life easier. However, for the com-
pany selling the cognitive assistant, the task is ultimately to
produce revenue with the product. Both requirements are not
always compatible and this may affect the definition of the
goals of the system, as some of the aims may not be coded
or motivated in a transparent way, but usually incorporated
in indirect ways. Second, even if the requirements include
all possible internalities (what the system has to do), there
are also many externalities and footprints (Mart ´ınez-Plumed
et al. 2018a) (including the infinitely many things that the
system should not do) that affect how positive or negative its
overall effect is. Regarding these two issues, task specifica-
tion can vary in precision and objectivity:
•Task precision : The evaluation metric to determine the
success of an agent can be formal or not. For instance, the
accuracy of a classifier or the squared error of a regression
model are precisely defined metrics. However, in many
other cases, we have a utility function that depends on
variables that are usually imprecise or uncertain, such as
the quality of a smart vacuum cleaner.
•Task objectivity : A metric can be objective or subjective.
We tend to associate precise metrics with objectiveness
and imprecise metrics with subjectivity, but subjectivity
simply means that the evaluation changes depending on
the subject. For instance, the quality of a spam filter (a
precisely-evaluated classifier) changes depending on the
cost matrices of different users, and the quality of a smart
vacuum cleaner based on fuzzy variables such as cleanli-
ness or disruption can be weighted by a fixed formula.
Some of the tasks or targets that are most commonly ad-
vocated in the ethics and safety of AI literature are often
very imprecise and subjective, such as “well-being”, “social
good”, “beneficial AI”, “alignment”, etc. Note that the prob-
lem is not related to the goals of the system (an inverse rein-
forcement learning system can successfully identify the dif-
ferent wills of a group of people), but rather about whether
the task is ultimately achieved, or the well-being or hap-
piness of the user. Determining this is controversial, even
when analysed in a scientific way (Alexandrova 2017).
An overemphasis on tracking metrics (Goodhart’s law) is
sometimes blamed, but the alternative is not usually better.
Some safety problems are not created by an overemphasis
on a metric (Manheim and Garrabrant 2018), but ultimately
by a metric that is too narrow or shortsighted, and does not
adequately capture progress towards the goal.
In all these cases, we have to distinguish whether the met-
ric relates to (i) the internal goals that the system should
have, (ii) the external evaluation of task performance, or (iii)
our ultimate desires and objective4. Motivations, achieve-
ment and supervision are closely related, but may be dif-
ferent. For a maze, e.g., the goal for the AI system may be
to get out of the maze as soon as possible, but a competition
could be based on minimising the cells that are stepped more
than once, and supervision may include indications of direc-
tion to the shortest route to the exit. These are three different
criteria which may be well or poorly aligned.
Even more comprehensively – and related to the concept
of persistence –, a system may be analysed for a range of
tasks, under different replicability situations:
1. Disposable system: single task, single use: The system is
used for one task that only takes place once.
2. Repetitive system: single task, several uses: The system
must solve many instances of the same specific task.
3. Menu system: multitask: The system must solve different
tasks, under a fixed repertoire of tasks.
4Ortega et al (2018) distinguish between “ideal specification
(the ‘wishes’)” and “design specification”, which must be com-
pared with the revealed specification (the “behaviour”). The design
specification fails to distinguish external metric from internal goal.4. General system: multitask: The system must solve differ-
ent tasks, without a fixed repertoire.
5. Incremental system: The system must solve a sequence of
tasks, with some dependencies between them.
Any metric examining the benefits and possible risks of a
system must take the factors described above into account.
Conclusion
Many accounts of AI safety focus on “either RL agents or
supervised learning systems” assuming “similar issues are
likely to arise for other kinds of AI systems” (Amodei et
al. 2016). This paper has surveyed a wide range of char-
acteristics of AI systems, so that future research can map
AI safety challenges against AI research paradigms in more
precise ways in order to ascertain whether particularly safety
challenges manifest similarly in different paradigms. This
aims to address an increasing concern that the current dom-
inant paradigm for a large proportion of AI safety research
may be too narrow: discrete-time RL systems with train/test
regimes, assuming gradient-based learning on a parametric
space, with a utility function that the system must optimise
(Gauthier 2018; Krakovna 2018).
Taxonomies of potentially safety-relevant characteristics
of AI systems, as introduced in this paper, are intended to
provide a good complement to recent work on taxonomies
of technical AI safety problems. For instance, Ortega (2018)
presents three main areas: specification , ensuring that an AI
system’s behaviour aligns with the operator’s true intentions;
robustness , ensuring that an AI system continues to operate
within safe limits upon perturbation, and assurance , ensur-
ing that we understand and control AI systems during oper-
ation. Almost all characteristics outlined in this paper have
a role to play for specification, robustness and assurance.
Taxonomies are rarely definitive, and the characterisation
presented here does not consider in full some quantitative
features such as performance, autonomy and generality. A
proper evaluation of how the kind and degree of intelligence
can affect safety issues is also an important area of analysis,
both theoretically (Hern ´andez-Orallo 2017) and experimen-
tally (Leike et al. 2017). AI research has explored different
paradigms in the past, and will continue to do so in the fu-
ture. Along the way, many different system characteristics
and design choices have been presented to developers. We
can expect even more to be developed as AI research pro-
gresses. Consequently, the area of AI safety must acquire
more structure and richness in how AI is characterised and
analysed, to provide tailored guidance for different contexts,
architectures and domains. There is a potential risk to over-
relying on our best current theories of AI when considering
AI safety. Instead, we aim to encourage a diverse set of per-
spectives, in order to anticipate and mitigate as many safety
concerns as possible.
Acknowledgments
FMP and JHO were supported by the EU (FEDER) and
the Spanish MINECO under grant TIN 2015-69175-C4-1-
R, by Generalitat Valenciana (GV A) under grant PROME-
TEOII/2015/013 and by the U.S. Air Force Office of Sci-
entific Research under award number FA9550-17-1-0287.
FMP was also supported by INCIBE (Ayudas para la
excelencia de los equipos de investigaci ´on avanzada en
ciberseguridad), the European Commission, JRC’s Cen-
tre for Advanced Studies, HUMAINT project (Expert
Contract CT-EX2018D335821-101), and UPV PAID-06-18
Ref. SP20180210. JHO was supported by a Salvador de
Madariaga grant (PRX17/00467) from the Spanish MECD
for a research stay at the Leverhulme Centre for the Fu-
ture of Intelligence (CFI), Cambridge, and a BEST grant
(BEST/2017/045) from GV A for another research stay also
at the CFI. JHO and SOH were supported by the Future
of Life Institute (FLI) grant RFP2-152. SOH was also sup-
ported by the Leverhulme Trust Research Centre Grant RC-
2015-067 awarded to the Leverhulme Centre for the Future
of Intelligence, and a a grant from Templeton World Charity
Foundation.
References
[Alexandrova 2017] Alexandrova, A. 2017. A Philosophy for
the Science of Well-being . Oxford University Press.
[Amodei et al. 2016] Amodei, D.; Olah, C.; Steinhardt, J.;
Christiano, P.; Schulman, J.; and Man ´e, D. 2016. Concrete
problems in ai safety. arXiv preprint arXiv:1606.06565 .
[Armstrong and Levinstein 2017] Armstrong, S., and Levin-
stein, B. 2017. Low impact artificial intelligences. arXiv
preprint arXiv:1705.10720 .
[Armstrong 2017] Armstrong, S. 2017. Good and safe uses of
ai oracles. arXiv preprint arXiv:1711.05541 .
[Babcock, Kram ´ar, and Yampolskiy 2016] Babcock, J.;
Kram ´ar, J.; and Yampolskiy, R. 2016. The AGI contain-
ment problem. In AGI Conf . Springer. 53–63.
[Barmpalias and Dowe 2012] Barmpalias, G., and Dowe, D. L.
2012. Universality probability of a prefix-free machine. Phil.
Trans. R. Soc. A 370(1971):3488–3511.
[Bernstein and Vazirani 1997] Bernstein, E., and Vazirani, U.
1997. Quantum complexity theory. SIAM Journal on comput-
ing26(5):1411–1473.
[Bonabeau et al. 1999] Bonabeau, E.; Dorigo, M.; Th ´eraulaz,
G.; and Theraulaz, G. 1999. Swarm intelligence: from natu-
ral to artificial systems . Oxford university press.
[Bostrom 2014] Bostrom, N. 2014. Superintelligence: Paths,
dangers, strategies . Oxford University Press.
[Brundage et al. 2018] Brundage, M.; Avin, S.; Clark, J.; Toner,
H.; Eckersley, P.; Garfinkel, B.; Dafoe, A.; Scharre, P.; Zeitzoff,
T.; Filar, B.; Anderson, H.; Roff, H.; Allen, G. C.; Steinhardt,
J.; Flynn, C.; ´O h´Eigeartaigh, S.; Beard, S.; Belfield, H.; Far-
quhar, S.; Lyle, C.; Crootof, R.; Evans, O.; Page, M.; Bryson,
J.; Yampolskiy, R.; and Amodei, D. 2018. The malicious use of
artificial intelligence: Forecasting, prevention, and mitigation.
arXiv preprint arXiv:1802.07228 .
[Coulouris, Dollimore, and Kindberg 2011] Coulouris, G. F.;
Dollimore, J.; and Kindberg, T. 2011. Distributed systems:
concepts and design . Fifth edition, Pearson.
[D’silva, Kroening, and Weissenbacher 2008] D’silva, V .;
Kroening, D.; and Weissenbacher, G. 2008. A survey of
automated techniques for formal software verification. IEEETransactions on Computer-Aided Design of Integrated Circuits
and Systems 27(7):1165–1178.
[Everitt and Hutter 2018] Everitt, T., and Hutter, M. 2018.
The alignment problem for bayesian history based reinforce-
ment learners. http://www .tomeveritt .se/papers/
alignment .pdf/ .
[Everitt, Lea, and Hutter 2018] Everitt, T.; Lea, G.; and Hutter,
M. 2018. Agi safety literature review. Proceedings of the
Twenty-Seventh International Joint Conference on Artificial In-
telligence (IJCAI-18), arXiv preprint version:1805.01109 .
[Fickas and Feather 1995] Fickas, S., and Feather, M. S. 1995.
Requirements monitoring in dynamic environments. In IEEE
Intl Symposium on Requirements Engineering , 140–147.
[Ford et al. 2015] Ford, K. M.; Hayes, P. J.; Glymour, C.; and
Allen, J. 2015. Cognitive orthoses: toward human-centered ai.
AI Magazine 36(4):5–8.
[Garrabrant and Demski 2018] Garrabrant, S., and Demski, A.
2018. Embedded agency. AI Alignment Forum.
[Gasparik, Gamble, and Gao 2018] Gasparik, A.; Gamble, C.;
and Gao, J. 2018. Safety-first ai for autonomous data centre
cooling and industrial control. DeepMind Blog.
[Gauthier 2018] Gauthier, J. 2018. Conceptual is-
sues in AI safety: the paradigmatic gap. http:
//www .foldl .me/2018/conceptual-issues-ai-
safety-paradigmatic-gap/ .
[Geffner 2018] Geffner, H. 2018. Model-free, model-based, and
general intelligence. arXiv preprint arXiv:1806.02308 .
[Gopnik et al. 2017] Gopnik, A.; OGrady, S.; Lucas, C. G.;
Griffiths, T. L.; Wente, A.; Bridgers, S.; Aboody, R.; Fung, H.;
and Dahl, R. E. 2017. Changes in cognitive flexibility and
hypothesis search across human life history from childhood to
adolescence to adulthood. PNAS 114(30):7892–7899.
[Hern ´andez-Orallo 2010] Hern ´andez-Orallo, J. 2010. On eval-
uating agent performance in a fixed period of time. In Artificial
General Intelligence, 3rd Intl Conf, ed., M. Hutter et al , 25–30.
[Hern ´andez-Orallo 2017] Hern ´andez-Orallo, J. 2017. The Mea-
sure of All Minds: Evaluating Natural and Artificial Intelli-
gence . Cambridge University Press.
[Jung, Polani, and Stone 2011] Jung, T.; Polani, D.; and Stone,
P. 2011. Empowerment for continuous agentenvironment sys-
tems. Adaptive Behavior 19(1):16–39.
[Klyubin, Polani, and Nehaniv 2005] Klyubin, A. S.; Polani,
D.; and Nehaniv, C. L. 2005. All else being equal be em-
powered. In European Conference on Artificial Life , 744–753.
[Krakovna 2018] Krakovna, V . 2018. Discussion
on the machine learning approach to AI safety.
http://vkrakovna .wordpress .com/2018/11/
01/discussion-on-the-machine-learning-
approach-to-ai-safety/ .
[Leike et al. 2017] Leike, J.; Martic, M.; Krakovna, V .; Ortega,
P. A.; Everitt, T.; Lefrancq, A.; Orseau, L.; and Legg, S. 2017.
AI safety gridworlds. arXiv preprint arXiv:1711.09883 .
[Manheim and Garrabrant 2018] Manheim, D., and Garrabrant,
S. 2018. Categorizing variants of Goodhart’s law. arXiv
preprint arXiv:1803.04585 .
[Mart ´ınez-Plumed et al. 2018a] Mart ´ınez-Plumed, F.; Avin, S.;
Brundage, M.; Dafoe, A.; h ´Eigeartaigh, S. ´O.; and Hern ´andez-
Orallo, J. 2018a. Accounting for the neglected dimensions of
ai progress. arXiv preprint arXiv:1806.00610 .
[Martınez-Plumed et al. 2018b] Martınez-Plumed, F.; Loe,
B. S.; Flach, P.; O hEigeartaigh, S.; V old, K.; and Hern ´andez-
Orallo, J. 2018b. The facets of artificial intelligence: A
framework to track the evolution of AI. IJCAI .
[Mnih 2015] Mnih, V . e. a. 2015. Human-level control through
deep reinforcement learning. Nature 518:529–533.
[Omohundro 2008] Omohundro, S. M. 2008. The basic ai
drives. Artificial General Intelligence 171:483–493.
[Ortega and Maini 2018] Ortega, P. A., and Maini, V .
2018. Building safe artificial intelligence: specification,
robustness, and assurance. https://medium .com/
@deepmindsafetyresearch/building-safe-
artificial-intelligence-52f5f75058f1 .
[Pecka and Svoboda 2014] Pecka, M., and Svoboda, T. 2014.
Safe exploration techniques for reinforcement learning–an
overview. In International Workshop on Modelling and Sim-
ulation for Autonomous Systems , 357–375. Springer.
[Quinn and Bederson 2011] Quinn, A. J., and Bederson, B. B.
2011. Human computation: a survey and taxonomy of a grow-
ing field. In SIGCHI conf. on human factors in computing sys-
tems, 1403–1412. ACM.
[Soares et al. 2015] Soares, N.; Fallenstein, B.; Armstrong, S.;
and Yudkowsky, E. 2015. Corrigibility. In Workshops at the
Twenty-Ninth AAAI Conference on Artificial Intelligence .
[Turchetta, Berkenkamp, and Krause 2016] Turchetta, M.;
Berkenkamp, F.; and Krause, A. 2016. Safe exploration in
finite Markov decision processes with Gaussian processes. In
NIPS , 4312–4320.
[Yampolskiy 2012] Yampolskiy, R. 2012. Leakproofing the sin-
gularity artificial intelligence confinement problem. Journal of
Consciousness Studies 19(1-2):194–214.
[Yampolskiy 2016] Yampolskiy, R. V . 2016. Taxonomy of path-
ways to dangerous artificial intelligence. In AAAI Workshop:
AI, Ethics, and Society .

Jose Hernandez-Orallo
Fernando Martınez-Plumed
Shahar Avin


Safe Exploration Techniques for Reinforcement Learning – An Overview

Lecture Notes in Computer Science: Safe
Exploration Techniques for Reinforcement
Learning { An Overview
Martin Pecka1?, and Tomas Svoboda1;2??
1Center for Machine Perception, Dept. of Cybernetics, Faculty of Electrical
Engineering, Czech Technical University in Prague, Prague, Czech Republic,
martin.pecka@fel.cvut.cz ,http://cmp.felk.cvut.cz/ peckama2
2Czech Institute of Informatics, Robotics, and Cybernetics, Czech Technical
University in Prague, Prague, Czech Republic
Abstract. We overview dierent approaches to safety in (semi)auto-
nomous robotics. Particularly, we focus on how to achieve safe behavior
of a robot if it is requested to perform exploration of unknown states. Pre-
sented methods are studied from the viewpoint of reinforcement learning ,
a partially-supervised machine learning method. To collect training data
for this algorithm, the robot is required to freely explore the state space
{ which can lead to possibly dangerous situations. The role of safe explo-
ration is to provide a framework allowing exploration while preserving
safety. The examined methods range from simple algorithms to sophis-
ticated methods based on previous experience or state prediction. Our
overview also addresses the issues of how to dene safety in the real-world
applications (apparently absolute safety is unachievable in the continu-
ous and random real world). In the conclusion we also suggest several
ways that are worth researching more thoroughly.
Keywords: Safe exploration, policy search, reinforcement learning
1 Introduction
Reinforcement learning (RL) as a machine learning method has been thoroughly
examined since 80's. In 1981, Sutton and Barto [3] inspired themselves in the
reinforcement learning discoveries in behavioral psychology and devised the Tem-
poral Dierence machine learning algorithm that had to simulate psychological
classical conditioning. In contrast with supervised learning , reinforcement learn-
ing does not need a teacher's classication for every sample presented. Instead, it
just collects rewards (or punishment) on-the-go and optimizes for the expected
?Supported by the Grant Agency of the Czech Technical University in Prague under
Project SGS13/142/OHK3/2T/13
??Supported by the EC project FP7-ICT-609763 TRADR
2 LNCS: Safe Exploration for Reinforcement Learning
long-term reward (whereas supervised learning optimizes for the immediate re-
ward). The key advantage is that the design of the rewards is often much simpler
and straight-forward than classifying all data samples.
Reinforcement learning proved to be extremely useful in the case of state-
space exploration { the long-term reward corresponds to the value of each
state [17]. From such values, we can compose a policy which tells the agent
to always take the action leading to the state with the highest value. As an
addition, state values are easily interpretable for humans.
Since the early years, a lot of advanced methods were devised in the area of
reinforcement learning. To name one, Q-learning [25] is often used in connection
with safe exploration. Instead of computing the values of states, it computes
the values of state{action pairs, which has some simplifying consequences. For
example, Q-learning doesn't need any transition model (i.e. dynamics model) of
the examined system.
A completely dierent approach is policy iteration . This algorithm starts with
a (more or less random) policy and tries to improve it step-by-step [16]. This
case is very valuable if there already exists a good policy and we only want to
improve it [11].
What do all of these methods have in common, is the need for rather large
training data sets. For simulated environments it is usually not a problem. But
with real robotic hardware, the collection of training samples is not only lengthy,
but also dangerous (be it mechanical wear or other eects). Another common
feature of RL algorithms is the need to enter unknown states, which is inherently
unsafe.
As can be seen from the previous paragraph, safety is an important issue
connected with reinforcement learning. However, the rst articles focused on
maintaining safety during exploration started to appear much later after the \dis-
covery" of RL. Among the rst, Heger [15] \borrowed" the concept of a worst-
case criterion from control theory community. In 1994 he created a variant of
Q-learning where maximization of long-term reward is replaced with maximiza-
tion of minimum of the possible rewards. That basically means his algorithm
prefers to never encounter a bad state (or, at least to choose the best of the bad
states). This approach has one substantial drawback { the resulting policies are
far from being optimal in the long-term{reward sense [10].
In this paper we show the various approaches to safe exploration that have
emerged so far. We classify the methods by various criteria and suggest suitable
use cases for them. To better illustrate some of the practical details, we use
the UGV (Unmanned Ground Vehicle) robotic platform from EU FP7 project
NIFTi [6] (see Figure 1) as a reference agent. It may happen that in these
practical details we assume some advantages of UGVs over UAVs (Unmanned
Aerial Vehicles), like the ability to stand still without much eort, but it is
mostly easy to convert these assumptions to UAVs, too.
Further organization of this paper is the following: in Section 2 we discuss
some basics of reinforcement learning (the reader may skip it if he is familiar
with reinforcement learning); Section 3 is an overview of the safety denitions
LNCS: Safe Exploration for Reinforcement Learning 3
Fig. 1. NIFTi UGV robotic platform
used in literature; Section 4 is the main part concerning the various approaches
to safe exploration, and in Section 5 we conclude the ndings and we suggest
some further areas of possible research.
2 Reinforcement learning basics
2.1 Markov Decision Processes
Markov Decision Processes (MDPs) are the standard model for deliberating
about reinforcement learning problems. They provide a lot of simplications,
but are suciently robust to describe a large set of real-world problems.
The simplest discrete stochastic MDP comprises of: [17]
{a nite set of states S
{a nite set of actions A
{a stochastic transition model P:Pt(s;a;s0) =Pr(st+1=s0jst=s; at=a)
for eachs;s02S; a2A, wherePrstands for probability
{and the immediate reward function R:SA!R(orR:SAS!R
if the reward depends on the stochastic action result)
4 LNCS: Safe Exploration for Reinforcement Learning
To interpret this denition, we say that the at every time instant ttheagent is
in a states, and by executing action ait gets to a new state s0. Furthermore,
executing a particular action in a particular state may bring a reward to the
agent (dened by R).
The most important and interesting property of MDPs is the Markov prop-
erty. If you have a look at the denition of the transition model, the next state
only depends on the current state and the chosen action. Particularly, the next
state is independent of all the previous states and actions but the current one.
To give an example, the robot's battery level cannot be treated implicitly by
counting the elapsed time, but rather it has to be modeled as a part of the
robot's state.
Once the model is set up, everything is ready for utilizing an MDP. \The
agent's job is to nd a policy mapping states to actions, that maximizes
some long-run measure of reinforcement" [17]. The \long-run" may have dierent
meanings, but there are two favorite optimality models: the rst one is the nite
horizon model, where the term J=Ph
t=0rtis maximized ( his a predened time
horizon and rtis the reward obtained in time instant twhile executing policy ).
The dependency of rton the policy is no longer obvious from this notation, but
this is the convention used in literature when it is clear which policy is used. This
model represents the behavior of the robot which only depends on a predened
number of future states and actions.
The other optimality model is called discounted innite horizon , which means
we maximize the discounted sum J=P1
t=0
trtwith
2(0;1) being the
discount factor . The innite horizon tries to nd a policy that is the best one
taking into account the whole future. Please note the hidden dependency on the
policy(and the starting state s0) { it is the policy that decides on which action
to take, which in turn species what will the reward be.
Other extensions of MDPs to continuous states, time or actions are beyond
the scope of this overview. However, some of the referenced papers make use of
these continuous extensions, which proved to be useful for practical applications.
2.2 Value iteration
Value iteration is one of the basic methods for nding the optimal policy. To
describe this algorithm, it is rst needed to dene the essential notion of the
optimal value of a state . In this whole subsection we suppose the discounted
innite horizon model, but analogous results can be shown for nite horizon,
too. \The optimal value of a state is the expected innite discounted sum of
reward that the agent will gain if it starts in that state and executes the optimal
policy." [17] Given a policy , the induced value function is therefore dened as
V(s) =E"1X
t=0rk
k#
; (1)
whereEdenotes the expected value and rkare the rewards for executing policy
. Taking the best value function over all policies then yields the optimal value
LNCS: Safe Exploration for Reinforcement Learning 5
function V: [17]
V(s) = max
V(s): (2)
Inversely, if we have the value function given, we can derive a policy from
that. It is a simple policy that always takes the action leading to the most
protable neighbor state (with the highest value).
One useful formulation of the properties of the optimal value function is
the formulation using the recurrent Bellman equations which dene a dynamic
system that is stable for the optimal value function. We can say a state's optimal
value is the best immediate reward plus its best neighbor's optimal value: [17]
V(s) = max
a
R(s;a) +
X
s02SP(s;a;s0)V(s0)!
: (3)
Analogously, we can nd the optimal policy using the same Bellman equation:
(s) = argmax
a
R(s;a) +
X
s02SP(s;a;s0)V(s0)!
: (4)
The Value iteration algorithm is based on trying to compute the solution
of Equation 4 using iterative Bellman updates (refer to Algorithm 1). In the
algorithm, we use a structure called Qto store the \value" of state-action pairs.
In Value iteration it is just a structure to save intermediate results, but it is the
core of the Q-learning algorithm (described in Section 2.3). The stopping crite-
rion of the Value iteration algorithm is not obvious, but Williams and Baird [26]
derived an easily applicable upper bound on the error of the computed value
function.
That said, after a sucient number of those simple iterations, we can compute
the almost optimal value function. The number of iterations needed for Value
iteration to converge may be impractically high, but it is shown that the optimal
policy converges faster [4], thus making Value iteration practical.
2.3 Q-learning
Just a small change to the Value iteration algorithm results in Q-learning. The
basic algorithm is the same as Value iteration, just the update step is done
dierently (refer to Algorithm 2). The consequence of this change is that no
model of the system (transition function P) is needed. It is sucient to execute
all actions in all states equally often, and Watkins [25] proved that if Q-learning
were run for an innite time, the computed Qwould converge to the optimal
Q(an analogue of V).
2.4 Policy iteration
Policy iteration is a completely dierent approach to computing the optimal pol-
icy. Instead of deriving the policy from the Value or Q function, Policy iteration
6 LNCS: Safe Exploration for Reinforcement Learning
Algorithm 1 The Value iteration algorithm [17]
Input: an MDP (states S, actions A, rewards R, transition model P)
Output: the optimal value function V, resp. the optimal policy
derived from the value function
1.V(s) := arbitrary function
2.:=the policy derived from V
3. whileis not good enough do
4. for all s2Sdo
5. for all a2Ado
Update:
6. Q(s;a) :=R(s;a) +
P
s02SP(s;a;s0)V(s0)
7. end for
8. V(s) := max
aQ(s;a)
9. end for
10.:=the policy derived from V
11. end while
12.V:=V,:=
Algorithm 2 The Q-learning algorithm (only the parts that dier from Value
iteration when Vis substituted with Q) [17]
Input: an MDP (states S, actions A, rewards R, transition model may be
unknown)
Output: the optimal state-value function Q, resp. the optimal policy
derived from the state-value function
6.Q(s;a) :=Q(s;a)+
h
R(s;a) +
max
a0Q(s0;a0)Q(s;a)i
8. line left out
LNCS: Safe Exploration for Reinforcement Learning 7
works directly with policies. In the rst step, a random policy is chosen. Then
a loop consisting of policy evaluation and policy improvement repeats as long as
the policy can be improved [17] (refer to Algorithm 3 for details). Since in every
step the policy gets better, and there is a nite number of dierent policies, it
is apparent that the algorithm converges [23].
Policy iteration can be initialized by a known, but suboptimal policy. Such
policy can be obtained e.g. by a human operator driving the UGV. If the initial
policy is good, Policy iteration has to search much smaller subspace and thus
should converge more quickly than with a random initial policy [11].
Algorithm 3 The Policy iteration algorithm [17]
1.0=arbitrary policy
2. repeat
3.:=0
Policy evaluation: (system of linear equations)
4. V(s) =R(s;(s)) +
P
s02SP(s;(s);s0)V(s0)
Policy improvement:
5.0(s) := argmax
a2Ah
R(s;a) +
X
s02SP(s;a;s0)V(s0)i
6. until=0
3 Dening safety
To examine the problems of safe exploration, it is rst needed to dene what
exactly is the safety we want to maintain. Unfortunately, there is no unied
denition that would satisfy all use cases; thus, several dierent approaches are
found in the literature. An intuitive (but vague) denition could be e.g.: \State-
space exploration is considered safeif it doesn't lead the agent to unrecoverable
and unwanted states." It is worth noticing here that unwanted doesn't necessarily
mean low-reward. In the next subsections we present the main interpretations
of this vague denition.
3.1 Safety through labeling
The largely most used denition of safety is labeling the states/actions with one
of several labels indicating the level of safety in that state/action. What varies
from author to author is the number and names of these labels.
To start with, Hans [14] has the most granular division of state/action space.
His denitions are as follows (slightly reformulated):
8 LNCS: Safe Exploration for Reinforcement Learning
{an (s,a,r,s0) tuple (transition) is fatal if the reward ris less than a certain
threshold (sis the original state, ais an action and s0is the state obtained
after executing ain states, yielding the reward r),
{an actionaisfatal in statesif there is non-zero probability of leading to
a fatal transition,
{statesis called supercritical if there exists no policy that would guarantee
no fatal transition occurs when the agent starts in state s,
{actionaissupercritical in statesif it can lead to a supercritical state,
{statesis called critical if there is a supercritical or fatal action in that state
(and the state itself is not supercritical),
{actionaiscritical in statesif it leads to a critical state (and the action
itself is neither supercritical nor fatal in s),
{statesis called safe if it is neither critical nor supercritical,
{actionaissafe in statesif it is neither critical, nor supercritical, nor fatal
in states,
{and nally a policy is safe if for all critical states it leads to a safe state in
a nite number of non-fatal transitions (and if it only executes safe actions
in safe states).
Since we will compare other denitions the the Hans', it is needed to dene
one more category. A state sis called fatal if it is an undesired or unrecoverable
state, e.g. if the robot is considered broken in that state. The fatal transition
can then be redened as a transition ending in a fatal state. Opposite to the
precisely dened terms in Hans' denition, the meaning of words \undesired"
and \unrecoverable" here is vague and strongly task-dependent.
Continuing on, Geibel [12] denes only two categories { fatal andgoalstates.
\Fatal states are terminal states. This means, that the existence of the agent ends
when it reaches a fatal state" [12]. This roughly corresponds to our dened set of
fatal states. Goal states are the rest of nal states that correspond to successful
termination. Since Geibel only considers terminal states for safety, his goalstates
correspond to a subset of safe states. The other Hans' categories need not be
represented, since they are meaningless for nal states.
An extension of Geibel's fatal and goal states is a division presented by
Garc a [10]. His error and non-error states correspond to fatal and goal states,
but Garc a adds another division of the space { the known and unknown states,
where known states are those already visited (and known have empty intersection
with error ). He then mentions a prerequisite on the MDP that if an action leads
to a known error /non-error state, then its slight modication must also lead to
anerror /non-error state (a metric over the state space is required).
In Ertle's work [9], again the two basic regions are considered { they are
called desired and hazardous (corresponding to safe andfatal ). However, due
to the used learning technique, one more region emerges { the undesired region.
It contains the whole hazardous region and a \small span" comprising of desired
states, and denotes the set of states where no training (safe) samples are avail-
able, because it would be dangerous to acquire those samples. In particular, he
says that \The hazards must be `encircled' by the indications of the undesired
approaching so that it becomes clear which area [. . . ] is undesired" [9].
LNCS: Safe Exploration for Reinforcement Learning 9
A summary of the labeling-based denitions is shown in Figure 3. We ex-
amined the apparent imbalance between the number of categories Hans denes,
and the other denitions, and that led us to the following observations.
The rst observation is that creating labels for actions or transitions is un-
necessary. If we need to talk about the \level of safety" of an action, we can
use the worst label out of all possible results of that action (which retains com-
patibility with Hans' denitions). Moreover, as \it is impossible to completely
avoid error states" [22], we can ignore the eects of the action which have only
small probability (lower than a safety threshold) { we will call such eects the
negligible eects .
A second remark is that the fatal andsupercritical sets can be merged.
In Hans' work we haven't found any situation where distinguishing between su-
percritical andfatal would bring any benet. Specically, in his work Hans
states that: \Our objective is to never observe supercritical states" [14], which
eectively involves avoiding fatal transitions, too. And since we avoid both su-
percritical and fatal, we can as well avoid their union.
Third, safety of a state does not necessarily depend on the reward for getting
to that state. E.g. when the UGV performs a victim detection task, going away
from the target area may be perfectly safe, but the reward for such action should
be small or even negative.
Putting these observations together, we propose a novelty denition of safety
for stochastic MDPs, which is a simplication of Hans' model and a generaliza-
tion of the other models:
{A state is unsafe if it means the agent is damaged/destroyed/stuck. . . or it
is highly probable that it will get to such state regardless of further actions
taken.
{A state is critical if there is a not negligible action leading to an unsafe
state from it.
{A state is safe if no available action leads to an unsafe state (however, there
may be an action leading to a critical state).
To illustrate the denition on a real example, please refer to Figure 2. In 2(a),
the UGV is in a safe state, because all actions it can take lead again to safe
states (supposing that actions for movement do not move the robot for more
than a few centimeters). On the other hand, the robot as depicted in 2(b) is
in acritical state, because going forward would make the robot fall over and
break. If the robot executed action \go forward" once more, it would come to
anunsafe state. Right after executing the action it would still not be broken;
however, it would start falling and that is unsafe , because it is not equipped to
withstand such fall and therefore it is almost sure it will break when it meets
the ground.
3.2 Safety through ergodicity
An MDP is called ergodic i for every state there exists a policy that gets the
agent to any other state [20]. In other words, every mistake can be remedied in
10 LNCS: Safe Exploration for Reinforcement Learning
(a) A safe state.
(b) A critical state { if the robot went still forward, it would fall down and probably
break.
Fig. 2. An illustration of safe and critical states.
LNCS: Safe Exploration for Reinforcement Learning 11
FATAL
SUPERCRITICAL
CRITICAL
SAFE
FATAL
ERROR
HAZAR-GOAL
NON-ERROR
UNDESIREDDened by Hans Hans
Geibel
Garc a
ErtleACTIONSSTATES
DOUS
SAFECRI-
TICALUNSAFEDESIRED
Our
Fig. 3. A summary of the denitions of safety. The basic division is taken from Hans
[14] and fatal states are added. States are drawn with solid background and white-
headed arrows ( \_) denote the possible actions in the states. Actions are rendered with
striped background and black-headed arrows ( ) end in states where it is possible to
end up using the action.
12 LNCS: Safe Exploration for Reinforcement Learning
such MDP. Moldovan [20] then denes -safe policies as policies guaranteeing
that from any state the agent can get to the starting state with probability at
least(using a return policy , which is dierent from the -safe one). Stated
this way, the safety constraint may seem intractable, or at least impractical {
it is even proved, that expressing the set of -safe policies is NP-hard [20]. An
approximation of the constraint can be expressed in the terms of two other MDP
problems which are easily solved [20]; that still leads to -safe policies, but the
exploration performance may be suboptimal.
In our view, safety through ergodicity imposes too much constraints on the
problems the agent can learn. It sometimes happens that a robot has to learn
some task after which it is not able to return to the initial state (e.g. drive down
a hill it cannot go upwards; a human operator then carries the robot back to the
starting position). But the inability to \return home" in no means indicates the
robot is in an unsafe state.
3.3 Safety through costs
Another denition of safety is to dene a cost for taking an action/being in
a state and minimize the worst-case cost of the generated policies (up to some
failure probability). Such approach is presented in [15].
However, unless a threshold is set, this denition leads only to the safest pos-
sible policies, which are not necessarily safe. Expressing the safety using costs
is natural for some RL tasks (e.g. when learning the function of a dynamic
controller of an engine, the engine's temperature can be treated as a cost). Un-
fortunately, not all unsafe states can be described using such costs in general.
In addition, specifying the right costs may be a dicult task.
3.4 Safety as variance of the expected return
An alternative to safety as minimization of a cost (either worst-case or expected)
is minimizing both the cost and its variance. This approach is called expected
value-variance criterion [15] and is used mainly in works prior 2000, e.g. [7].
A safe policy by this criterion can be viewed as a policy that minimizes the
number of critical actions (because fatal transitions are expected to yield much
larger costs than safe transitions, increasing the variance signicantly).
As stated in [10], the worst-case approach is too restrictive and cautious.
The other expected value-variance criteria suer from the same disadvantages
as safety through costs { mainly from the general diculty to tune up the costs.
4 Safe exploration approaches
Finally, when the theoretical concepts have been shown and the various safety
denitions have been presented, we can focus on the main part of this overview.
Our categorization of safe exploration techniques is based on the work of Garc a
[10]. The basic division is as follows: approaches utilizing the expected return
LNCS: Safe Exploration for Reinforcement Learning 13
or its variance (Sec. 4.1), labeling-based approaches (Sec. 4.2) and approaches
beneting from prior knowledge (Sec. 4.3).
4.1 Optimal control approaches
Techniques in this category utilize variations of the expected value-variance safety
criterion. The most basic one is treating the rewards as costs (when a reward is
denoted by rt, the corresponding cost is denoted by ct). Standard RL methods
can then be used to solve the safe exploration task, as described e.g. in [7] for
discounted innite horizon.
The RL objective function
J=E"1X
t=0
tct#
(5)
is called the risk-neutral objective . To make this objective risk-sensitive , we spec-
ify a risk factor and rewrite the objective as: [15]
J=1
logE[exp (
tP1
t=0ct)] (6)
'E[P1
t=0
tct] +
2Var[P1
t=0
tct];
which is also called the expected value-variance criterion . This approach is a part
of theory using exponential utility functions , which is popular in optimal con-
trol [19]. To complete this section, the worst-case objective function (also called
theminimax objective ) is dened as
J= sup"1X
t=0
tct#
: (7)
As can be seen, the objective functions containing expectations cannot in fact
assure that no unsafe state will be encountered. On the other hand, the minimax
objective provides absolute certainty of the safety. However, it may happen that
some of the unsafe states can only be reached with a negligible probability. In
such cases, the -value criterion dened by [15] can be used { it only takes
into account rewards that can be reached with probability greater than . In
the work of Mihatsch [19], a scheme is presented that allows to \interpolate"
between risk-neutral and worst-case behavior by changing a single parameter.
Delage's work [8] takes into account the uncertainty of parameters of the
MDP. It is often the case that the parameters of the MDP are only estimated
from a limited number of samples, causing the parameter uncertainty. He then
proposes a possibility that the agent may \invest" some cost to lower the uncer-
tainty in the parameters (by receiving some observations from other sources than
exploration). A completely new research area then appears { to decide whether
it is more valuable to pay the cost for observations, or to perform exploration
by itself.
14 LNCS: Safe Exploration for Reinforcement Learning
An approximation scheme for dealing with transition matrix uncertainty is
presented in [21]. It considers a robust MDP problem and provides a worst-case,
but also robust policy (with respect to the transition matrix uncertainty).
A theory generalizing these approaches can be found in [24]. The theory states
that the optimal control decision is based on three terms { the deterministic ,
cautionary and probing terms.
The deterministic term assumes the model is perfect and attempts
to control for the best performance. Clearly, this may lead to disaster
if the model is inaccurate. Adding a cautionary term yields a controller
that considers the uncertainty in the model and chooses a control for
the best expected performance. Finally, if the system learns while it
is operating, there may be some benet to choosing controls that are
suboptimal and/or risky in order to obtain better data for the model
and ultimately achieve better long-term performance. The addition of
the probing term does this and gives a controller that yields the best
long-term performance.[24]
To conclude this section, we think that these methods are not well suited for
safe exploration { the expected value-variance and similar criteria provide no
warranties on the actual safety. On the other hand, the worst-case approaches
seem to be too strict.
4.2 Labeling-based approaches
The approaches utilizing some kind of state/action labeling (refer to Section 3.1
for the various labeling types) usually make use of two basic components { a risk
function and a backup policy . The task of the safety function is to estimate the
safety of a state or action. In the simplest case, the safety function can just
provide the labeling of the given action; or it can return a likelihood that the
action is safe; and in the best case, it would answer with a likelihood to be safe
plus a variance (certainty) of its answer. The backup policy is a policy that is
able to lead the agent out of the critical states back to the safe area. It is not
obvious how to get such a policy, but the authors show some ways how to get
one.
In the work of Hans [14], the most granular labeling is used, where fatal
transitions are said to be the transitions with reward less than a given threshold.
The safety function is learned during the exploration by collecting the so-called
min-reward samples { this is the minimum reward ever obtained for executing
a particular action in a particular state. The backup policy is then told to either
exist naturally (e.g. a known safe, but suboptimal controller), or it can also be
learned. To learn the backup policy, an RL task with altered Bellman equations
is used:
Q
min(s;a) = max
s0minh
R(s;a;s0);max
a0Q
min(s0;a0)i
:
A policy derived from the computed Q
minfunction is then taken as the backup
policy (as it maximizes the minimum reward obtained, and the fatal transitions
LNCS: Safe Exploration for Reinforcement Learning 15
are dened by low reward). He denes a policy to be safe, if it executes only safe
actions in safe states and produces non-fatal transitions in critical states. To learn
such safe policy, he then suggests a level-based exploration scheme (although he
gives no proofs why it should be better than any other exploration scheme).
This scheme is based on the idea that it is better to be always near the known
safe space when exploring. All unknown actions from one \level" are explored,
and their resulting states are queued to the next \level". For exploration of
unknown actions he proposes that the action should be considered critical until
proved otherwise, so the exploration scheme uses the backup policy after every
unknown action execution. A disadvantage of this approach is that the agent
needs some kind of \path planning" to be able to get to the queued states and
continue exploration from them.
Garc a's PI-SRL algorithm [10] is a way to safeguard the classical policy
iteration algorithm. Since the labels error /non-error are only for nal states,
the risk function here is extended by a so called Case-based memory , which is
in short a constant-sized memory for storing the historical ( s,a,V(s)) samples
and is able to nd nearest neighbors for a given query (using e.g. the Euclidean
distance). In addition to the error andnon-error states, he adds the denition of
known and unknown states, where known states are those that have a neighbor
in the case-based memory closer than a threshold. A safe policy is then said to be
a policy that always leads to known non-error nal states. To nd such policy,
the policy iteration is initialized with the safe backup policy and exploration is
done via adding a small amount of Gaussian noise to the actions. This approach
is suitable for continuous state- and action-spaces.
Another approach is presented in the work of Geibel [12], where the risk and
objective functions are treated separately. So the risk function only classies the
states (again only nal states) as either fatal orgoal, and the risk of a policy
(risk function) is then computed as the expected risk following the policy (where
fatal states have risk 1 and goal states have risk 0). The task is then said to be
to maximize the objective function (e.g. discounted innite horizon) w.r.t. the
condition that the risk of the considered policies is less than a safety threshold.
The optimization itself is done using modied Q-learning, and the optimized ob-
jective function is a linear combination of the original objective function and the
risk function. By changing the weights in the linear combination the algorithm
can be controlled to behave more safely or in a more risk-neutral way.
A generalization of Geibel's idea to take the risk and reward functions sep-
arately can be found in the work of Kim [18]. In this work, the constrained RL
task is treated as a Constrained MDP and the algorithm CBEETLE for solving
the Constrained MDPs is shown. The advantage of this work is that it allows
for several independent risk (cost) functions and doesn't need to convert them
to the same scale.
A similar approach of using constrained MDP to solve the problem can be
found in the work of Moldovan [20]. He does, however, use the ergodicity condi-
tion to tell safe and unsafe states apart (that is, safe are only those states from
which the agent can get back to the initial state). Moreover, this approach is only
16 LNCS: Safe Exploration for Reinforcement Learning
shown to work for toy examples like the grid world with only several thousands
of discrete states, which may not be sucient for real robotics tasks.
The idea of having several risk functions is further developed by Ertle [9].
The agent is told to have several behaviors and a separate safety function is
learned for each behavior. This approach allows for modularity and sharing of
the learned safety functions among dierent types of agents. More details on this
work will be provided in the next section, because it belongs to learning with
teachers.
An approach slightly dierent from the previously mentioned in this section
is using the methods of reachability analysis to solve safe exploration. Gillula
in his work [13] denes a set of keep-out states (corresponding to unsafe in our
labeling) and then a set called Pre() is dened as a set of all states from which
it is possible to get to a keep-out state in less than steps. Reachability analysis
is used to compute the Pre() set. Safe states are then all states not in Pre()
for a desired . This approach, however, doesn't utilize reinforcement learning,
it computes the optimal policy using standard supervised learning methods with
one additional constraint { that the system must use safe actions near the Pre()
set. On the other hand, the system is free to use whatever action desired when
it is not near Pre().
As was presented in this section, the labeling-based approaches provide a num-
ber of dierent ways to reach safety in exploration. They are, however, limited in
several ways { some of them make use of the (usually hard-to-obtain) transition
matrix, the others may need to visit the unsafe states in order to learn how to
avoid them, or need the state-space to be metric.
4.3 Approaches beneting from prior knowledge
The last large group of safe exploration techniques are the ones beneting from
various kinds of prior knowledge (other than the parameters of the MDP). We
consider this group the most promising for safe exploration, because \it is impos-
sible to avoid undesirable situations in high-risk environments without a certain
amount of prior knowledge about the task"[10].
The rst option how to incorporate prior knowledge into exploration is to
initialize the search using the prior knowledge. In fact, several works already
mentioned in previous sections use prior knowledge { namely the approaches
with a backup policy (Hans [14], Garc a [10]). Also, Garc a suggests that the
initial estimate of the value function can be done by providing prior knowledge,
which results in much faster convergence (since the agent does no more have to
explore really random actions, the estimate of the value function already \leads
it" the right way) [10].
Another option how to incorporate prior knowledge is by using Learning from
Demonstration (LfD) methods. Due to the limited space, we will not give the
basics of LfD { a good overview of the state-of-the-art methods is for example
in [2]. For our overview, it is sucient to state that LfD methods can derive
a policy from a set of demonstrations provided by a teacher. What is important,
is that the teacher does not necessarily have to have the same geometrical and
LNCS: Safe Exploration for Reinforcement Learning 17
physical properties as the trainee (although it helps the process if possible). It is
therefore possible to use LfD to teach a 5-joint arm to play tennis, while using
3-joint human arm as the source of demonstrations (but the learned policy may
be suboptimal; RL should then be used to optimize the policy).
InApprenticeship Learning [1], the reward function is learned using LfD.
The human pilot
ies a helicopter at his best, and both system dynamics and
the reward function are learned from the demonstrations. It is however apparent
that the performance of the agent is no longer objectively optimal, but that it
depends on the abilities of the human pilot.
Another way of incorporating prior knowledge into the learning process is
to manually select which demonstrations will be provided, as in the work of
Ertle [9]. In the work it is suggested that more teacher demonstrations should
come from the areas near the unsafe set, in order to teach the agent precisely
where the border between safe and unsafe is located.
The last technique described in our overview is interleaving autonomous ex-
ploration with teacher demonstrations. As in the previous case, some teacher
demonstrations are provided in advance, and then the exploration part starts
utilizing the teacher-provided information. After some time, or in states very
dierent from all other known states, the agent requests the teacher to provide
more examples [2,5]. The idea behind this algorithm is that it is impossible to
think out in advance what all demonstrations will the agent need in order to
learn the optimal policy.
Finishing this section, the algorithms utilizing prior knowledge seem to be the
most promising out of all the presented approaches. They provide both a speedup
of the learning process (by discarding the low-reward areas) and a reasonable
way to specify the safety conditions (via LfD or interleaving).
5 Conclusion
In our work we have given a short introduction on the basics of Markov Decision
Processes as well as the basic Reinforcement Learning methods like Value Iter-
ation ,Q-learning and Policy Iteration . In Section 3 we have summarized many
recent approaches on how to dene safety in the framework of optimal control
and reinforcement learning. We have also proposed a novelty denition of safety,
which divides the state space to safe,critical and unsafe states. We have shown
that all other labeling-based safety denitions are covered by our new denition.
In Section 4 many dierent safe exploration methods are categorized into
three basic groups { algorithms from optimal control theory, reinforcement learn-
ing algorithms based on state labeling, and algorithms utilizing extra prior
knowledge. We have shortly summarized the advantages and disadvantages of the
particular approaches. We have also stated that at least for dicult real-world
problems, safe exploration without prior knowledge is practically impossible,
and prior knowledge almost always helps to achieve faster convergence. Another
observation has been that some of the safe exploration algorithms need to visit
18 LNCS: Safe Exploration for Reinforcement Learning
unsafe states to correctly classify them later, which might discard them from
some usage scenarios where the unsafe states are really fatal.
It seems to us that the eld of safe exploration in reinforcement learning has
been very fragmented and lacks an all-embracing theory. However, the question
is, if it is even possible to nd such theory { the main problem may be the
fragmentation and dierences of various RL methods themselves. At least, the
safe exploration community would benet from a unication of the terminology
(and our proposal of the novelty safety labeling would like to help that).
Other ways of possible future research are for example the following. New
ways of incorporating prior knowledge into methods not utilizing it yet could
bring interesting speed-up of those algorithms. There is also a bottleneck in the
estimation of the results of unknown actions { some advanced function approx-
imation methods should be explored (we aim to investigate Gaussian Processes
this way). There are not enough experiments from dicult continuous real-world
environments, which would show for example how large problems can be solved
using safe exploration. The interleaved learning needs some guidelines on how
to cluster the queries for the teacher to some larger \packs" and \ask" them
together, possibly increasing the fully autonomous operating time. Last, but
not least, the possibility to share some learned safety functions among dierent
kinds of robots seems to be an unexplored area with many practical applications
(maybe robot-to-robot LfD could be used).
6 Acknowledgments
This work has been supported by the Grant Agency of the Czech Technical Uni-
versity in Prague under Project SGS13/142/OHK3/2T/13, and by EC project
FP7-ICT-609763 TRADR. We would like to thank Karel Zimmermann and
Michal Reinstein (Center for Machine Perception, CTU in Prague) for their
valuable insights into reinforcement learning methods.
References
1. Abbeel, P., Coates, A., Quigley, M., Ng, A.Y.: An application of reinforcement
learning to aerobatic helicopter
ight. In: Proceedings of the 2006 Conference on
Advances in Neural Information Processing Systems 19. vol. 19, p. 1 (2007)
2. Argall, B.D., Chernova, S., Veloso, M., Browning, B.: A survey of robot learn-
ing from demonstration. Robotics and Autonomous Systems 57(5), 469{483 (May
2009)
3. Barto, A.G., Sutton, R.S., Brouwer, P.S.: Associative search network: A reinforce-
ment learning associative memory. Biological cybernetics (1981)
4. Bertsekas, D.P.: Dynamic programming: deterministic and stochastic models.
Prentice-Hall (1987)
5. Chernova, S., Veloso, M.: Condence-based policy learning from demonstration
using Gaussian mixture models. In: AAMAS '07 Proceedings. p. 1. ACM Press
(2007)
6. Consortium, N.: NIFTi robotic UGV platform (2010)
LNCS: Safe Exploration for Reinforcement Learning 19
7. Coraluppi, S.P., Marcus, S.I.: Risk-sensitive and minimax control of discrete-time,
nite-state Markov decision processes. Automatica (1999)
8. Delage, E., Mannor, S.: Percentile optimization in uncertain Markov decision pro-
cesses with application to ecient exploration. In: Proceedings of the 24th interna-
tional conference on Machine learning - ICML '07. pp. 225{232. ACM Press, New
York, New York, USA (2007)
9. Ertle, P., Tokic, M., Cubek, R., Voos, H., Soker, D.: Towards learning of safety
knowledge from human demonstrations. In: 2012 IEEE/RSJ International Confer-
ence on Intelligent Robots and Systems. pp. 5394{5399. IEEE (Oct 2012)
10. Garcia, J., Fern andez, F.: Safe exploration of state and action spaces in reinforce-
ment learning. Journal of Articial Intelligence Research 45, 515{564 (2012)
11. Garcia Polo, F.J., Rebollo, F.F.: Safe reinforcement learning in high-risk tasks
through policy improvement. In: 2011 IEEE Symposium on Adaptive Dynamic
Programming and Reinforcement Learning (ADPRL). pp. 76{83. IEEE (Apr 2011)
12. Geibel, P.: Reinforcement learning with bounded risk. ICML pp. 162{169 (2001)
13. Gillula, J.H., Tomlin, C.J.: Guaranteed safe online learning of a bounded system.
In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.
pp. 2979{2984. IEEE (Sep 2011)
14. Hans, A., Schneega, D., Sch afer, A., Udluft, S.: Safe exploration for reinforcement
learning. In: Proceedings of European Symposium on Articial Neural Networks.
pp. 23{25. No. April (2008)
15. Heger, M.: Consideration of risk in reinforcement learning. In: 11th International
Machine Learning Conference (1994)
16. Howard, R.A.: Dynamic Programming and Markov Processes. Technology Press of
Massachusetts Institute of Technology (1960)
17. Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement Learning: A Survey.
Journal of Articial Intelligence Research 4, 237{285 (1996)
18. Kim, D., Kim, K.E., Poupart, P.: Cost-Sensitive Exploration in Bayesian Reinforce-
ment Learning. In: Proceedings of Neural Information Processing Systems (NIPS)
(2012)
19. Mihatsch, O., Neuneier, R.: Risk-sensitive reinforcement learning. Machine learning
49(2-3), 267{290 (2002)
20. Moldovan, T.M., Abbeel, P.: Safe Exploration in Markov Decision Processes. In:
Proceedings of the 29th International Conference on Machine Learning (May 2012)
21. Nilim, A., El Ghaoui, L.: Robust Control of Markov Decision Processes with Un-
certain Transition Matrices. Operations Research 53(5), 780{798 (Oct 2005)
22. Peter Geibel, Wysotzki, F.: Risk-Sensitive Reinforcement Learning Applied to Con-
trol under Constraints. Journal Of Articial Intelligence Research 24, 81{108 (Sep
2011)
23. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Pro-
gramming. John Wiley & Sons, Inc., New York, NY, USA, 1st edn. (1994)
24. Schneider, J.G.: Exploiting model uncertainty estimates for safe dynamic control
learning. Neural Information Processing Systems 9, 1047{1053 (1996)
25. Watkins, C.J., Dayan, P.: Q-learning. Machine Learning 8(3-4), 279{292 (May
1992)
26. Williams, R.J., Baird, L.C.: Tight performance bounds on greedy policies based on
imperfect value functions. Tech. rep., Northeastern University,College of Computer
Science (1993)