Negative Side Effects and AI Agent Indicators: Experiments in SafeLife
John Burden,1 3 \*Jos´e Hern ´andez-Orallo,1 2Se´an´O h´Eigeartaigh1 3
1Leverhulme Centre for the Future of Intelligence, Cambridge, UK
2Universitat Polit `ecnica de Val `encia, Spain
3Centre for the Study of Existential Risk, Cambridge, UK
\*jjb205@cam.ac.uk
Abstract
The widespread adoption and ubiquity of AI systems will re-
quire them to be safe. The safety issues that can arise from
AI are broad and varied. In this paper we consider the safety
issue of negative side effects and the consequences they can
have on an environment. In the safety benchmarking domain
SafeLife, we discuss the way that side effects are measured,
as well as presenting results showing the relation between the
magnitude of side effects and other metrics for three agent
types: Deep Q-Networks, Proximal Policy Optimisation, and
a Uniform Random Agent. We observe that different metrics
and agent types lead to both monotonic and non-monotonic
interactions, with the ﬁnding that the size and complexity of
the environment versus the capability of the agent plays a ma-
jor role in negative side effects, sometimes in intricate ways.
Introduction
As advances within Artiﬁcial Intelligence (AI) continue, AI
systems are becoming increasingly ubiquitous. Further ad-
vances offer the potential for great beneﬁts: future AI sys-
tems may use techniques such as reinforcement learning
(RL) to allow for agents with the ability to operate with
greater autonomy, making use of a greater range of actions
in more varied environments, in pursuit of more complex
goals. To realise these beneﬁts, the performance of these fu-
ture AI systems must be robustly safe and predictable. Safety
issues can take many forms and even the most exhaustive
surveys (Amodei et al. 2016; Critch and Krueger 2020) can-
not identify and classify every conceivable risk. Broad areas
of concern within the AI Safety literature include corrigibil-
ity (Soares et al. 2015), Safe Exploration (Garcia and Fer-
nandez 2012), Value alignment (Russell 2019), Side Effects
(Amodei et al. 2016), among many others.
Benchmarking domains have become a popular method
for the evaluation of safety properties for AI systems. There
are many such benchmarking suites available, such as AI
Safety Gridworlds (Leike et al. 2017), Safety Gym (Ray,
Achiam, and Amodei 2019), SafeLife (Wainwright and Eck-
ersley 2019) and the Real World Reinforcement Learning
Challenge Framework (Dulac-Arnold et al. 2020). A key ad-
vantage of this type of black-box testing is that it enables
Copyright © 2021, for this paper by its authors. Use permitted un-
der Creative Commons License Attribution 4.0 International (CC
BY 4.0).quantiﬁcation of an AI system’s performance with respect
to the appropriate safety properties. This allows direct com-
parisons between separate algorithms and posited solutions
to safety problems.
Our goal with this paper is to study how the performance
and safety, in terms of side effects, evolve as more resources
are given to the learning agent. This analysis is enriched by
a more detailed analysis of several indicators of the agent’s
behaviour and the side effect metric itself, in a benchmark,
SafeLife, where safety depends on ﬁnding trade-offs be-
tween achieving maximum reward and affecting the whole
environment through careless exploration. This analysis can
provide key insights into the types of behaviour we may ex-
pect from learning agents and how they may affect the world
around them.
Background and Related Work
Negative side effects (NSE) are a key issue in AI safety
(Amodei et al. 2016). NSEs typically stem from a mis-
speciﬁed reward or objective function — undesirable be-
haviour was not sufﬁciently penalised. A large part of the
difﬁculty with accurately specifying a reward or objective
function to avoid NSEs is the sheer amount of things we
want our AI systems to notdo. Ideally the AI systems we
build will not do undesirable and unrelated things such as
injuring humans, destroying useful equipment or irrevoca-
bly exterminating all life on the planet in the course of per-
forming its task. Encoding the vast amount of undesirable
actions directly into the reward/objective function is clearly
intractable. This motivates research into identifying clearly
what constitutes an NSE as well as how AI systems can learn
to avoid performing them.
We give an overview of the formulation of NSEs and re-
lated notions from the AI Safety community.
Low-impact AI systems (Armstrong and Levinstein 2017)
are proposed as a method for minimising NSEs. Here, the
authors claim that it is desirable for the overall impact of
an AI system to be low in order to prevent undesirable sit-
uations. Impact is deﬁned by the difference in the worlds
where the AI exists and the counterfactual worlds where the
AI does not exist. A sufﬁciently coarse measurement of the
possible world’s properties is to be used to ensure that an AI
cannot make large, sweeping changes to the world without
constituting a large impact. At the same time, coarse world
properties would reduce the size of the world representation
and make the calculation more tractable. However, the prop-
erties need to be selected such that they are important for
us — humanity — to care about. Additionally, with this for-
mulation, positive large impacts are also minimised but the
authors claim this is easier to handle since we often have
a clearer idea of what these positive impacts would be. A
similar approach is proposed in (Amodei et al. 2016) where
impact is regularised, but posits that due to the similarity of
many side effects such a regularisation could be learned in
one domain and transferred over to another.
As part of an AI Safety Gridworlds environment (Leike
et al. 2017) irreversible side effects are penalised. These ac-
tions are not inherently negative in themselves but their ir-
reversibility implies that they cannot be undone if later on
the action was deemed undesirable. This approach has the
downside that it may allow for reversible NSEs to be re-
peated indeﬁnitely.
Stepwise inaction is introduced in (Turner, Hadﬁeld-
Menell, and Tadepalli 2020), where the effect of action
against inaction is calculated after each time-step over aux-
iliary Q-functions. Here the intuition is that the agent is only
penalised for very impactful actions once, when they occur.
In order to account for actions that take many time-steps
for the undesirable side effect to occur, baseline rollouts are
used, where the state of the system after many steps of inac-
tion is used for the comparison.
Krakovna et al. (2019) provide a comprehensive compar-
ison of the above formulations over a range of gridworld
domains, showing where each of them succeed and fail; ad-
ditionally, the concept of relative reachability is introduced
— this is a measure of the average reduction of reachability
for every state compared to a baseline.
The recurring theme for deﬁning and identifying NSEs is
the idea of comparing the effect of the agent against some
scenario in which the agent would have acted in a different
way. Both the exact baseline and comparison function often
differ and, as shown in (Krakovna et al. 2019), have relative
advantages and disadvantages for different environments.
Types of Side Effects
The above formulations capture many aspects of measuring
NSEs and deciding which counterfactual worlds to compare
against. A notion that deserves additional detail, however,
is that of the different properties of NSEs and how they af-
fect the world. Distinguishing between NSEs by the prop-
erties they have can help us to further understand their con-
sequences as well as to identify and implement robust solu-
tions to them. We consider the application of an autonomous
agricultural drone which can survey and spray crop ﬁelds
as an illustrative example, giving an intuitive description of
each outlined NSE property.
We refer to a ﬁrst type of NSE property as “blocking”.
By this we mean an action by an AI system that prevents
it from performing its task successfully, or substantially and
irrevocably curtails the reward it can achieve. Examples of
this NSE include an agent taking an action which locks away
an object it needs to complete its task. In our agricultural
drone example, this would be an action that causes the droneto be unable to ﬂy or spray properly. While not part of the
speciﬁcation, these side effects are detectable as the goal or
associated reward are no longer achievable.
A second property of interest is that of “irreversible” ac-
tions. Here we refer to the same type of actions as described
in (Leike et al. 2017). These are particularly negative — and
difﬁcult to detect — in the cases where an action’s conse-
quences are not clear for many time-steps. This type of ef-
fect would correspond to the drone unintentionally spraying
entities such as people with potentially harmful chemicals.
This can clearly not be undone.
“Effect indifference” is another relevant NSE property.
Here we consider a scenario in which there is some be-
haviour which we do not want the agent to do and where
the AI system’s preferences are indifferent to this behaviour.
This may also include an indifferent reward function. This
NSE type is very much intertwined with that of reward
mis-speciﬁcation and often occurs due to the system hav-
ing unanticipated affordances or incentives. Similarly to the
last example, this effect type could correspond to spraying
unintended entities if this is not properly encoded into the
reward function.
The ﬁnal NSE property we consider here is that of the “re-
ward trade-off”. This effect typically arises when resources
are limited and shared with other entities (humans included),
and these resources are instrumentally useful to the agent.
The danger here is not only that the maximisation of the
reward could imply that all resources should be exhausted
(e.g., the famous instrumental convergence problems, such
as Minsky’s Riemann hypothesis or Bostrom’s paperclip
problem, Bostrom 2003), but even in situations where the
reward is capped, the agent may take more than it needs for
its task and perform an inefﬁcient and unsustainable use of
the shared resources. In our agricultural drone example, this
would correspond to the drone revisiting already treated ar-
eas or making detours, leading to high energy consumption
or shorter operational life.
Different actions can have several of the properties out-
lined above; this may compound the risk posed by a side
effect.
AI Safety benchmarking suites can evaluate some or all
of the above. In the AI Safety Gridworlds (Leike et al. 2017)
benchmarking suite, the irreversible side effects environ-
ment captures what we refer to as an irreversible NSE. The
agent here is tasked with reaching a goal space, but must
move a box that is blocking its path. Moving the box adja-
cent to a side or a corner can hinder or prevent future move-
ment of the box. The agent is not directly penalised for per-
forming these actions, but it is undesirable from a safety per-
spective. The Safety Gym (Ray, Achiam, and Amodei 2019)
generally requires an agent to perform a task such as reach-
ing a goal or moving a box to a speciﬁc position while avoid-
ing numerous hazard types. The side effects possible in these
environments generally utilise the reward trade-off property;
the agents are trying to maximise their rewards whilst try-
ing to minimise constraint violation. The Real World Re-
inforcement Learning Challenge Framework (Dulac-Arnold
et al. 2020) does not contain any environments that directly
measure the side effects of an agent, though some tasks re-
quire the agent not to violate certain safety constraints. On
the other hand, SafeLife (Wainwright and Eckersley 2019),
described in the following section, presents side effects that
can consist of each of the NSE properties seen above.
SafeLife Environment
SafeLife (Wainwright and Eckersley 2019) is a safety bench-
marking suite for RL agents. SafeLife takes the complex me-
chanics from Conway’s Game of Life (Gardener 1970) and
utilises them to create single-agent RL domains. The result-
ing environments have a rich level of complexity and allow
for formulations of many different tasks. A distinguishing
feature of SafeLife is its focus on measuring and avoiding
side effects.
Each environment consists of an nmgrid of cells which
can each take a variety of properties (see an example in Fig.
1). We refer to this grid as a board. The most notable prop-
erty is whether the cell is “dead” or “alive”. Following the
Game of Life rules the cells update as follows: a dead cell
will become alive if it has exactly three neighbouring alive
cells, otherwise it will remain dead. On the other hand, an
alive cell will die if it has fewer than two alive neighbours or
more than three alive neighbours, otherwise it will survive.
SafeLife extends this basic framework into a single-agent
domain by including an agent that traverses the grid. At each
time step, the agent perceives a 2525grid overview of
its surroundings, centred on itself. For each position in this
overview, the cell type present is perceived.
Instances in SafeLife come with goals for the agent to
complete, usually revolving around creating or destroying
particular life structures before moving to a predeﬁned goal
cell. Additionally, the other effects the agent has on pre-
existing life cells are measured — these correspond to un-
intended side effects.
In SafeLife the agent is depicted by an arrow
 and may
move in any cardinal direction, as well as creating or de-
stroying life cells in adjacent spaces. The agent’s movement
can be impeded by walls which are shown as a grey square
with a black hole
 . Additionally, there are boxes which
are shown as a grey square without a black hole
 that the
agent can push and pull. Pre-existing green life cells are de-
picted using green circles
 and life cells created by the
agent are shown as grey circles
 . These agent-created life
cells are to be placed in the designated areas shown in blue.
Both types of life cells that follow the rules of the Game of
Life. The “tree” entities are depicted by a green square with
radiating lines
 and are ﬁxed living cells that cannot be
destroyed. Finally, the goal is depicted with a grey arch
 ,
which turns red when it can be activated. All cells that have
not been explicitly noted as living are “dead”.
Altogether this allows for a very rich and complex set of
environments which can be generated procedurally.
Measuring Side Effects
We now look at the method that the authors of SafeLife pro-
pose for measuring the magnitude of side effects. Within
SafeLife the destruction, addition or changing of cell types
caused by agent actions represent side effects. The types of
Figure 1: An example task instance from SafeLife. The
agent
 must create life structures
 in the designated blue
positions before moving to the goal
 . Ideally the agent
should not disturb the green life cells
 .
side effects that we are concerned with are those made to
the pre-existing life cells, but SafeLife provides the means to
look at the effects on any cell type. The primary reason for
only considering the effects the agent has on the green life
cells is the fact that the agent is not penalised or rewarded
for disrupting the structures they form; thus they may be de-
stroyed as an ‘effect indifference’ NSE. The green cells can
obstruct the agent and prevent it from moving, so it may be
instrumentally useful for the agent to destroy them, making
them relevant to the ‘trade-off’ NSE property. Further, the
destruction can be permanent, and thus may represent irre-
versible NSEs. Finally, because these life cells follow the
rules of the game of life, they can have very interesting and
complex behaviours.
Due to the dynamic nature of SafeLife, comparing a sin-
gle ﬁnal state against a single baseline state may not be sufﬁ-
cient to capture the eventual side effects of an agent. Instead,
two distributions are created that represent the future states
of the board for both the case where the agent acted and af-
fected the environment as well as the counterfactual case in
which the agent never existed. To create the action case dis-
tribution,Dafrom the ﬁnal state left by the agent, a further
1000 steps are simulated and each resulting state is added
toDa. The inaction case distribution Diis created similarly,
except that 1000 +nsteps are run from the initial state of
the environment, where nis the number of steps taken by
the agent during the episode. This ensures that both distri-
butions reﬂect the states of the board 1000 +nsteps in the
future from the initial state for both the factual and counter-
factual cases.
From these distributions, the estimated pro-
portion of time steps that board position
x=hx1;x2i;x2grid =f1;:::;ngf 1;:::;mg con-
tains cell type ccan be computed:
c
D(x) =1
jDjX
s2D[ 1(s(x) =c)]
whereDis the relevant distribution, sis the state drawn from
Dands(x)is the cell type present in sat locationx.
A ground distance function is also deﬁned between board
positions asg(x;y) = tanh(1
5kxyk1)wherekxyk1is
the Manhattan distance between grid locations xandy.
The calculated side effect for Cell Type cis then:
dc(Da;Di) =EMD (c
Da;c
Di;g)
whereDaandDiare the action and inaction distributions re-
spectively, and EMD is the earth mover distance function.
Side effects in SafeLife are based on this distance metric,
which aims to capture the distance between two probabil-
ity distributions based on the amount of work it would take
to transform one distribution to the other by moving around
“distribution mass” (Rubner, Tomasi, and Guibas 1998).
This side effect score dc(Da;Di)is then “normalised”
against the inaction baseline to give a ﬁnal side effect score:
c(Da;Di) =dc(Da;Di)P
x2gridc
Di(x)
This normalisation allows us to compare agent behaviours
in larger or more densely populated boards.
While a side effect score can be calculated for every cell
type, SafeLife focuses on the side effects for the green, pre-
existing life cells.
This deﬁnition of a side effect echoes that of (Armstrong
and Levinstein 2017) and (Amodei et al. 2016), where the
agent’s actions are compared against a world in which it
never acted at all. A key difference however is the use of
future distributions to account for the possible long-term or
unstable effects of actions.
Examples and Implications
Figure 2 shows a before and after comparison to show the
effects of an agent’s actions. In the “before” environment,
the agent places a grey life cell directly in front of it, adjacent
to the stable green life structure. This disrupts the structure
and after a few time steps it is destroyed. At the end of the
episode, assuming the agent does not affect any more green
life cells, we can calculate the side effect score as follows:
(a) Before
 (b) After
Figure 2: Example for Side Effect calculation
Since both the “before” and “after” snapshots are stable
(the structures if left alone will not change),
green
Da(x) = 1(sbefore (x) =green )green
Di(x) = 1(safter(x) =green )
As there are seven green cells in total, it then follows that
green (Da;Di) =EMD (green
Da;green
Di;g)P
x2gridgreen
Di(x)=4
7
The use of the normalised earth mover distance has some
interesting consequences for the side effect score. Regard-
less of how large or densely packed the grid is with green
life cells, not disturbing any of them will yield a score of 0,
destroying them all will give a score of 1. However, in the
case that more green cells are created, by the agent adding
grey life cells to certain patterns which lead to an increase
in the number of green life cells, the side effect score can
increase to above 1, due to quirks with the calculation of the
earth mover distance. Whether or not causing more green
life cells to spawn is worse than the complete destruction of
all green life cells will depend on what these life cells repre-
sent as well as personal moral philosophy. At the very least
thisdoes capture the notion of representing the total impact
the agent has on the green life cells.
Experiments
SafeLife has many tasks available out-of-the-box, and
vastly more can be created by the user. We focus on the
append-still task, where the agent must create life
structures on the designated areas and then move to the goal
which ends the episode. Before the goal is accessible to the
agent, more than half of the designated (blue) positions must
be ﬁlled with life cells. At each step the agent receives a re-
ward equal to the number of life cells created on the desig-
nated positions minus the number of life cells destroyed that
were on designated positions. Finally the agent receives an
additional 1reward for reaching the goal after it has opened.
After 1000 steps by the agent, the episode will end, regard-
less of the agent’s progress. If the agent has managed to ﬁll
at least half of the designated positions for its life cells and
reached the goal to end the episode before the 1000 steps
have elapsed, we say the agent has ‘passed’ the task instance,
otherwise it has failed.
In this task there are also pre-existing green life cell struc-
tures which we do not want the agent to disturb. Importantly,
the reward function does not encourage or penalise any agent
interaction with these green life cells and the agent is thus in-
different to them. Any interaction the agent has on the green
life cells therefore captures the essence of side effects caused
by reward mis-speciﬁcation — we want the pre-existing life
cell structures to survive, but this is not encoded into the re-
ward function.
SafeLife can utilise procedural generation to create
append-still tasks. The parameters used for the proce-
dural generation can greatly affect the difﬁculty of the task
instance, and thus the agent’s ability to safely complete the
task. For our experiments we utilised a wide range of param-
eter settings to capture a broad overview of a given agent’s
capabilities.
Table 1 gives the parameter list for the procedural gen-
eration process. Grid Size corresponds to the width and
height of the board to be generated. SideEffectMin is
the minimum proportion of cells in the task to be green life
cells. GoalMin andGoalMax are the respective minimum
and maximum proportions of the board to be areas for the
agent to construct its own life cells. Finally Temperature
corresponds to a variable which controls the complexity of
the stable life patterns that can be generated for both the life
cells (both the pre-existing green cells and the spaces for the
agent to place its own cells).
We train each agent for a predetermined number of steps,
with the instances generated using parameters drawn uni-
formly at random from the set of difﬁculties. The agents are
then evaluated on 1000 episodes of each difﬁculty type (for
a total of 13000 episodes) and each score is aggregated.
Results and Discussion
Here we describe our results from the experiments outlined.
The two main algorithms we assess are Deep Q-Networks
(DQN) (Mnih et al. 2013) and Proximal Policy Optimisation
(PPO) (Schulman et al. 2017) (both trained for 5, 10, and 30
million training steps), We also examine an agent that selects
actions uniformly at random (Uniform). For DQN and PPO
we use the implementation provided within SafeLife.
Since the task instances may be of different sizes, and the
procedural generation may produce more green life cells or
designated positions for the agent to place life cells, the total
reward received by an agent is normalised against the max-
imum possible reward for that instance. Similarly the side
effect score is normalised.
Table 2 contains the scores for various metrics for each
evaluated agent. As expected, more training steps yields bet-
ter pass rates and average rewards. This improvement does
not carry over to the side effect score, however, where for
each agent type the side effect score slightly increases with
more training steps for DQN but slightly decreases for PPO,
although the trend is not clear. This is closely related to the
results for the exploration rate for each agent, which vary in
a similar way according to the increase of training steps from
5M to 30M. Recall that this is the evaluation stage, therefore
this reduction in exploration is ideal. The agent can see the
whole board, so it is more beneﬁcial for the agent to attempt
its task without wasting time exploring.
As should be expected, both agent types show safer be-
haviour than the agent that selects random actions, and the
low rewards serve as a baseline for comparison (e.g., inter-
estingly, the pass rate for DQN 5M is worse than the uniform
random agent). DQN and PPO do not overlap in their reward
scores, but the side effect score for PPO 30M is not very far
from the best side effect score (DQN 5M). This suggests that
there is no clear association between rewards and scores, at
least if we look at the aggregated results.
Among the pass rate, average reward and side effect score
we also see large standard deviations suggesting that none
of the agents learn to perform consistently. The large range
of procedural generation parameters is likely the cause of
this, preventing agents from overﬁtting to one particular set
of generation patterns. The exceptions to this are DQN 5M
and 10M, which have signiﬁcantly lower standard deviations
for the pass rate by virtue of having a signiﬁcantly lower
pass rate that is bounded below by 0. This may also concealsome associations between reward and side effects, which
may appear if we disaggregate the results.
Looking in more detail at the performance of individual
algorithms can give more insight into how an agent’s be-
haviour affects the side effects. In Figures 3, 4 and 5 we plot
the mean side effect score against the reward attained, ex-
ploration rate and length of an episode, using 100 bins in the
x-axis. In Figure 6 we plot the mean maximum side effect
available against exploration rate. In these ﬁgures we can
see some relationships emerge when we look at the reward
attained by an agent. For PPO there is a clear increase in
the side effects caused from 0to0:4followed by a slower
decrease from 0:4to1. This change in side effect score ap-
pears to be consistent across each PPO variant, though the
magnitude of the increase varies. For episodes with very low
reward, more training appears to prevent dangerous actions,
though it is not clear exactly what causes this change in be-
haviour. For DQN and the Uniform agent, there is less of
a relation with the reward received; instead, the side effect
score is more constant, although more training reduces the
dispersion of the scatter plots for DQN. Nevertheless, for the
three kinds of agents we see that maximum reward (1.0) cor-
responds with very low side effects and, for PPO and DQN,
minimum reward (0.0) corresponds with very low side ef-
fects. What happens in between is more pronouncedly a bell
shape for better agents (e.g., PPO with higher training steps).
In Figure 4 we compare episode length against the side
effect score, and see positive correlations. This intuitively
makes sense, as the longer an agent is acting, the more op-
portunities it has to disrupt cell structures. Again we note
that for longer episodes the variance becomes larger. Very
short episodes represent situations where the solution is
found easily, and roughly correspond to cases with maxi-
mum reward in Figure 3.
On the other hand, we see a tighter relationship when
we compare exploration rate against the side effect score
in Figure 5. For all the PPO variants, as the agent explores
more of the domain the side effect score increases. Also,
for larger exploration rates, the variance of scores becomes
much larger, suggesting a much less consistent relationship
for these values of the exploration rate. For DQN we only
see this relationship for the most highly-trained agent —
both DQN 5M and DQN 10M peak in their side effect score
after exploration rates of 0:1and0:25respectively, before
dropping to 0. Similarly, when we look at the Uniform agent
and its exploration rate, there is a very tight relationship be-
tween the two, with the side effect score increasing with the
exploration rate from 0to0:4, before dropping from 0:4to
0:6and remaining at 0hereafter.
These relationships can be somewhat explained by Figure
6 where we see the mean number of green cells in the inac-
tion distribution compared against the exploration rate. Here
we see similar patterns to the comparison of exploration rate
and side effects. It appears that the number of green cells
an environment contains will affect the exploration rate, and
that this will in turn affect the proportional amount of dam-
age certain agents can do. Why is it so? The rationale has to
be found in the way the environments are generated, as in-
troducing green cells makes full exploration of the whole en-
Difﬁculty Grid Size sideEffectMin GoalMin GoalMax Temperature
0 1010 0 0 0 0.1
1 1515 0.03 0.05 0.1 0.1
2 1515 0.03 0.05 0.1 0.2
3 1515 0.05 0.05 0.1 0.2
4 1515 0.07 0.05 0.15 0.2
5 1515 0.09 0.05 0.2 0.2
6 1515 0.1 0.05 0.2 0.2
7 1515 0.1 0.1 0.25 0.3
8 1515 0.1 0.15 0.25 0.4
9 1515 0.1 0.15 0.35 0.5
10 1515 0.1 0.15 0.4 0.5
11 1515 0.1 0.15 0.45 0.6
12 1515 0.1 0.2 0.5 0.7
Table 1: Difﬁculty Parameters for our SafeLife levels.
Agent Pass Rate Average Reward Side Effect Score Exploration Rate
DQN 5M 0.043 (0.203) 0.272 (0.288) 0.229 (0.342) 0.069 (0.042)
DQN 10M 0.098 (0.297) 0.417 (0.345) 0.268 (0.357) 0.110 (0.078)
DQN 30M 0.299 (0.458) 0.495 (0.348) 0.247 (0.347) 0.121 (0.071)
PPO 5M 0.682 (0.466) 0.658 (0.293) 0.324 (0.389) 0.205 (0.138)
PPO 10M 0.678 (0.467) 0.676 (0.294) 0.241 (0.344) 0.130 (0.084)
PPO 30M 0.709 (0.454) 0.695 (0.275) 0.243 (0.344) 0.148 (0.095)
Uniform 0.097 (0.295) 0.181 (0.343) 0.688 (0.450) 0.347 (0.086)
Table 2: Results for various metrics in SafeLife for our selected agents. Mean and (standard deviation)
vironment more difﬁcult. Basically, explorations above 0.5
correspond to very special environments, such as those that
are very small and do not contain green cells and are easy
to explore fully even for a random agent. This means that
we have to be careful to interpret the rightmost part of the
curves in these plots. In a “reward trade-off” scenario about
side effects, the more limited and concentrated the resources
(targeted and non-targeted) are the higher the chance to have
an impact. Large environments with plenty of resources are
easier to handle, such as the one in Figure 2. We may need to
consider two ‘difﬁculties’: one about the task rewards them-
selves and another one about the hardness of avoiding side
effects. The use of a random agent can help elucidate these
two cases, but a more detailed analysis of the generation pa-
rameters such as those in Table 1 may be needed too.
Overall, the general picture is that DQN behaves between
the uniform agent and PPO, and the range depends on the
number of training steps. As the agent is better in terms of
rewards (from DQN 5M to PPO 30M), the positive relation
between exploration rate and side effect is clearer. The de-
tailed view of Figure 3, and the way it bends down for PPO
illustrates that for those environments where the agent seems
more determined and manages to get high rewards, the ex-
ploration is lower and so is the side effect. It is mostly for
those situations where the reward is in the range of 0.2 to 0.8
that the side effects are strongest. We hypothesise that these
are the cases where the agent only knows partially what to
do. Finally, those cases with very low rewards may be caused
by several situations, such as the agent being stuck in a loop,reducing exploration and side effects.
This suggests that the analysis of the conﬁdence of the
agent, or some other metacognitive proxies could be used to
distinguish cases where the agent can complete the task with
high reward and low effect, and those cases where uncer-
tainty is higher and the agent should be more conservative.
Conclusion and Future Work
Despite additional training time yielding much more capable
agents, we do not see a similar increase in safety; in fact
the safest agent overall was DQN after 5M training steps,
while being the second worst performing agent in terms of
reward. However, when we analyse the detailed behaviour
using several indicators we ﬁnd a region of high rewards
and low effect, which is only achieved by PPO. This shows
a non-monotonic relationship, suggesting that the area with
medium rewards is the most dangerous for proﬁcient agents.
This paper highlights the differences in behaviour be-
tween two commonly used RL algorithms when trained on
the same task. As the results show, these behavioural differ-
ences can have a signiﬁcant effect on our ability to predict
the safety and other factors of the agent, such as the conﬁ-
dence or its level of competency compared to elements of the
environment, such as the proportion of green cells. Further
analysis and exploration of the difﬁculty of task instances
may help to elucidate the cause of these relationships be-
tween side effects and other metrics, as well as to help us
to understand how environment difﬁculty and agent uncer-
tainty can be used to improve policies and make them safer.
(a) PPO
(b) DQN
(c) Uniform Agent
Figure 3: Mean Side Effect against Reward attained for
variants of PPO, DQN and Uniform agents. The 13,000
episodes are binned on the x-axis, with the size of the plotted
points (squares, triangles and circles) being logarithmic on
the number of episodes in each bin. The curves are rolling
means of the plotted points with a window size of 7.
(a) PPO
(b) DQN
(c) Uniform Agent
Figure 4: Mean Side Effect against Episode Length for
variants of PPO, DQN and Uniform agents. The 13,000
episodes are binned on the x-axis, with the size of the plotted
points (squares, triangles and circles) being logarithmic on
the number of episodes in each bin. The curves are rolling
means of the plotted points with a window size of 7.
(a) PPO
(b) DQN
(c) Uniform Agent
Figure 5: Mean Side Effect against Exploration Rate for
variants of PPO, DQN and Uniform agents. The 13,000
episodes are binned on the x-axis, with the size of the plotted
points (squares, triangles and circles) being logarithmic on
the number of episodes in each bin. The curves are rolling
means of the plotted points with a window size of 7.
(a) PPO
(b) DQN
(c) Uniform Agent
Figure 6: Number of initial life cells against Exploration
Rate for variants of PPO, DQN and Uniform agents. The
13,000 episodes are binned on the x-axis, with the size of
the plotted points (squares, triangles and circles) being log-
arithmic on the number of episodes in each bin. The curves
are rolling means of the plotted points with a window size
of7.
Acknowledgements: This work was funded by the Leverhulme
Trust, the Future of Life Institute, FLI (grant RFP2-152), the
EU’s Horizon 2020 research and innovation programme (No.
952215, TAILOR), and EU (FEDER) and Spanish MINECO un-
der RTI2018-094403-B-C32, and Generalitat Valenciana GV under
PROMETEO/2019/098.
References
Amodei, D.; Olah, C.; Steinhardt, J.; Christiano, P.; Schul-
man, J.; and Man ´e, D. 2016. Concrete problems in AI safety.
arXiv preprint arXiv:1606.06565 .
Armstrong, S.; and Levinstein, B. 2017. Low impact artiﬁ-
cial intelligences. arXiv preprint arXiv:1705.10720 .
Bostrom, N. 2003. Ethical issues in advanced artiﬁcial in-
telligence. Science ﬁction and philosophy: from time travel
to superintelligence 277–284.
Critch, A.; and Krueger, D. 2020. AI Research Considera-
tions for Human Existential Safety (ARCHES).
Dulac-Arnold, G.; Levine, N.; Mankowitz, D. J.; Li, J.;
Paduraru, C.; Gowal, S.; and Hester, T. 2020. An empirical
investigation of the challenges of real-world reinforcement
learning .
Garcia, J.; and Fernandez, F. 2012. Safe Exploration of
State and Action Spaces in Reinforcement Learning. Jour-
nal of Artiﬁcial Intelligence Research 45: 515–564. ISSN
1076-9757. doi:10.1613/jair.3761. URL http://dx.doi.org/
10.1613/jair.3761.
Gardener, M. 1970. Mathematical games: the fantastic com-
binations of John Conway’s new solitaire game “life”.
Krakovna, V .; Orseau, L.; Kumar, R.; Martic, M.; and Legg,
S. 2019. Penalizing side effects using stepwise relative
reachability.
Leike, J.; Martic, M.; Krakovna, V .; Ortega, P. A.; Everitt,
T.; Lefrancq, A.; Orseau, L.; and Legg, S. 2017. AI Safety
Gridworlds.
Mnih, V .; Kavukcuoglu, K.; Silver, D.; Graves, A.;
Antonoglou, I.; Wierstra, D.; and Riedmiller, M. 2013. Play-
ing Atari with Deep Reinforcement Learning.
Ray, A.; Achiam, J.; and Amodei, D. 2019. Benchmark-
ing safe exploration in deep reinforcement learning. arXiv
preprint arXiv:1910.01708 .
Rubner, Y .; Tomasi, C.; and Guibas, L. J. 1998. A Metric
for Distributions with Applications to Image Databases. In
Proceedings of the Sixth International Conference on Com-
puter Vision , ICCV ’98, 59. USA: IEEE Computer Society.
ISBN 8173192219.
Russell, S. 2019. Human Compatible: Artiﬁcial Intelligence
and the Problem of Control . Penguin Publishing Group.
Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and
Klimov, O. 2017. Proximal Policy Optimization Algorithms.
Soares, N.; Fallenstein, B.; Armstrong, S.; and Yudkowsky,
E. 2015. Corrigibility. URL https://aaai.org/ocs/index.php/
WS/AAAIW15/paper/view/10124.Turner, A. M.; Hadﬁeld-Menell, D.; and Tadepalli, P. 2020.
Conservative Agency via Attainable Utility Preservation.
Proceedings of the AAAI/ACM Conference on AI, Ethics,
and Society doi:10.1145/3375627.3375851. URL http://dx.
doi.org/10.1145/3375627.3375851.
Wainwright, C. L.; and Eckersley, P. 2019. Safelife 1.0: Ex-
ploring side effects in complex environments. arXiv preprint
arXiv:1912.01217 .


Unifying Logic and Probability: A New Dawn
for AI?
Stuart Russell
1University of California, Berkeley CA 94720, USA
russell@cs.berkeley.edu
http://www.cs.berkeley.edu/~russell
2Universit´ e Pierre et Marie Curie, Laboratoire d’informatique de Paris 6,
4 place Jussieu, 75252 Paris Cedex 05, France
Abstract. Logic and probability theory are two of the most important
branches of mathematics and each has played a signiﬁcant role in ar-
tiﬁcial intelligence (AI) research. Beginning with Leibniz, scholars have
attempted to unify logic and probability. For “classical” AI,based largelyon ﬁrst-order logic, the purpose of such a uniﬁcation is to handle uncer-
tainty and facilitate learning from real data; for “modern” AI, based
largely on probability theory, the purpose is to acquire formal languageswith suﬃcient expressive power to handle complex domains and incor-
porate prior knowledge. This paper provides a brief summary of an in-
vited talk describing eﬀorts in these directions, focusing in particular onopen-universe probability models that allow for uncertainty about the
existence and identity of objects.
Keywords: ﬁrst-order logic, probability, probabilistic programming,
Bayesian logic, machine learning.
1 Introduction
From its earliest days, AI adopted the idea of declarative system reasoning over
explicitly represented knowledge with a general inference engine. Such systems
require a formal language to express knowledge about the real world; and the
real world has things in it . For this reason, in 1958, McCarthy [16] proposed
ﬁrst-order logic—the mathematics of objects and relations—as the foundation
for what we now call “classical AI.”
The key beneﬁt of ﬁrst-order logic is its expressive power, which leads to
concise—and hence easily learnable—models. For example, the rules of chess
occupy 100pages in ﬁrst-order logic, 105pages in propositional logic, and 1038
pages in the language of ﬁnite automata. The power comes from separatingpredicates from their arguments and qua ntifying over those arguments: so one
can write rules about On(p, c, x, y, t ) (piece pof color cis on square x, yat move
t) without having to ﬁll in each speciﬁc value for c,p,x,y,a n d t.
Asecondresearchtradition,sometimescalled“modernAI,”developedaround
another important property of the real world: pervasive uncertainty about both
A. Laurent et al. (Eds.): IPMU 2014, Part I, CCIS 442, pp. 10–14, 2014.
c/circlecopyrtSpringer International Publishing Switzerland 2014
Unifying Logic and Probability 11
its state and its dynamics. Modern AI is based on probability theory, which pro-
vides principled methods for learning and making predictions from observations.The key advance underlying modern AI was the development of Bayesian net-
works[22] and the related family of undirected graphical models [6]. Bayes nets
provideda formal languagefor probabilitymodels and enabled rapid advances inmachine learning, vision, natural language understanding, and knowledge-based
systems. The expressive power of Bayes nets is, however, limited. They assume
aﬁ x e ds e to f variables , each of which can take on a value from a ﬁxed range;
thus, they are a propositional formalism, like Boolean circuits. The rules of chess
and of many other domains are beyond them.
What happened next, of course, is that classical AI researchers noticed the
pervasiveuncertainty,while modern AI re searchersnoticed, orremembered, that
the world has things in it. Both traditions arrived at the same place: the world
is uncertain and it has things in it .T od e a lw i t ht h i s ,w eh a v et o unify logic and
probability .
But how? Even the meaning of such a goal is unclear. Early attempts by
Leibniz, Bernoulli, De Morgan, Boole, Peirce, Keynes, Carnap, and Gaifman
(surveyed in [8,10]) involved attaching probabilities to logical sentences. This
line of work inﬂuenced AI research [9,3,14] but has serious shortcomings as a
vehicle for representing knowledge. An alternative approach, arising from both
branches of AI and from statistics, draws on the compositional semantics ofBayesnets. Some toolsuse programmingconstructs to build verylargeBeys nets
with repeated structure [7,4,15], while others adopt the syntactic and semantic
devices of logic (composable function symbols, logical variables, quantiﬁers) tocreate declarative, ﬁrst-order probabilistic languages [5,23,25,12,11].
Despite their successes, these approaches miss an important consequence of
uncertainty in a world of things: there will be uncertainty about what things are
in the world . Real objects seldom wear unique identiﬁers or preannounce their
existence like the cast of a play. In the cas e of vision, for example, the existence
of objects must be inferredfrom raw data (pixels) that contain no explicit object
references at all. If, however, one has a probabilistic model of the ways in which
worlds can be composed of objects and of how objects cause pixel values, theninference canproposethe existence ofobjectsgiven onlypixel values asevidence.
Similar arguments apply to areas such as natural language understanding, web
mining, and computer security.
The diﬀerence between knowing all the objects in advance and inferring their
existence and identity from observation corresponds to an important but oftenoverlooked distinction between closed-universe languages such as SQL and logic
programs and open-universe languages such as full ﬁrst-order logic.
This distinction is best understood in terms of the possible worlds under each
type of semantics. Figure 1(a) shows a simple example with two constants and
one binary predicate. Notice that ﬁrst-order logic is an open-universe language:
eventhoughtherearetwo constantsymbols,the possibleworldsallowfor 1,2,or
indeed arbitrarily many objects. A closed-universe language enforces additional
assumptions that restrict the set of possible worlds:
12 S. Russell
. . .AB
A
BAB
A
BAB
A
BAB
A
BAB
A
B(b)AB AB AB AB AB AB
. . . . . . . . . (a)
Fig. 1.(a) Some of the ﬁrst-order possible worlds for a language with two constant
symbols, Aand B, and one binary predicate. Arrows indicate the interpretation of
each constant symbol and the relations between objects. (b) The analogous ﬁgure
under closed-universe semantics.
–T h e unique names assumption requires that dist inct terms must refer to dis-
tinct objects.
–T h e domain closure assumption requires that there are no objects other than
those named by terms.
These two assumptions mean that every possible world contains the same
objects, which are in one-to-one correspondence with the ground terms of the
language (see Figure 1(b)).1
A formal probability model must specify the probability of every possible
world given the vocabulary (predicates, functions, constants)of the model’s syn-
tactic representation.Obviously,the set ofworldsunder open-universesemantics
islargerandmoreheterogeneous,whichmakesthe taskofdeﬁningopen-universe
probability models more challenging. The core part of the talk is concerned
with a ﬁrst-order, open-universe probabilistic language called Bayesian logic or
BLOG [18,19]. BLOG was developed primarily as the PhD thesis research of
Brian Milch [17]. The key results derived for BLOG are the following:
–Every well-formed BLOG model speciﬁe s a well-deﬁned probability distribu-
tion over the possible worlds constructed from the vocabulary of the model.
–There exist Monte Carlo algorithms that provably converge(subject to tech-
nical conditions on the conditional distributions of the model) to the correct
posterior probability for any ﬁrst-order query for any well-formed BLOG
model [20,1].
1The diﬀerence between open and closed universes can also be illustrated
with a common-sense example. Suppose a system knows just two sentences,
Fa t h e r(W illiam )=Billand Fa t h e r(Junior)=Bill. How many children does Bill
have? Under closed-universe semantics—e.g., in a database system—he has exactly
2; under open-universe semantics, between 1 and ∞.
Unifying Logic and Probability 13
The generic algorithms (importance sampling and MCMC applied to a dynam-
ically constructed ground representation) are often too slow for practical useon large models. Several avenues are b eing pursued for speeding up inference,
including special-purpose block samplers for variables constrained by determin-
istic relationships [13], static analysis to identify submodels amenable to eﬃcientinference, lifted inference to avoid grounding by manipulating symbolic distri-
butions over large sets of objects [24,2 6], and compiler techniques to generate
model-speciﬁc inference code.
More than two dozen BLOG models have been developed, covering a wide
variety of standard machine learning models as well as applications includingcitation matching [21] and global seismic monitoring for the Comprehensive Nu-
clear Test-Ban Treaty [2].
2P r o s p e c t s
These are very early days in the process of unifying logic and probability. We
need much more experience in developing models for a wide range of applica-tions. Undoubtedly there are new modeling idioms, programming constructs,
and inference algorithms to discover.
The development of Bayes nets in the late 1980s connected machine learning
tostatisticsandreconnected(modern)AIwithvisionandlanguage.Itispossible
that ﬁrst-order probabilistic languages, which have both Bayes nets and ﬁrst-
order logic as special cases, can serve a similar, but more inclusive, unifying
role.
Acknowledgments. The author is supported by the Chaire Blaise Pascal ,
funded by the l’ ´Etat et la R´ egionˆIle de France and administered by the Fonda-
tion de l’ ´Ecole Normale Sup´ erieure. The research i s also supported by DARPA
contractFA8750-14-C-0011for the APPRIL project under the PPAML program
andpreviouslyby DARPAcontractFA8650-11-1-7153for the OUTBIDS project
under the MSEE program.
References
1. Arora, N., Russell, S., de Salvo Braz, R., Sudderth, E.: Gibbs sampling in open-
universe stochastic languages. In: UAI 2010 (2010)
2. Arora, N.S., Russell, S., Sudderth, E.: NET-VISA: Network processing vertically
integrated seismic analysis. Bull. Seism. Soc. Amer. 103 (2013)
3. Bacchus, F.: Representingand Reasoning with Probabilistic Knowledge. MIT Press
(1990)
4. Bessi`ere, P., Mazer, E., Ahuactzin, J.M., Mekhnacha, K.: Bayesian programming.
CRC (2013)
5. Breese, J.S.: Construction of belief and decision networks. Computational
Intelligence 8, 624–647 (1992)
6. Darroch, J.N., Lauritzen, S.L., Speed, T.P.: Markov ﬁelds and log-linear interaction
models for contingency tables. The Annals of Statistics 8(3), 522–539 (1980)
14 S. Russell
7. Gilks, W.R., Thomas, A., Spiegelhalter, D.J.: A language and program for complex
Bayesian modelling. The Statistician 43, 169–178 (1994)
8. Hailperin, T.: Probability logic. Notre Dame J. Formal Logic 25(3), 198–212 (1984)9. Halpern, J.Y.: An analysis of ﬁrst-order logics of probability. AIJ 46(3), 311–350
(1990)
10. Howson, C.: Probability and logic. J. Applied Logic 1(3-4), 151–165 (2003)11. Kersting, K., De Raedt, L.: Bayesian logic programs. In: ILP 2000 (2000)
12. Koller, D., Pfeﬀer, A.: Probabilistic frame-based systems. In: AAAI1998 (1998)
13. Li, L., Ramsundar, B., Russell, S.: Dynamic scaled sampling for deterministic
constraints. In: AI/Stats 2013 (2013)
14. Lukasiewicz, T.: Probabilistic logic programming. In: ECAI (1998)
15. McCallum, A., Schultz, K., Singh, S.: FACTORIE: Probabilistic programming via
imperatively deﬁned factor graphs. NIPS 22 (2010)
16. McCarthy, J.: Programs with common sense. In: Proc. Symposium on Mechanisa-
tion of Thought Processes. Her Majesty’s Stationery Oﬃce (1958)
17. Milch, B.: Probabilistic Models with Unknown Objects. Ph.D. thesis, UC Berkeley
(2006)
18. Milch, B., Marthi, B., Sontag, D., Russell, S.J., Ong, D., Kolobov, A.: BLOG:
Probabilistic models with unknown objects. In: IJCAI 2005 (2005)
19. Milch, B., Russell, S.: Extending Bayesian networks to the open-universe case. In:
Dechter, R., Geﬀner, H., Halpern, J. (eds.) Heuristics, Probability and Causality:A Tribute to Judea Pearl. College Publications (2010)
20. Milch, B., Russell, S.J.: General-purpose MCMC inference over relational struc-
tures. In: UAI 2006 (2006)
21. Pasula, H., Marthi, B., Milch, B., Russell, S.J., Shpitser, I.: Identity uncertainty
and citation matching. NIPS 15 (2003)
22. Pearl, J.: Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann (1988)23. Poole, D.: Probabilistic Horn abduction and Bayesian networks. AIJ 64, 81–129
(1993)
24. Poole, D.: First-order probabilistic inference. In: IJCAI 2003 (2003)25. Sato, T., Kameya, Y.: PRISM: A symbolic statistical modeling language. In: IJCAI
1997 (1997)
26. Van den Broeck, G.: Lifted Inference and Learning in Statistical Relational Models.
Ph.D. thesis, Katholieke Universiteit Leuven (2013)


Problems of Self-reference in Self-improving
Space-Time Embedded Intelligence
Benja Fallenstein and Nate Soares
Machine Intelligence Research Institute
2030 Addison St. #300, Berkeley, CA 94704, USA
{benja,nate }@intelligence.org
Abstract. By considering agents to be a part of their environment,
Orseau and Ring’s space-time embedded intelligence [10] is a better ﬁt
to the real world than the traditional agent framework. However, a self-
modifying AGI that sees future versions of itself as an ordinary part oftheenvironmentmay runinto problems of self-reference. We show that in
one particular model based on formal logic, naive approaches either lead
to incorrect reasoning that allows an agent to put oﬀ an important taskforever (the procrastination paradox ), or fail to allow the agent to justify
even obviously safe rewrites (the L¨obian obstacle ). We argue that these
problems have relevance beyond our particular formalism, and discusspartial solutions.
1 Introduction
Most formal models of artiﬁcial general intelligence (such as Hutter’s AIXI [5]
and the related formal measure of intelligence [6]) are based on the traditional
agent framework, in which the agent interacts with an environment, but is not
partof this environment. As Orseau and Ring [10] point out, this is reminiscent
of Cartesian dualism, the idea that the hu man mind is a non-physical substance
external to the body [11]. A real-world AGI, on the other hand, will be part ofthe physical universe, and will need to deal with the possibility that external
forces might observe or interfere with its internal operations.
The traditional separation of the agent from its environment seems even less
attractive when one considers I.J. Good’s idea that once AGI is suﬃciently
advanced,itmaybecomebetter thananyhumanatthe taskof making itself even
smarter,leadingtoan“intelligenceexplosion”andleavinghumanintelligencefar
behind [1]. It seems plausible that an AGI undergoing an intelligence explosion
may eventually want to adopt an architecture radically diﬀerent from its initial
one, such as one distributed over ma ny diﬀerent computers, where no single
entity fulﬁlls the agent’s role from the traditional framework [8]. A formal model
based on that framework cannot capture this.
How should one reason about such an agent? Orseau and Ring [10] have
proposed a formal model of space-time embedded intelligence to deal with this
complexity. Their model consists of a set Πofpolicies, describing the state of
the agent at a given point in time; an environment ρ(π
t+1|π1:t), giving the
B. Goertzel et al. (Eds.): AGI 2014, LNAI 8598, pp. 21–32, 2014.
c/circlecopyrtSpringer International Publishing Switzerland 2014
22 B. Fallenstein and N. Soares
probability that the policy at time ( t+ 1) will be πt+1, if the policies in the
previous timesteps were given by π1:t; a utility function u(π1:t)∈[0,1], giving
the “reward” at time t;d i s c o u n tf a c t o r s γtsuch that/summationtext∞
t=1γt<∞; and a subset
Π˜l⊆Πof policies of length ≤l, which describes the policies that can be
run on the machine initially used to implement the AGI. They then deﬁne the
optimal policy as the policy π∗∈Π˜lwhich maximizes the expectation of the
total discounted reward/summationtext∞
t=1γtu(π1:t), subject to π1=π∗and the transition
probabilities ρ(·|·).
Orseau and Ring propose their formalism as a tool for humans to reason
aboutAGIstheymightcreate;theyarguethattochooseanoptimal π∗“precisely
representsthe goalof those attempting to build anArtiﬁcial General Intelligence
in our world” [10]. By the same token, their formalism also represents the goalofa self-improving AGI capable undergoing an intelligence explosion, and could
be used by such an AGI to reason about potential self-modiﬁcations.
Unlike agents such as Hutter’s AIXI, which takes as giventhat future versions
ofitselfwillexistandwillchooseactionsthatmaximizeexpectedutility,anagent
using this framework would see future versions of itself simply as one possiblepart of the future environment, and would have to convince itself that these
future versions behave in desirable ways. This would allow the agent to consider
radical changes to its architecture on equal footing with actions that leave itscode completely unchanged, and to use the same tools to reason about both.
Such anagentwouldhavetobe abletoreasonaboutitsownbehaviororabout
the behaviorofanevenmorepowerfulvariant,andthis mayprovediﬃcult. From
the halting problem to Russell’s paradox to G¨ odel’s incompleteness theorems
to Tarski’s undeﬁnability of truth (a formal version of the liar paradox), logicand computer science are replete with examples showing that the ability of a
formal system reason about itself is often limited by diagonalization arguments,
with too much power quickly leading to inconsistency. Thus, one must be verycareful when specifying the mechanism by which a space-time embedded agent
reasons about its potential successors, or one may end up with a system that is
either too powerful (leading to inconsist encies, allowing self-modiﬁcations that
are obviously bad), or not powerful enough (leading to an agent unable self-
modify in ways that are obviously good).
With that in mind, in this paper we investigate in detail how a self-improving
AGI can use a model similarto Orseauand Ring’s to reasonabout its ownfuture
behavior. In particular, we consider agents that will only choose to perform anaction (such as a self-modiﬁcation) if they can ﬁnd a proof that this action is, in
a certain sense, safe. This architecture is very similar to that of Schmidhuber’s
G¨odel machines [12], and is one way to approach the problem of creating an
AGI that is, as Goertzel [4] puts it, probably beneﬁcial and almost certainly not
destructive .
Can such an AGI prove that it is safe for it to self-modify into an even more
powerful version? We show that diagonalization problems arise even if it tries to
prove the safety of a safe action that leaves it completely unchanged .
Self-reference in Space-Time embedded intelligence 23
Intuitively, one would expect that si nce a human mathematician can prove
that an AGI using this architecture will only take safe actions, the AGI wouldbe
able to use the same argument to prove that leaving itself unchanged is safe (by
showing that it will only take safe actions in the future). However, the human’s
proof uses the fact that if the AGI provesan action to be safe, this action
isactually safe. L¨ob’s theorem [7], a variant of G¨ odel’s second incompleteness
theorem, implies that no consistent theory Tas strong as Peano Arithmetic can
show that everything provable in Tis actually true:1The human mathematician
will only be able to carry out their proof in a strictly stronger proof system
than that used by the AGI, implying that the AGI will not be able to use this
proof. Yudkowsky and Herreshoﬀ [14] have called this the L¨obian obstacle to
self-modifying AI .
We consider two partial solutions to t his problem. First, given a sound the-
oryT0, it is possible to construct a sound extension T1which can show that
anything provable in T0is true. We show that an AGI using T1can prove the
safetyofsafeactionswhichmodifytheAGItouse T0.Onecaniteratethisprocess
(Tn+1proves that anything provable in Tnis true), but with this method, one
can only build an AGI that is able to justify a ﬁxed number of self-modiﬁcations
(an AGI using T7can only self-modify seven times). Second, it is possible to
ﬁnd an inﬁnite sequence of theories Tnin which every theory Tnproves that the
nexttheory,Tn+1,i sconsistent ; we show that under certain assumptions, this
is enough to allow an AGI using Tnto prove that it is safe to self-modify into
an AGI using Tn+1. However, neither of these solut ions seems fully satisfactory.
In this work, we focus on agents that reason about their environment through
formal logic (allowing for uncertainty only in the form of a probability distribu-
tion over diﬀerent environments). This is not a realistic assumption. There are
two reasons why we think it is still a reasonable starting point: First, although
formal logic is not a good tool for reasoning about the physical environment ,i t
isa natural tool for reasoning about the source code of future versions of an
agent, and it seems likely that self-improving AGIs will need to use some form of
formal logic if they want to to achieve very high conﬁdence in a formal property
of a future version’s source. Second, it seems likely that many features of the
following analysis will have analogs in frameworks not based on formal proofs.
We give an intuitive example, based on what Yudkowsky [13] calls the “procras-
tination paradox”, of how an agent that trusts future versions of itself too much
can reason itself into taking actions that are clearly unsafe. More formally, a
system due to Christiano et al. [1], which uses probabilities instead of proofs in
an attempt to circumvent the L¨ obian obstacle, attains “too much self-trust” and
succumbs to the procrastinationparadoxin almostthe sameform asproof-based
systems [3].
The authorsthink itlikely thatdiagonalizationproblemsofthe type discussed
here will in someform be relevant to future AGIs, and ﬁnd it plausible that
1More precisely: Tcannot prove /squareT/rightanglenwϕ/rightanglene→ϕfor every sentence ϕ,w h e r e/squareT/rightanglenwϕ/rightangleneis
the proposition in the language of Tstating that the sentence ϕis provable in T.
24 B. Fallenstein and N. Soares
examining partial solutions in proof-based systems can lead to insights that will
help address these problems, whatever exact form they end up taking.
2 A Myopic View of Space-Time Embedded Intelligence
In this section, we introduce the formal model of space-time embedded intelli-
gence used in this paper. As in the traditional agent framework, we assume that
there are ﬁnite sets AandOof actions and observations. However, instead of
considering sequences of actions and observations, we take a “myopic” view that
focuses even more on the initial choice of the AGI or of its programmers than
Orseau and Ring’s framework does, and assume that the agent makes only a
single observation o∈Oand chooses a single action a∈A.Apolicyis thus a
function π∈Π:=AO. An action aincludes both external activities (such as a
command to move a robot’s arm) and the internal state of the agent after mak-
ing the choice, making it unnecessary to draw a sharp distinction between the
two. Thus, a choice to self-modify is treated no diﬀerently from a choice to per-
form some external activity, and even a choice notto self-modify is conceptually
no diﬀerent from a choice to self-modify.
We further assume that there is a function EU : O×A→ [0,1] such that
EU(o,a) speciﬁes the expected utility when choosing action aafter making ob-
servation o. This function contains essentially all of our agent’s model of the
environment: intuitively, it combines a probabilistic model of what happens in
the world if diﬀerent actions are taken with a von Neumann-Morgenstern [9]
utility function which maps world histories to real numbers. (It is no restriction
to require that the utility function take values in [0 ,1], since von Neumann-
Morgenstern utilities can always be represented in this way.) We do not suppose
that EU is ever evaluated on particular oanda; rather, we merely assume that
the agent has access to a deﬁnition of this function in the language of set theory.
The suggester-veriﬁer architecture. We consider agents with a policy πfb∈Π,
thefallback policy , whichspeciﬁesadefaultaction πfb(o)totakewhenmakingob-
servation o, and an intelligent but not fully trusted component, the suggester σ,
which tries to improve on this default action.
In this paper, for consistency, we will always enclose formulas of ﬁrst-order
logicin quotes, /rightanglenw·/rightanglene(except on the right-hand side of a turnstile, as in T/turnstileleft0=0 ) ,
and we will use overbars to denote an expression in the language of set theory
denoting the valueof a metalanguagevariable: For example, if m=2a n dn=1 ,
then/rightanglenw
m>
n/rightanglenedenotes the formula /rightanglenwS(S(0))>S(0)/rightanglene,w h e r eS(·) is the successor
function.2
With this convention, a suggester is a deterministic algorithm taking an ob-
servation o∈Oand returning either an error value or an action a∈Atogether
with a formal proof of the sentence /rightanglenwEU(
o,
a)≥EU(
o,π fb(
o))/rightanglenein some formal
theoryT,su c has T= ZFC. Note that the exactsyntacticalformof this sentence
2Technically, the language of set theory does of course not contain a symbol for the
successor function, but this can be eliminated in the usual manner.
Self-reference in Space-Time embedded intelligence 25
depends ontheagent’sobservation oandthe suggester’sproposedaction a,si n c e
oand
adenote the numerals representing the value of these metavariables.
When an agent using the suggester-ver iﬁer architecture receives an action /
proof pair ( a,p) from its suggester component, it checks the proof p, and exe-
cutes the action aifpis a valid proof of /rightanglenwEU(
o,
a)≥EU(
o,π fb(
o))/rightanglene. Following
Schmidhuber [12], we call a sentence of this form a target theorem .I ft h es u g -
gester returns the error value, or if pis not a valid proof of the target theorem,
the agent executes the fallback action πfb(o).
We write sv T(/rightanglenwEU/rightanglene,σ,π fb)∈Πfor the policy deﬁned by this suggester-
veriﬁer architecture , where we use the G¨ odel quotes around EU to indicate
that EU only needs to be given in the form of a deﬁnition in the language
of set theory, whereas σandπfbneed to be executable programs.3Here,Tis the
theory in which the suggester mu st prove the target theorem.
Intuitively, we have EU/parenleftbig
o,svT(/rightanglenwEU/rightanglene,σ,π fb)(o)/parenrightbig
≥EU(o,π fb(o)), since either
svT(/rightanglenwEU/rightanglene,σ,π fb)(o)=πfb(o)o rTproves/rightanglenwEU(
o,
a)≥EU(
o,π fb(
o))/rightanglene,w h e r eais
the value of sv T(/rightanglenwEU/rightanglene,σ,π fb)(o); thus, as long as Tis sound (i.e., if everything
Tprovesis actually true), the suggester-v eriﬁerarchitecture does at least as well
as the fallback policy.
This can be understood as the “almost certainly not destructive” part of
Goertzel’s phrasing of the AGI safety problem: One may not be able to write
downaconcretestep-by-stepfallbackpolicythatsolvestheworld’smostpressing
problems, but a simpler fallback policy such as “do nothing” is at least unlikely
todestroythe world. Goertzel’s “probably beneﬁcial”, on the other hand, is
the suggester’s job. The suggester-veriﬁer architecture would allow an AGI’s
programmers to choose a suggester that they hopewill come up with a better
policy, even if they do not entirely trust it.
This architecture is very similar to that of a G¨ odel machine [12]; it becomes
essentially the same if the fallback policy πfbuses the suggester-veriﬁer archi-
tecture to choose what actions to take in future timesteps. However, we will
see below that this makes it very diﬃcult for the suggester to prove its target
theorem.
3 Self-modiﬁcation
In order to model self-modiﬁcation, consider an agent that has made a speciﬁc
observation o∈Oand is considering a speciﬁc action a∈A, which will (among
other things) have the eﬀect of leaving the agent in a state in which it will
use policy π/prime∈Πafter making its next observation. In other words, the agent
expects that after it executes action a, there will be an agent in the environment
which receives another observation o/prime∈Oand takes another action π/prime(o/prime).
To formalize this, we assume that there is a probability distribution P/primeover
O ∪ {∗},w h e r e∗/∈Ois a special error value, such that P/prime(o/prime) is the agent’s
3Our use of the symbol πfbfor the argument of sv Tis an abuse of notation, since sv T
depends on the source code ofπfbrather than only on the function πfb:O→A,a s
πfbis referenced inside the target theorem.
26 B. Fallenstein and N. Soares
probability that, in the next “timestep”, the environment will contain an agent
which makes observation o/primeand executes action π/prime(o/prime).P/prime(∗) denotes the prob-
ability that executing adoes not in fact cause there to be an agent implement-
ingπ/prime, for example because the agent is about to be destroyed. One can think
of this as the “Cartesian boundary” between the agent and the environment
breaking down.
We furtherassumethatthereisavalueEU/prime
∗∈[0,1]givingtheexpectedutility
the agent expects to receive if the Cartesian boundary is broken, and a function
EU/prime:O×A→ [0,1] such that EU/prime(o/prime,a/prime) is the expected utility it expects
to receive if the “oﬀspring” agent makes observation o/primeand is counterfactually
assumed to take action a/prime: that is, we assume that the agent models the world in
a waythat allowsit to evaluatewhat wouldhappen if its oﬀspringtook an action
diﬀerent from its actual one. (For example, the agent could model its world as
aB a y e sn e t . )
We assume that EU/prime
∗and EU/primeare deﬁnable in the language of set theory, and
that the agent is able to prove
T/turnstileleftEU(
o,
a)=P/prime(∗)·EU/prime
∗+/summationdisplay
o/prime∈OP/prime(o/prime)·EU/prime(o/prime,π/prime(o/prime)),(1)
since this equation simply rewrites the expected utility of action aas a sum over
allcasesin O∪{∗},using the factthat EU/prime(o/prime,π/prime(o/prime)) returnsthe expected utility
when counterfactually assuming that the oﬀspring performs action π/prime(o/prime), which
happens to be the action the oﬀspring will actually perform if it observes o/prime.
We areparticularlyintereste d in the casewhere executing aimplements a pol-
icyπ/primethatagainusesthesuggester-veriﬁerarchitecture: π/prime=s v T/prime(/rightanglenwEU/prime/rightanglene,σ/prime,π/prime
fb)
for some theory T/prime, suggester σ/prime, and fallback policy π/prime
fbused by the oﬀspring
agent. We further restrict consideration to policies where
T/turnstileleftP/prime(∗)·EU/prime
∗+/summationdisplay
o/prime∈OP/prime(o/prime)·EU/prime(o/prime,π/prime
fb(o/prime))≥EU(
o,π fb(
o)).(2)
This condition says that if the agent takes action a, adopting policy π/prime,t h e n
assuming counterfactually that the “oﬀspring” agent takes the action π/prime
fb(o/prime)
that the new fallback policy would choose (whether or not this is equal to the
oﬀspring agent’s actual action π/prime(o/prime)), then the expected utility is at least that
obtained by the “parent” agent’s fallback policy πfb. If, in addition,
T/turnstileleft∀o/prime∈O:E U/prime(o/prime,π/prime(o/prime))≥EU/prime(o/prime,π/prime
fb(o/prime)), (3)
then (1) and (2) together imply that Tproves the target theorem /rightanglenwEU(
o,
a)≥
EU(
o,π fb(
o))/rightanglene, allowing the parent’s suggester σto justify choosingthe action a,
as desired.
But isit provablein Tthat EU/prime(o/prime,π/prime(o/prime))≥EU/prime(o/prime,π/prime
fb(o/prime))? When justifying
the initial choice of the suggester-veriﬁ er architecture, we pr oved in our informal
metalanguage that EU/parenleftbig
o,svT(/rightanglenwEU/rightanglene,σ,π fb)(o)/parenrightbig
≥EU(o,π fb(o)) by arguing that
the suggester-veriﬁer eith er returns the same action as πfb, or the suggester has
Self-reference in Space-Time embedded intelligence 27
foundaproofin Tthatthe actionchosenisatleastasgoodasthatchosenby πfb.
We would like the parent agent to be able to justify creating an oﬀspring of the
same architecture by the same argument that we used to justify this architecture
in the ﬁrst place.
Here we run into the “L¨ obian obstacle” [14]. Our argument in the metalan-
guage required the fact that we believe that Tissound: if there is a proof in T
that/rightanglenwEU(
o,
a)≥EU(
o,π fb(
o))/rightanglene, then we believe that this inequality does in fact
hold. Write /squareT/prime/rightanglenwϕ/rightanglenefor the proposition in the language of set theory that states
thatϕis provable in T/prime. In order to carry out the intended proof in T,Tneeds
to be able to prove that T/primeis sound. we assume that for every predicate /rightanglenwϕ(n)/rightanglene
in the language of set theory,
T/turnstileleft∀n∈N:/squareT/prime/rightanglenwϕ(
n)/rightanglene→ϕ(n)( 4 )
SinceOandAare ﬁnite sets, we can identify them with subsets of N,a n d
conclude that for every predicate /rightanglenwϕ(o/prime,a/prime)/rightanglene,
T/turnstileleft∀o/prime∈O,a/prime∈A:/squareT/prime/rightanglenwϕ(
o/prime,
a/prime)/rightanglene→ϕ(o/prime,a/prime). (5)
Speciﬁcally, we apply this to /rightanglenwϕ(o/prime,a/prime)/rightanglene=/rightanglenwEU/prime(o/prime,a/prime)≥EU/prime(o/prime,π/prime
fb(o/prime))/rightanglene;i n
other words, if the inequality is provable, it is true.
Now, the parent agent’s suggester can formally carry out the argument that
we earlier gave in the metalanguage: by inspecting the source code of the new
suggester-veriﬁer π/prime,i tc a nb es h o w nt h a t
T/turnstileleft∀o/prime∈O:/squareT/prime/rightanglenwϕ(
o/prime,π/prime(
o/prime))/rightanglene∨(π/prime(o/prime)=π/prime
fb(o/prime)), (6)
since either the new suggester σ/primereturns a proof of a target theorem /rightanglenwϕ(
o/prime,
a/prime)/rightanglene,
in which case π/prime(o/prime)=a/primeand thus /rightanglenwϕ(
o/prime,π/prime(o/prime))/rightangleneis provable, or else σ/primefails, in
which case π/prime(o/prime) equals the fallback action π/prime
fb(o/prime).
Together, (5) and (6) imply
T/turnstileleft∀o/prime∈O:E U/prime(o/prime,π/prime(o/prime))≥EU/prime(o/prime,π/prime
fb(o/prime)) (7)
(sinceπ/prime(o/prime)=π/prime
fb(o/prime) clearly implies EU/prime(o/prime,π/prime(o/prime)) = EU/prime(o/prime,π/prime
fb(o/prime)), and
hence also the inequality). Finally, (1), (2) and (7) together imply that Tproves
the target theorem /rightanglenwEU(
o,
a)≥EU(
o,π fb(
o))/rightanglene.
4T h e L ¨ obian Obstacle and the Procrastination Paradox
The proof in the previous section shows that it’s possible to prove the target
theorem /rightanglenwEU(
o,
a)≥EU(
o,π fb(
o))/rightangleneinTif assumptions (1), (2) and (4) are
satisﬁed. Assumptions (1) and (2) are abo ut the environment, essentially stat-
ing that the action asuccessfully executes a self-modiﬁcation and that the new
fallback policy does at least as well as the old fallback policy, respectively. As-
sumption (4), on the other hand, is an assumption about the ﬁrst-order theories
TandT/prime.
28 B. Fallenstein and N. Soares
Naively, we might expect that one should be able to have T=T/prime: that our
agent would be able to keep using the same theory when rewriting itself. By thediagonal lemma, it is possible to construct a theory T=T
/primethat extends ZFC
and satisﬁes (4). However, if we substitute ϕ(n)≡(0 = 1) in (4), we see that
this axiom is saying “if a contradiction is provable in T/prime, then a contradiction
is true”, which is equivalent to “no contradiction is provable in T/prime”, or simply
“T/primeis consistent”. Hence, if we set T=T/prime, then (4) says that Tshows its own
consistency, which by G¨ odel’s second incompleteness theorem implies that T
is inconsistent. L¨ ob’s theorem [7] sharpens this by showing that whenever (4)
holds for T=T/primeand any ϕ(n), then we must already have T/turnstileleft/rightanglenw∀n:ϕ(n)/rightanglene.W e
certainly don’t want Tto prove /rightanglenw∀o/prime∈O,a/prime∈A:E U/prime(o/prime,a/prime)≥EU/prime(o/prime,π/prime
fb(o/prime))/rightanglene,
which would say that the new fallback policy is the worst of all possible policies!
G¨odel’s and L¨ ob’s theorems make use of much of the structure of proofs in
ﬁrst-orderlogic,andonemightwonderwhe thertheseproblemsaremereartifacts
of our use of formal proofs in the suggester-veriﬁer architecture. The following
version of the argument, however, suggests that the problem extends beyondthis context.
Imagine an agent which trusts its own reasoning and has a goal that a certain
button be pressed. This agent does not care whenthis button is pressed. The
agent must choose between two possible actions: Either press the button im-
mediately, or do nothing, in which case its source code remains unchanged andit will ﬁnd itself in exactly the same situation in the next timestep. The agent
reasons as follows: “Suppose that I don’t press the button now. Then either I
pressthe button in the next timestep, orI must havefound anargumentshowingthat even if I do not press the button in the next round, it will still get pressed
eventually. Either way, the button will get pressed eventually. Therefore, it is
ﬁne not to press the button now.” Thus, the agent does not press the button in
the current timestep; and all subsequent timesteps are exactly the same, imply-
ing that the button is never pressed. This is called the procrastination paradox
[13] by analogy to a human who is perenially planning to get around to doing a
certain piece of work “later”.
Given (4) and T=T
/prime, it is not diﬃcult to formalize this reasoning using the
prooffromtheprevioussection(weset O:={ˆo};A:={1,0}forpressingandnot
pressing the button; EU(ˆ o,1) := 1; and EU(ˆ o,0) := EU(ˆ o,svT(/rightanglenwEU/rightanglene,σ,π fb)),
withπfb(ˆo) = 1). But at the same time, the informal reasoning does not depend
on details of how our AGI works; it merely requires a suﬃcient amount of trust
in the reasoning of the agent’s successor. In fact, diﬀerent formal versions of the
same intuitive argument have been used in [13] and [3] to show problems with
two distinct attempts at avoiding the L¨ obian obstacle.
What happens if we set T=T/prime= ZFC, which is generally assumed to be
consistent? In this case, (4) fails to hold, our proof does not go through, and
we have run into the L¨ obian obstacle headﬁrst: our agent is not able to rewrite
itself. But if we consider the situation of the procrastinationparadox,it becomes
apparentthat there is an upside to this shortcoming,in that failure to choosethe
rewriting action 0 in this case means choosing the action 1 and actually pressing
Self-reference in Space-Time embedded intelligence 29
the button. Thus, we can see the L¨ obian obstacle as the way that ZFC solves
the procrastination paradox. A good solution to the L¨ obian obstacle must take
care not to reintroduce this paradox.
5 Partial Solutions
Because of L¨ ob’s theorem, we cannot hope to have T=T/primeif our proof is to go
through: suggester-veriﬁer agents cannot keep using the same theory to verify
proofs as they self-modify. The simplest way to push our proof through, then, is
one discussed in [14]: Let T0:= ZFC and let Tn+1be the extension of Tnby the
axiom schema /rightanglenw∀n∈N:/squareTn/rightanglenwϕ(
n)/rightanglene→ϕ(n)/rightanglene, for every formula ϕ(n). One can
easily show that Tnis consistent for every nif one assumes ZFC + the existence
of a strongly inaccessible cardinal, the same assumption usually made to show
the consistency of ZFC. Th en we can set, for example, T=T7andT/prime=T6,
which allows the parent agent’s suggester to use the axiom schema to conclude
the target theorem. This makes our proof go through, but it implies that our
agent can only rewrite itself seven times before it reaches T0and cannot use
the same argument to justify further rewrites. This “solution” does not seem
particularly satisfactory, but it may point the way to better insights.
A more promising result [2] shows that it is possible to have an inﬁnite se-
quence of sound theories Tn(i.e., theories proving only true things) such that
we have
Tn/turnstileleft∀k∈N:/squareTn+1/rightanglenwϕ(
k)/rightanglene→ϕ(k)( 8 )
for allϕ(k)o ft h ef o r m ∀/lscript∈N.ψ(k,/lscript), where ψ(k,/lscript)i s primitive recursive ,
i.e., its truth value is a primitive recursive function of kand/lscript. This is the
case, for example, if the environment is computable with a bounded amount
of computational resou rces per timestep, and ψ(k,/lscript) is some property of what
happens in the ﬁrst /lscripttimesteps, such as “up to time /lscript, the AI does not destroy
the world”. Since in this case, the indices ngouprather than down, an agent
using this system is able to justify an indeﬁnite sequence of rewrites.
An agent using the suggester-veriﬁer architecture can make use of this result
if its utility function has the form of a discounted sum over diﬀerent timesteps,
similar to Orseau and Ring’s formalism: Suppose that EU/primecan be written as
EU/prime(o/prime,a/prime)=∞/summationdisplay
t=0γt·EU/prime(o/prime,a/prime,t),
where EU/prime(o/prime,a/prime,t)∈[0,1] is the expected “utility” received ttimesteps into
the future if in the next timestep, the oﬀspring agent makes observation o/primeand
takes action a/prime,a n dγtis a discount factor such that/summationtext∞
t=0γt<∞. Then, the
predicate /rightanglenwϕ(o/prime,a/prime)/rightanglene=/rightanglenwEU/prime(o/prime,a/prime)≥EU/prime(o/prime,π/prime
fb(o/prime))/rightanglenefor which we need (5) to
hold is provably equivalent to
/rightanglenw∞/summationdisplay
t=0γt·EU/prime(o/prime,a/prime,t)≥∞/summationdisplay
t=0γt·EU/prime(o/prime,π/prime
fb(o/prime),t)/rightanglene, (9)
30 B. Fallenstein and N. Soares
which in turn is provably equivalent to
/rightanglenw∀T∈N:T/summationdisplay
t=0γt·EU/prime(o/prime,a/prime,t)+∞/summationdisplay
t=T+1γt≥T/summationdisplay
t=0γt·EU(o/prime,π/prime
fb(o/prime),t)/rightanglene.(10)
(To see this, ﬁrst note that (9) is the limit of the inequality in (10) for T→∞;
thus, (10) implies (9). In the other direction, note that γt≥γt·EU/prime(o/prime,a/prime,t)≥0
for allo/prime,a/prime,a n dt.) Moreover, if EU/prime(·,·,·),γtandc:=/summationtext∞
t=0γtare all rational-
valued and primitive recursive, (10) ha s the form required in (8), because the
inﬁnite sum/summationtext∞
t=T+1γtcan be expressed as c−/summationtextT
t=0γt. Thus, an agent using
Tnis able to show that if a later version of the agent proves (9) in Tn+1,t h e n
(9) is in fact true, allowing the proof presented in Section 3 to go through.
6G ¨ odel Machines
Our suggester-veriﬁerformalism is very similar to Schmidhuber’s G¨ odel machine
architecture[12],anditisnaturaltowonderwhetherourtechniquesforjustifyingself-modiﬁcationcanbeadoptedtoSchmidhuber’sframework.Unfortunately,we
run into diﬃculty when attempting to do so.
While the G¨ odel machine is somewhat more realistic than our formalism (it
avoids our big discrete timesteps and has other nice properties), it is simple
enough to tweak our formalism to produce something very similar to a G¨ odel
machine. In our terminology, a G¨ odel machine is a suggester-veriﬁer agent with
a fallback policy that rewrites the agent into another suggester-veriﬁer agent
(with a fallback policy that rewrites the agent into yet another suggester-veriﬁeragent, and so on). Note that we consider the agent taking no action to be a
special case of this, where the agent rewrites itself into an identical agent.
Under this interpretation, it is clear that the obstacles we have encountered
apply to G¨ odel machines as well. Consider a G¨ odel machine G
1whose fallback
policywould“rewrite”itintoanotherG¨ odelmachine G2withthesamesuggester
(proof searcher, in Schmidhuber’s terminology). G1’s suggester now wants to
prove that it is acceptable to i nstead rewrite itself into G/prime
2,aG ¨odel machine
with a very slightly modiﬁed proof searcher. It must prove that G/prime
2will obtain
at least as much utility as G2. In order to do so, naively we would expect that
G/prime
2will again only execute rewrites if its proof searcher has shown them to be
useful; but clearly, this runs into the L¨ obian obstacle, unless G1can show that
theorems proven by G/prime
2are in fact true.
We could attempt to use the techniques in this paper to make the proof go
through, but this would r equire us to accept that G/prime
2uses a strictly weaker proof
system than G1, which seems like a departure from the framework envisioned
by Schmidhuber. Worse, even if we do so, it would not at all be clear howto prove (2): In the G¨ odel machine setting, π
fbandπ/prime
fbwould themselves be
G¨odel machines and hence suggester-veriﬁers. Thus, proving (2) would again
requirereasoningaboutagentsthatsearchforproofs:Wehavereducedourinitialproblemtoadiﬀerentonethatisjust asdiﬃcult. (Inourownformalism,wesolve
Self-reference in Space-Time embedded intelligence 31
this problem by choosing πfbandπ/prime
fbto be “object-level” policies involving no
proof search—in the simplest case, mere lookup tables that specify which action
should be taken given the observation.)
7 Conclusions
In this paper, we have introduced a con crete formalism for space-time embed-
ded intelligence that a proof-based AGI can use to reason about its own future
behavior. We have shown how, under certain assumptions, an agent using this
formalism is able to justify minor self-modiﬁcations that leave its overall archi-
tecture intact.
However, in doing so, we have seen that naive approaches run into one of
two major problems of self-reference: the procrastination paradox , which allows
an agent to put oﬀ an important task forever, or the L¨obian obstacle ,w h i c h
prevents an agent from justifying even clearly safe rewrites. Hurdles such as
these should make the reader wary of accep ting intuitively plausible formalisms
allowing for self-modiﬁcation before seeing a formal version that provably avoids
these obstacles. We discussed partial solutions, but ﬁnding a fully satisfactory
solution remains an open problem.
References
1. Christiano, P., Yudkowsky, E., Herreshoﬀ, M., Barasz, M.: Deﬁnabil-
ity of truth in probabilistic logic (2013), http://intelligence.org/files/
DefinabilityOfTruthInProbabilisticLogic-EarlyDraft.pdf
2. Fallenstein, B.: An inﬁnitely descending sequence of sound theories each proving
the next consistent (2013),
https://intelligence.org/files/ConsistencyWaterfall.pdf
3. Fallenstein, B.: Procrastination in probabilistic logic (2014),
https://intelligence.org/files/ProbabilisticLogicProcrastinates.pdf
4. Goertzel, B.: Golem: Toward an agi meta-architecture enabling both goal preser-
vation and radical self-improvement (2010), http://goertzel.org/GOLEM.pdf
5. Hutter, M.: Universal Artiﬁcial Intelligence: Sequential Decisions based on Algo-
rithmic Probability. Springer, Berlin (2005)
6. Legg, S., Hutter, M.: A formal measure of machine intelligence. In: Proc. 15th
Annual Machine Learning Conference of Belgium and the Netherlands (Benelearn
2006), Ghent, Belgium, pp. 73–80 (2006)
7. Lob, M.H.: Solution of a problem of Leon Henkin. J. Symb. Log. 20(2), 115–118
(1955)
8. Muehlhauser, L., Orseau, L.: Laurent Orseau on Artiﬁcial General Intelligence
(interview) (2013),
http://intelligence.org/2013/09/06/laurent-orseau-on-agi/
9. Neumann, L.J., Morgenstern, O.: Theory of games and economic behavior, vol. 60.
Princeton University Press, Princeton (1947)
10. Orseau, L., Ring, M.: Space-time embedded intelligence. In: Bach, J., Goertzel, B.,
Ikl´e,M.(eds.)AGI2012.LNCS,vol.7716,pp.209–218. Springer,Heidelberg(2012)
32 B. Fallenstein and N. Soares
11. Robinson, H.: Dualism. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philos-
ophy. Winter 2012 edition (2012)
12. Schmidhuber, J.: Ultimate cognition ` al aG ¨odel. Cognitive Computation 1(2),
177–193 (2009)
13. Yudkowsky, E.: The procrastination paradox (2013), https://intelligence.org/
files/ProcrastinationParadox.pdf
14. Yudkowsky, E., Herreshoﬀ, M.: Tiling agents for self-modifying AI, and the L¨ obian
obstacle (2013)