Vingean Reﬂection: Reliable Reasoning for Self-Improving Agents

Vingean Re
ection: Reliable Reasoning for Self-Improving Agents
Benja Fallenstein andNate Soares
Machine Intelligence Research Institute
fbenja,nate g@intelligence.org
Abstract
Today, human-level machine intelligence is in
the domain of futurism, but there is every rea-
son to expect that it will be developed eventu-
ally. Once articial agents become able to im-
prove themselves further, they may far surpass
human intelligence, making it vitally important
to ensure that the result of an \intelligence ex-
plosion" is aligned with human interests. In this
paper, we discuss one aspect of this challenge:
ensuring that the initial agent's reasoning about
its future versions is reliable, even if these fu-
ture versions are far more intelligent than the
current reasoner. We refer to reasoning of this
sort as Vingean re
ection .
A self-improving agent must reason about the
behavior of its smarter successors in abstract
terms, since if it could predict their actions in
detail, it would already be as smart as them.
This is called the Vingean principle , and we ar-
gue that theoretical work on Vingean re
ection
should focus on formal models that re
ect this
principle. However, the framework of expected
utility maximization, commonly used to model
rational agents, fails to do so. We review a body
of work which instead investigates agents that
use formal proofs to reason about their succes-
sors. While it is unlikely that real-world agents
would base their behavior entirely on formal
proofs, this appears to be the best currently
available formal model of abstract reasoning,
and work in this setting may lead to insights ap-
plicable to more realistic approaches to Vingean
re
ection.
1 Introduction
In a 1965 article, I.J. Good introduced the concept of
an \intelligence explosion" (Good 1965):
Let an ultraintelligent machine be dened
as a machine that can far surpass all the
Research supported by the Machine Intelligence Research
Institute (intelligence.org). Published as Technical report
2015{2.intellectual activities of any man however
clever. Since the design of machines is one
of these intellectual activities, an ultraintel-
ligent machine could design even better ma-
chines; there would then unquestionably be
an `intelligence explosion,' and the intelli-
gence of man would be left far behind. Thus
the rst ultraintelligent machine is the last
invention that man need ever make.
Almost fty years later, a machine intelligence that
is smart in the way humans are remains the subject of
futurism and science ction. But barring global catas-
trophe, there seems to be little reason to doubt that
humanity will eventually create a smarter-than-human
machine. Whether machine intelligence can really leave
the intelligence of biological humans far behind is less
obvious, but there is some reason to think that this
may be the case (Bostrom 2014): First, the hardware
of human brains is nowhere close to physical limits; and
second, not much time has passed on an evolutionary
timescale since humans developed language, suggesting
that we possess the minimal amount of general intelli-
gence necessary to develop a technological civilization,
not the theoretical optimum.
It's not hard to see that if building an articial su-
perintelligent agent will be possible at some point in
the future, this could be both a great boon to human-
ity and a great danger if this agent does not work as
intended (Bostrom 2014; Yudkowsky 2008). Imagine,
for example, a system built to operate a robotic labora-
tory for nding a cure for cancer; if this is its only goal,
and the system becomes far smarter than any human,
then its best course of action (to maximize the proba-
bility of achieving its goal) may well be to convert all of
Earth into more computers and robotic laboratories|
and with sucient intelligence, it may well nd a way
to do so. This argument generalizes, of course: While
there is no reason to think that an articial intelligence
would be driven by human motivations like a lust for
power, anygoals that are not quite ours would place it
at odds with our interests.
How, then, can we ensure that self-improving
smarter-than-human machine intelligence, if and when
1
it is developed, is benecial to humanity?
Extensive testing may not be sucient. A smarter-
than-human agent would have an incentive to pretend
during testing that its goals are aligned with ours, even
if they are not, because we might otherwise attempt to
modify it or shut it down (Bostrom 2014). Hence, test-
ing would only give reliable information if the system
is not yet suciently intelligent to deceive us. If, at
this point, it is also not yet intelligent enough to realize
that its goals are at odds with ours, a misaligned agent
might pass even very extensive tests.
Moreover, the test environment may be very dier-
ent from the environment in which the system will ac-
tually operate. It may be infeasible to set up a testing
environment which allows a smarter-than-human sys-
tem to be tested in the kinds of complex, unexpected
situations that it might encounter in the real world as
it gains knowledge and executes strategies that its pro-
grammers never conceived of.
For these reasons, it seems important to have a the-
oretical understanding of why the system is expected to
work, so as to gain high condence in a system that will
face a wide range of unanticipated challenges (Soares
and Fallenstein 2014a). By this we mean two things:
(1) a formal specication of the problem faced by the
system; and (2) a rm understanding of why the sys-
tem (which must inevitably use practical heuristics) is
expected to perform well on this problem.
It may seem odd to raise these questions today, with
smarter-than-human machines still rmly in the do-
main of futurism; we can hardly verify that the heuris-
tics employed by an articial agent work as intended
before we even know what these heuristics are. How-
ever, Soares and Fallenstein (2014a) argue that there
isfoundational research we can do today that can help
us understand the operation of a smarter-than-human
agent on an abstract level.
For example, although the expected utility maxi-
mization framework of neoclassical economics has seri-
ous shortcomings in describing the behavior of a realis-
tic articial agent, it is a useful starting point for asking
whether it's possible to avoid giving a misaligned agent
incentives for manipulating its human operators (Soares
and Fallenstein 2015). Similarly, it allows us to ask
what sorts of models of the environment would be able
to deal with the complexities of the real world (Hutter
2000). Where this framework falls short, we can ask
how to extend it to capture more aspects of reality, such
as the fact that an agent is a part of its environment
(Orseau and Ring 2012), and the fact that a real agent
cannot be logically omniscient (Gaifman 2004; Soares
and Fallenstein 2015). Moreover, even when more re-
alistic models are available, simple models can clarify
conceptual issues by idealizing away diculties not rel-
evant to a particular problem under consideration.
In this paper, we review work on one foundational
issue that would be particularly relevant in the context
of an intelligence explosion|that is, if humanity does
not create a superintelligent agent directly, but insteadcreates an agent that attains superintelligence through
a sequence of successive self-improvements. In this case,
the resulting superintelligent system may be quite dif-
ferent from the initial veried system. The behavior of
the nal system would depend entirely upon the abil-
ity of the initial system to reason correctly about the
construction of systems more intelligent than itself.
This is no trouble if the initial system is extremely
reliable: if the reasoning of the initial agent were at
least as good as a team of human AI researchers in all
domains, then the system itself would be at least as safe
as anything designed by a team of human researchers.
However, if the system were only known to reason well
inmost cases, then it seems prudent to verify its rea-
soning specically in the critical case where the agent
reasons about self-modications.
At least intuitively, reasoning about the behavior
of an agent which is more intelligent than the reasoner
seems qualitatively more dicult than reasoning about
the behavior of a less intelligent system. Verifying that
a military drone obeys certain rules of engagement is
one thing; verifying that an articial general would suc-
cessfully run a war, identifying clever strategies never
before conceived of and deploying brilliant plans as ap-
propriate, seems like another thing entirely. It is cer-
tainly possible that this intuition will turn out to be
wrong, but it seems as if we should at least check : if
extremely high condence must be placed on the ability
of self-modifying systems to reason about agents which
are smarter than the reasoner, then it seems prudent to
develop a theoretical understanding of satisfactory rea-
soning about smarter agents. In honor of Vinge (1993),
who emphasizes the diculty of predicting the behavior
of smarter-than-human agents with human intelligence,
we refer to reasoning of this sort as Vingean re
ection .
2 Vingean Re
ection
The simplest and cleanest formal model of intelligent
agents is the framework of expected utility maximiza-
tion. Given that this framework has been a productive
basis for theoretical work both in articial intelligence
in general, and on smarter-than-human agents in par-
ticular, it is natural to ask whether it can be used to
model the reasoning of self-improving agents.
However, although it can be useful to consider mod-
els that idealize away part of the complexity of the real
world, it is not dicult to see that in the case of self-
improvement, expected utility maximization idealizes
away too much. An agent that can literally maximize
expected utility is already reasoning optimally; it may
lack information about its environment, but it can only
x this problem by observing the external world, not
by improving its own reasoning processes.
A particularly illustrative example of the mis-
match between the classical theory and the problem
of Vingean re
ection is provided by the standard tech-
nique of backward induction , which nds the optimal
2
policy of an agent facing a sequential decision problem
by considering every node in the agent's entire decision
tree. Backward induction starts with the leaves, gur-
ing out the action an optimal agent would take in the
last timestep (for every possible history of what hap-
pened in the previous timesteps). It then proceeds to
compute how an optimal agent would behave in the
second-to-last timestep, given the behavior in the last
timestep, and so on backward to the root of the decision
tree.
A self-improving agent is supposed to become more
intelligent as time goes on. An agent using backward
induction to choose its action, however, would have to
compute its exact actions in every situation it might
face in the future in the very rst timestep|but if it is
able to do that, its initial version could hardly be called
less intelligent than the later ones!
Since we are interested in theoretical understand-
ing, the reason we see this as a problem is not that
backward induction is impractical as an implementa-
tion technique. For example, we may not actually be
able to run an agent which uses backward induction
(since this requires eort exponential in the number of
timesteps), but it can still be useful to ask how such
an agent would behave, say in a situation where it may
have an incentive to manipulate its human operators
(Soares and Fallenstein 2015). Rather, the problem is
that we are trying to understand conceptually how an
agent can reason about the behavior of a more intelli-
gent successor, and an \idealized" model that requires
the original agent to already be as smart as its succes-
sors seems to idealize away the very issue we are trying
to investigate.
The programmers of the famous chess program Deep
Blue, for example, couldn't have evaluated dierent
heuristics by predicting, in their own heads, where each
heuristic would make Deep Blue move in every possible
situation; if they had been able to do so, they would
have been able to play world-class chess themselves.
But this does not imply that they knew nothing about
Deep Blue's operation: their abstract knowledge of the
code allowed them to know that Deep Blue was trying
to win the game rather than to lose it, for example.
Like Deep Blue's programmers, any articial agent
reasoning about smarter successors will have to do so
using abstract reasoning, rather than by computing out
what these successors would do in every possible situa-
tion. Yudkowsky and Herresho (2013) call this obser-
vation the Vingean principle , and it seems to us that
progress on Vingean re
ection will require formal mod-
els that implement this principle, instead of idealizing
the problem away.
This is not to say that expected utility maximization
has no role to play in the study of Vingean re
ection.
Intuitively, the reason the classical framework is unsuit-
able is that it demands logical omniscience: It assumes
that although an agent may be uncertain about its en-
vironment, it must have perfect knowledge of all math-
ematical facts, such as which of two algorithms is moreecient on a given problem or which of two bets leads
to a higher expected payo under a certain computable
(but intractable) probability distribution. Real agents,
on the other hand, must deal with logical uncertainty
(Soares and Fallenstein 2015). But many proposals for
dealing with uncertainty about mathematical facts in-
volve assigning probabilities to them, which might make
it possible to maximize expected utility with respect to
the resulting probability distribution.
However, while there is some existing work on for-
mal models of logical uncertainty (see Soares and Fall-
enstein [2015] for an overview), none of the approaches
the authors are aware of are models of abstract rea-
soning. It is clear that any agent performing Vingean
re
ection will need to have some way of dealing with
logical uncertainty, since it will have to reason about the
behavior of computer programs it cannot run (in par-
ticular, future versions of itself). At present, however,
formal models of logical uncertainty do not yet seem up
to the task of studying abstract reasoning about more
intelligent successors.
In this paper, we review a body of work which in-
stead considers agents that use formal proofs to rea-
son about their successors, an approach rst proposed
by Yudkowsky and Herresho (2013). In particular,
following these authors, we consider agents which will
only perform actions (such as self-modications) if they
can prove that these actions are, in some formal sense,
\safe". We do not argue that this is a realistic way for
smarter-than-human agents to reason about potential
actions; rather, formal proofs seem to be the best formal
model of abstract reasoning available at present, and
hence currently the most promising vehicle for study-
ing Vingean re
ection.
There is, of course, no guarantee that results ob-
tained in this setting will generalize to whatever forms
of reasoning realistic articial agents will employ. How-
ever, there is some reason for optimism: at least one
such result (the procrastination paradox [Yudkowsky
2013], discussed in Section 4) both has an intuitive in-
terpretation that makes it seem likely to be relevant be-
yond the domain of formal proofs, and has been shown
to apply to one existing model of self-referential reason-
ing under logical uncertainty (Fallenstein 2014b).
The study of Vingean re
ection in a formal logic
framework also has merit in its own right. While for-
mal logic is not a good tool for reasoning about a
complex environment, it is a useful tool for reasoning
about the properties of computer programs. Indeed,
when humans require extremely high condence in a
computer program, they often resort to systems based
on formal logic, such as model checkers and theorem
provers (US DoD 1985; UK MoD 1991). Smarter-than-
human machines attempting to gain high condence
in a computer program may need to use similar tech-
niques. While smarter-than-human agents must ulti-
mately reason under logical uncertainty, there is some
reason to expect that high-condence logically uncer-
tain reasoning about computer programs will require
3
something akin to formal logic.
The remainder of this paper is structured as fol-
lows. In the next section, we discuss in more detail
the idea of requiring an agent to produce formal proofs
that its actions are safe, and discuss a problem that
arises in this context, the L obian obstacle (Yudkowsky
and Herresho 2013): Due to G odel's second incom-
pleteness theorem, an agent using formal proofs cannot
trust the reasoning of future versions using the same
proof system. In Section 4, we discuss the procrasti-
nation paradox , an intuitive example of what can go
wrong in a system that trusts its own reasoning too
much. In Section 5, we introduce a concrete toy model
of self-rewriting agents, and discuss the L obian obstacle
in this context. Section 6 reviews partial solutions to
this problem, and Section 7 concludes.
3 Proof-Based Systems
When humans want extremely high condence in the
properties of a computer program, they often resort
to machine-checked proofs. Schmidhuber (2007) sug-
gests a proof-based architecture for self-modifying ar-
ticial agents, called G odel machines . These machines
run both an initial problem solver, which determines
the machine's interactions with the external world and
has been veried by conventional methods, and a proof
searcher, which attempts to nd rewrites (of the initial
problem solver, the proof searcher itself, or the runtime
hosting both of these) that it can formally prove to be
an improvement on the current version. (For details,
see Schmidhuber [2007]. However, the issues of self-
reference discussed in this paper appear to apply to the
G odel machine architecture as envisioned by Schmid-
huber; see Fallenstein and Soares [2014].)
Yudkowsky and Herresho (2013) have proposed
a related architecture in which a generally intelligent
suggester handles both rewrites and interactions with
the external world, but must prove that any action
it chooses to perform fullls a certain safety property.
The action is only performed after this proof has been
checked by a simple verier . This leaves open what hap-
pens if the suggester is not able to nd such a proof.
In analogy with the G odel machine's initial problem
solver, in this case control can be diverted to a fall-
back policy which has been veried to satisfy the safety
property, but which may otherwise be quite suboptimal
(Fallenstein and Soares 2014).
This set-up is reminiscent of the simplex architecture
of Seto et al. (1998), a software technology designed
to allow the upgrade of a real-time system while it is
running. A system employing the simplex architecture
simultaneously runs two dierent controllers and a con-
trol switching logic. The rst, experimental controller
is allowed to run the system initially, but if the system
goes outside a safety envelope, the control switching
logic hands control over to the second, baseline con-
troller. Like the experimental controller in the simplexarchitecture, the suggester is run in hopes that it will
get better results than the fallback policy (which is anal-
ogous to the baseline controller). An acceptable base-
line level of performance is guaranteed, because like the
control switching logic, the verier will divert control to
the fallback policy if necessary.
However, unlike in the simplex architecture, we do
not need to rely on detecting departures from a safety
envelope, which may be dicult in the case of a gen-
erally intelligent system; instead, the suggester is re-
sponsible for convincing the verier that its proposed
plan of action satises the safety property. Moreover,
the suggester is able to require future versions to prove
an even stronger safety property, as long as it can also
exhibit a new fallback policy that it can prove to sat-
isfy this property; thus, the agent can ratchet up the
eectiveness of its \baseline controller" over time.
Of course, constructing such an agent requires that
wehave formally dened safety properties that apply
to the actions of an articial agent. There are rea-
sons to believe that developing such a safety predicate
would be a very dicult task, but these issues are be-
yond the scope of the present paper; see Soares (2015)
for discussion. Here, we assume that such a safety
predicate is given, and focus on diculties that arise
when an agent with veriable safety properties, us-
ing a suggester-verier architecture, attempts to self-
improve.
In particular, consider a suggester which is propos-
ing a trivial self-modication, or even a suggester at-
tempting to execute an action that will leave the agent
unchanged. In order to prove that such an action is
safe, it must show not only that the direct eects of
this action satisfy the safety property, but also that all
actions the modied agent will take in the future will
satisfy this property as well.
Outside the system, wehave concluded that the
agent is safe by the following argument: either the agent
takes an action that its suggester has proved to be safe,
or it executes an action chosen by its fallback policy,
whose safety has been separately veried; in either case,
the action is safe. Intuitively, we expect that the agent
itself should be able to follow a similar line of reasoning
when reasoning about the safety of future versions of
itself.
As Yudkowsky and Herresho (2013) point out,
however, while the suggester can show that the new
version will either execute a (safe) fallback policy or
an action that it has proven safe, it cannot in general
conclude that an action which has been proven safe
isin fact safe. Intuitively, this follows from G odel's
second incompleteness theorem: No consistent formal
system as strong as Peano Arithmetic is able to prove
its own consistency. Thus, any rst-order theory T
our agent might use must have non-standard models
that believe Tto be inconsistent (i.e., models in which
the sentence Con( T), which asserts T's consistency, is
false). But in such a model, the fact that an action
is provably safe clearly does not in general imply that
4
this action is in fact safe, since if Tis inconsistent, all
actions are provably safe. More formally, according to
L ob's theorem (L ob 1955) (which generalizes the second
incompleteness theorem), Tcan only prove a sentence
of the form \if Tproves', then'" for formulas 'which
it already proves unconditionally; thus, Tcannot prove
that any provably safe action is in fact safe unless it can
prove all actions safe.
This may seem like a rather abstract problem, trace-
able to a quirk of formal logic. However, there is some
reason to think that it may apply far more generally.
We illustrate this in the next section by the procrastina-
tion paradox (Yudkowsky 2013; Fallenstein and Soares
2014), a concrete way in which an agent which trusts
future versions too much may go wrong. When our
agents use formal logic, this is very similar to G odel's
proof, but the informal argument seems likely to apply
to other approaches as well.
4 The Procrastination Paradox
Consider a suggester-verier agent with a very simple
goal: ensuring that a certain button gets pressed, ei-
ther now or at some point in the future. Suppose fur-
ther that time is divided into innitely many discrete
timesteps, and the agent must choose between only two
options; it can either press the button in the current
timestep, or leave everything unchanged, in which case
it will be in exactly the same state and faced with
exactly the same choice at the beginning of the next
timestep.
Intuitively, it is clear that in order to achieve its
goal, the agent must press the button immediately|
otherwise, since the next timestep is exactly the same,
it will again decide to \procrastinate," and so on in
all future timesteps. We now show how an agent that
trusts future versions too much can argue itself into
believing the opposite.
WriteTp'qfor the proposition that the sentence '
is provable in the theory T. For any recursively enumer-
able theory T, dene the uniform re
ection principle
REF(T) to be the collection of all sentences of the form
8n:Tp'(n)q!'(n); (1)
where'() ranges over formulas with one free variable in
the language of arithmetic, and nindicates the numeral
corresponding to n; e.g., ifn= 2, thenp'(n)qis the
G odel number of the sentence '(S(S(0))).
Assume that our agent is using a particular the-
oryTextending Peano Arithmetic ( PA) and write G
for the proposition that the button is pressed some-
time after the current timestep. Our suggester attempts
to prove that Gis true, in which case it \procrasti-
nates"; otherwise, our agent executes its fallback pol-
icy, which is to press the button immediately. If T
can prove this by inspection of the agent's source code,
thenT` :G!TpGq; although we do not need
to do so, if we assume that the suggester will nd aproof if there is one,1then the implication turns into
an equivalence, and Gturns into the G odel sentence \I
am not provable" ( T`G$:TpGq). As in G odel's
rst incompleteness theorem, if Tis sound,Gmust be
true but unprovable: The agent must press the button
immediately.
However, if Tproves its own re
ection principle (i.e.,
ifT`REF(T)), our agent can also reason as follows:
\Either I will press the button in the next timestep or I
won't" (T`G\_:G). \The only case in which I would
notpress the button is if I have found a proof that
the button gets pressed at a later time" ( T`:G!
TpGq). \If it's provable that I will press the button,
then by (1), I will indeed do so" ( T`TpGq!G).
\Thus, whether or not I press the button in the next
timestep, it eventually gets pressed" ( T`G); \hence,
I do not need to press it now!"
SinceT`GimpliesT`TpGq(if a recursively
enumerable theory Tproves a sentence ', then any
theory as strong as PAprovesTp'q), we also have
T`:G, implying that Tis inconsistent; although it
makes stronger assumptions, this is very similar in
a-
vor to the proof of the second incompleteness theorem.
However, the informal version of the agent's reason-
ing uses no special properties of rst-order logic, and
suggests that similar diagonalization issues may aect
many other approaches to Vingean re
ection. In fact,
Fallenstein (2014b) shows that this is indeed a problem
for a formalism for self-reference in logical uncertainty
proposed by Christiano et al. (2013), which uses prob-
abilities rather than formal proofs.
5 Agents in Botworld
The fact that requiring T`REF(T) leads to inconsis-
tency implies that the most straight-forward way of for-
malizing the intuition why a suggester should be able
to justify simple self-modications does not work; to
model Vingean re
ection with the suggester-verier ar-
chitecture, the proof of safety must use an alternate
path.
In order to do so, it will be helpful to have a more
formal model of the problem. In this section, we in-
troduce two tools that have been developed for this
purpose: A formalism that agents using the suggester-
verier architecture can use to reason about the decision
problem they are facing (Fallenstein and Soares 2014),
based on the \space-time embedded intelligence" model
of Orseau and Ring (2012); and Botworld (Soares and
Fallenstein 2014b), a concrete toy environment exhibit-
ing some features of the real world that satisfactory
Vingean re
ection should be able to handle.
Botworld is structured as a cellular automaton.
Each cell may contain multiple items and robots . The
items come in many varieties, including cargo , which
the robots are trying to collect for its intrinsic value,
and robot parts , which may be used to create robots.
1. This requires a halting oracle.
5
Each robot houses a simple register machine, which
is run during each timestep in order to determine the
robot's action in that timestep. Robots may manipu-
late (lift, drop, and carry) items. They may also cre-
ate new robots (from component parts), destroy robots
(into component parts), and inspect robots (reading the
states of their register machines).
Consider \games" where the initial state of the au-
tomaton is xed except for the programming of one
particular robot. The task is to choose the program of
this robot in such a way that a particular objective will
be met (perhaps after the initial program has under-
gone many self-improvements). This provides a fairly

exible environment in which to study agents perform-
ing Vingean re
ection. This paper lacks the space for
a full description of Botworld, but further details and
a complete, executable specication are given in Soares
and Fallenstein (2014b).
To illustrate the L obian obstacle, consider a sim-
plebodyguard game , in which there are (at least) two
robots: The bodyguard robot is the agent attempting
to self-improve. The assignment robot is a robot that
must not be destroyed. As long as the assignment sur-
vives, the player gets points for any cargo the body-
guard robot collects. In simple versions of this game,
the player need only ensure that the bodyguard never
unwittingly destroys the assignment. In more complex
variants, other robots which attempt to destroy the as-
signment robot may be included in the environment.
We now introduce a simplied version of the for-
malism proposed by Fallenstein and Soares (2014). In-
stead of considering a sequence of interactions with the
environment (such as considered by Orseau and Ring
[2012]), this formalism focuses on a single choice: An
agent makes a single observation o2O and chooses a
single action a2A. In Botworld, the observation is the
input that the agent receives from the environment, and
the action includes both the robots's behavior and the
new state of its register machine. (Thus, decisions to
self-modify and decisions notto self-modify are treated
in exactly the same way.)
Dene a policy to be a function 2 which spec-
ies, for every observation o2O, the action (o)2A
that the agent will take if it makes this observation.
Assume that there is a set Qof \outcomes" describing
all features of the world, at all future times, that mat-
ter to the agent's goals. In the bodyguard game, set
Q:= (f0;1gN)N, where for q2Q, the rst element
ofq(n) is 1 if the assignment robot is still \alive" in
then'th timestep, and the second element is the total
value of all cargo held by the bodyguard robot at this
time. Each agent is equipped with an \environment
function"Q:OA!Q which maps observations and
actions onto outcomes such that Q(o;a) denotes the
future state achieved after making observation oand
executing action a. This function must be fully speci-
ed, but need not be feasibly computable : the agent will
prove things about the outcome its actions lead to, but
will not look into the future to determine the preciseoutcome achieved. (The full formalism of Fallenstein
and Soares (2014) allows the agent to have uncertainty,
in the form of a probability distribution, over the results
of its actions.)
Assume further that there is a safety predicate
Safe(q), dened on outcomes q2 Q, which species
the safety property the agent's actions must satisfy. In
the bodyguard game, this predicate is
Safe(q) :$8n:9x:q(n) = (1;x);
which simply states that the assignment robot is alive
in all future timesteps.
Consider agents running a suggester-verier archi-
tecture with a fallback policy fb2 which species a
fallback action fb(o) to be taken after making observa-
tiono. This fallback action will be executed if the agent
cannot nd another provably safe action to take, and
thus the player must verify Safe( Q(o;fb(o))) for allo
themselves (in order to verify that the system is safe).
In the simple versions of the bodyguard game (where
there are no robots besides the bodyguard and the as-
signment, and the assignment never attempts to decon-
struct itself), this can simply be the policy which exe-
cutes the Pass command in all future timesteps, thus
never taking any external action at all.
An agent running a suggester-verier architecture
also contains an untrusted suggester , another com-
putable function. This suggester may have some moti-
vation, such as collecting high-value cargo, but cannot
be trusted to suggest only safe actions. For example,
may notice that the assignment robot is built from very
expensive parts, and may suggest destroying the assign-
ment robot and collecting its components|an action
the verier should not allow. Formally, the suggester
takes an observation o2O and attempts to return an
actiona2Atogether with a formal proof of the target
sentencepSafe(Q(o;a))qin some formal theory Tsuch
asT=PA, whereoandaare the numerals denoting o
anda. If no such action/proof pair can be found, the
suggester returns an error value.
The verier runs (o) and, if this returns a pair ( a;p)
within the allotted time limit, checks whether pis
a valid proof of the target sentence from the axioms
ofT. If so, it executes action a; otherwise, ac-
tionfb(o). Denote this policy for selecting actions by
svT(pSafeq;pQq;; fb)2. (Here, we write pSafeq
andpQqin G odel quotes because the suggester-verier
only makes use of their denitions in rst-order logic,
whileandfbneed to be provided as runnable code.)
The question is, can such an agent justify performing a
self-modication?
Consider an agent as described above which has
made some specic observation ^ o2 O and which is
considering some specic action ^ a2A, which leaves
the agent in a state such that, in the next step, it will
execute the policy ^ 0= sv T0(pSafe0q;pQ0q;0;0
fb)2
for some theory T0, safety predicate Safe0, outcome cal-
culatorQ0, suggester 0, and fallback policy 0
fb. Obvi-
ously, not all such policies are safe (consider the policy
6
where Safe0(q) :$(0 = 0)). But if the policy issafe|
say, ifT=T0, Safe = Safe0and0
fbis provably safe on
all inputs|then will the current suggester be able to
exhibit a proof of this fact?
In order to prove the safety of ^ 0, the agent must
rst prove that the new agent's safety predicate is at
least as strong as the existing safety predicate. Thus,
it must be the case that:
T` 8q2Q:Safe0(q)!Safe(q): (2)
Note that it is ne for the predicate Safe0to be
more restrictive than Safe: for example, it could be
that Safe0(q) :$8n:(9x:q(n) = (1;x)^x7), which
ensures that the assignment robot survives and that the
agent scores at least 7 points.
Of course, this property only matters if the suc-
cessor agent accurately assesses world outcomes. The
safety predicate Safe0is of no consequence if Q0(o;a) =
(1;1000) unconditionally. Thus, the agent must also
prove
T` 9o02O:Q0(o0;^0(o0)) =Q(^o;^a) (3)
which intuitively states that Q0agrees with Qabout
world outcomes.
Next, the agent must be able to show that the fall-
back policy 0
fbof ^0is safe:
T` 8o02O:Safe0(Q0(o0;0
fb(o0))) (4)
By the implementation of the suggester-verier ar-
chitecture, this allows our agent to reason that execut-
ing action ^a(and thus adopting policy ^ 0) will either
result in the execution of 0
fb(o0) (which, by (4), is safe),
orof an action a0such that it is provable in the the-
oryT0thatpSafe0(Q0(o0;a0))q. Thus, ifT`REF(T0),
then our agent can prove that executing the aciton ^ a
is safe, as desired. (Fallenstein and Soares [2014] dis-
cuss this proof in detail.) But the last condition requires
thatT06=T: otherwise, Twill be inconsistent and our
agent would be able to prove any action \safe," for the
reasons described in Section 4.
In the next section, three partial solutions to the
L obian obstacle are discussed.
6 Partial Solutions
6.1 Descending Trust
One obvious way to address this problem of self-
reference is to require that the child agent use a weaker
system for verifying proofs than the parent agent (Yud-
kowsky and Herresho 2013). If this allows the parent
agent to verify the consistency and soundness of the
child agent's proof system, then the parent agent can
license the creation of a safe child. We call this the
descending trust method of Vingean re
ection.
Consider a parent agent using the theory T:=PA+
REF( PA). If the child agent ^ 0under considerationuses the theory T0=PAto verify its proofs, then T`
REF(T0). This, along with (2), (3) and (4), allows the
parent agent to prove its target theorem and license the
action ^awhich constructs the agent ^ 0. But if the child
agent wants to license the creation of another agent, it
will be in the same situation as if Thad been PAto
start with.
This issue can be alleviated by using a tower of theo-
ries of increasing strength. For example, an agent using
the theory PA+ REF( PA+ REF(PA)) may license the
construction of an agent using PA+ REF( PA), which
can license the construction of an agent using PA. Us-
ing the notation T+ 1 to denote T+ REF(T), it is easy
to see that an agent verifying proofs using PA+nmay
license descendant chains up to depth nbefore being
forced to construct successor agents that verify proofs
with a theory weaker than PA.
Thus, for tasks that require only a nite number of
timesteps, it may suce to build agents with verier
components using the theory PA+nfor some absurdly
largen. Instead of using nite n, any computable or-
dinalmay also be used. (Such \transnite progres-
sions" have been studied by Turing [1939] and Feferman
[1962].) Unfortunately, by the well-foundedness of the
ordinals, this method can still only license the creation
of nite chains of successors.
Even if this method can be employed for nite tasks,
the result is somewhat unsatisfacotry. These agents,
which are ostensibly self-modifying in attempts to self-
improve , can only verify the construction of agents
which use weaker proof systems in their verier compo-
nent.
6.2 Herresho's Waterfall
One might hope that there exists some innite sequence
of theories Tnsuch that each Tnproves REF( Tn+1).
Unfortunately, any such sequence of recursively axiom-
atizable theories must be unsound, by a variant of the
procrastination paradox discussed in Section 4 (Yud-
kowsky 2013). However, it turns out to be possible to
nd a sequence Tnsuch thatTn`Con(Tn+1) for ev-
eryn.
Even this requires care. Na vely, one might attempt
to dene the sequence by Tbad
n:=PA+Con(Tbad
n+1) (that
is, where each theory is constructed from PAand ax-
ioms which state that the next theory in the sequence is
consistent|the recursive denition can be made formal
by using the diagonal lemma to dene a primitive recur-
sive predicate '(n;m), which is true if mis the G odel
number of an axiom of Tn). However, it turns out that
every theory Tbad
ndened in this way is inconsistent
(Stiennon 2013).
Instead, consider the set of theories
Tn:=PA+ ( (n)!Con(Tn+1)); (5)
where (n) is true if nisnotthe G odel number of
a proof of an inconsistency from the axioms of ZFC.
7
Then, although each Tnproves (n) (assuming that
ZFC is consistent) and therefore Con( Tn+1), this fact is
not provable in PA, and it turns out that because of this,
allTnare consistent (Fallenstein 2014a) (and therefore
sound, since the only axiom of Tnthat is not an axiom of
PAmerely asserts the consistency of Tn+1). This idea,
due to Marcello Herresho, was originally described by
Yudkowsky and Herresho (2013), and the version pre-
sented here was proposed by Fallenstein (2014a).
It is well-known that if a theory Textends PAand
ifT`Con(T0), thenTproves the re
ection principle
restricted to  1formulas (recall that a formula '(n) is
1if it is provably equivalent to 8m:~'(m;n), where
~'(m;n) is some predicate that can be decided by a
primitive recursive function); i.e.,
T`8n:T0p'(n)q!'(n)
for all'21. Thus, so long as the safety predicate
Safe that an agent must prove in order to construct
its successor agent is a  1sentence, a suggester-verier
agent using Tnmay license the creation of a successor
agent using Tn+1.
This result is heartening, if not entirely satisfactory.
Agents reasoning using the theory Tncan trust the next
theoryTn+1, and can indeed trust the theory Tn+mfor
any nite m (so long as they can prove (n+m)),
but such agents cannot conclude that8m: (n+m).
Thus, such agents must entertain the possibility that
they will one day run out of ability to verify new agents.
We, outside the system, know that they never will, but
they, inside the system, cannot show this fact. This
seems like a strange epistemic state for a rational agent
to occupy.
6.3 Model Polymorphism
It seems odd to demand that the theory used by the
agent's verier component must change with every
timestep. Is there some assumption that could be re-
laxed in order to allow the agent to safely use the same
theory in each timestep?
It turns out there is. One approach for doing so
ismodel polymorphism (see Yudkowsky and Herresho
2013, where it is referred to as parametric polymor-
phism ). This approach can be implemented by a theory
MPin the language of PAaugmented by a new constant
symbol, which consists of the axioms of PA(with
the axiom schema of induction ranging over formulas
in the extended language) extended by all sentences of
the form
>0!8n:
MPp'(n)q!'[:=1](n)
;(6)
where'() ranges over one-variable formulas in the ex-
tended language, and '[:=1] denotes the formula
obtained by replacing all occurrences of the constant
in'by the term 1.
Model polymorphism requires a modication to the
suggester-verier architecture in order to work. Insteadof requiring a proof that an action is safe for all future
times, require instead that an action is safe \for the
nexttimesteps". Outside the system, we can show
that (N;=n)MPfor anyn, i.e., that the theory
MPis sound ifis interpreted as any natural number.
Thus, we can be condent that the system is safe for
an arbitrary number of steps. The system itself cannot
reason in this way, and thereby avoids the standard
G odelian diculties. The proof of soundness begins
by noting that ( N;= 0) trivially models MP(as its
special axiom schema is vacuously true in this case),
and then proceeds to show by induction that ( N;=
n)MPimplies ( N;=n+ 1)MP; for details, see
Yudkowsky and Herresho (2013).
MPrequires replacing the unary predicate Safe( q)
(which veries a safety property in all future timesteps)
by a binary predicate Safe( n;q) (which veries a safety
property in the nthtimestep). In the bodyguard game,
this is the predicate Safe( n;q) :$9x: q(n) = (1;x),
which states that the assignment robot is still alive in
then'th timestep.
An agent using MPwhich can perform Vingean re-

ection can be constructed as follows: rst, the agent
must know the current timestep t. Its target theorem
for the action ais
p8nt+:Safe(n;Q(o;a))q:
Then, in any universe with discrete timesteps, a par-
ent agent verifying proofs with MPmay license the cre-
ation of a successor agent also running MP. The trick
is that the parent agent need only prove that the child
agent executes safe actions for 1 steps, as the child
agent won't begin running until the following timestep.
The suggester must prove slightly modied versions
of (2) and (4), with the new Safe predicates:
MP`8n:8q2Q:Safe0(n;q)!Safe(n;q) (7)
MP`8nt+:8o02O:Safe0(n;Q0(o0;0
fb(o0))) (8)
Then the agent will be able to reason (by inspec-
tion of the suggester-verier architecture) that either
the successor will execute its fallback policy (which
is safe for at least steps by (8)) or will execute
some other action a0after proving in theory T0that
p8n<t+ 1 +:Safe0(n;Q0(o0;a0))q.
Now, ifT0=MPthen the suggester can use the MP
axiom schema to deduce
8n<t+ 1 +1:Safe0(n;Q0(o0;a0)); (9)
which it can rewrite to
8n<t+:Safe0(n;Q0(o0;a0)): (10)
This, along with (7) and (3) gives us the target the-
orem, as desired.
8
An agent reasoning using model polymorphism can
license the construction of another agent using the same
theory. Unlike Herresho's waterfall, such agents can
reason that they will be able to continue licensing the
creation of new safe agents indenitely.
However, it is dicult to interpret proofs about
intuitively as arguments made by a rational agent; it
is hard to make intuitive sense of the epistemic state
of an agent that has proven a certain action to be safe
for \" timesteps. In addition, model polymorphism
only works in systems where time can be divided into
discrete steps. While model polymorphism seems like
a signicant step forward, it is by no means a fully
satisfactory solution.
7 Discussion
In this paper, we have discussed a pair of obstacles to
Vingean re
ection that arise in the context of proof-
based suggester-verier systems: the L obian obstacle
and the procrastination paradox . In order to create an
agent that can self-modify with sucient foresight to
navigate an intelligence explosion, it seems that we will
have to chart a course between the L obian obstacle's
rock of having so little self-trust that the agent cannot
execute simple self-modications, and the procrastina-
tion paradox's hard place of having so much self-trust
that the agent's reasoning becomes unsound.
In the form we have discussed them, the L obian ob-
stacle and the procrastination paradox stem from the
demand for proofs of safety; this is of course unreal-
istic in practice: an agent operating with uncertainty
cannot realistically be expected to formally prove that
specic action leads to an outcome with desirable prop-
erties. (However, Fallenstein [2014b] shows that a ver-
sion of the procrastination paradox applies to the self-
referential formalism for logical uncertainty proposed
by Christiano et al. [2013].)
And yet, it intuitively seems that a system should
be able to observe that another system using the same
reasoning process has concluded 'and, from this, con-
clude': if reasoning of the form \that system reasons
as I do and deduced ', therefore'" cannot be formal-
ized by arguments of the form \a system using the same
theory as me proved it, and therefore it is true," then
how canthis intuitively desirable reasoning be formal-
ized?
One path is to relax the condition for action: in-
stead of considering agents that execute action aonly
after proving that ais safe, allow agents to execute aif
they can prove that ais safe orif they can prove that
systems of similar strength prove ais safe orif they
can prove that systems of similar strength prove that
systems of similar strength prove ais safe, and so on.
This is similar to the approach taken by Weaver (2013).
However, this still does not allow an agent to execute a
if it knows that another agent in the same situation,
which is using the same condition for action, has takenactiona: the fact that the agent knows 9n:np'qdoes
not allow it to conclude np'qfor any concrete n.
Another more generic solution would be to develop
some sort of non-monotonic self-trust, allowing an agent
to trust its own reasoning only in non-paradoxical cases,
in the same way that the reader can trust their own rea-
soning without trusting their belief about the sentence
\the reader does not believe this very sentence." How-
ever, it is not yet clear how to construct a reasoning
system which has non-monotonic self-trust in this way.
The authors hope that continued study of Vingean
re
ection in proof-based models will shed light on how
to move forward in one of these directions.
One topic for future work is to push the proof-based
models towards more realistic settings. Fallenstein and
Soares (2014) present a version of the suggester-verier
formalism in which agents maintain a probability dis-
tribution over external states; Yudkowsky (2014) and
Soares (2014) discuss versions of this in the setting of
dynamic Bayesian networks.
A second topic is to address the fact that the
suggester-verier agent formalism described in this pa-
per only considers satiscing agents, in the sense of
Simon (1956). The suggester in a suggester-verier ar-
chitecture is not required to prove that its chosen ac-
tion is optimal ; rather, it merely needs to nd an action
that is good enough (in the sense that it satises some
goal predicate). There is a considerable gap between
this kind of agent and a real-world agent that acts as
intended: Suppose, for example, that an agent is con-
structed which satisces its goals if it saves 3 billion
lives in expectation. The agent discovers an asteroid
en route to Earth which will end 7 billion lives if not
stopped. This agent might nd a plan that denitely
stops the asteroid and then decide to
ip a coin and de-
ploy the plan if the coin lands heads: this gamble saves
3.5 billion lives in expectation, and thereby satises the
agent's goal.
Intuitively, there should be some way to \bound-
edly maximize" expected utility, picking the best ac-
tion available given the cost of spending more resources
searching for better actions. \Staged maximization"
(Yudkowsky 2014) is an attempt at beginning to for-
malize this intuition.
Astaged maximizer divides the entire action space
into categories and considers those. This idea is ex-
plored by Yudkowsky (2014), who discusses a number
of new hurdles related to staged maximization.
To illustrate, consider a parent agent which must
select an action from the set fcontinue;stopg. If the
parent selects continue then the child will select an
action from the set fa1;a2:::;a 9g. If the parent selects
stop then the child must select the action fstoppedg.
Each of the ten child actions has an associated true
utility, and the parent makes an unbiased noisy estimate
of each true utility. Say the true utility of each action
a1:::a 9is 0, the true utility of stopped is 0.1, and the
noise is drawn from a normal distribution with mean
0 and standard deviation 1. In this case, the parent is
9
biased towards continue , because that action leads to
more possibilities, which leads to more opportunities for
the unbiased noise estimates to be higher: the parent,
if implemented na vely, steers towards noise. This is
known as the \optimizer's curse" (Smith and Winkler
2006), and it indicates that further research is necessary
to better understand satisfactory staged maximization.
Regardless of what direction is taken, further re-
search into Vingean re
ection seems important. It may
not be possible to delegate the study of Vingean re
ec-
tion to self-modifying systems unless those systems are
already at the level of a team of human researchers in all
relevant regards, and it is plausible that signicant self-
modication (and, therefore, Vingean re
ection) may
be required to get to that point: Just as computer sys-
tems can be procient at chess without being generally
intelligent, it is plausible that a computer system could
become procient at some types of reasoning before be-
coming procient at the kind of mathematical philoso-
phy necessary to develop high-condence methods for
Vingean re
ection.
Satisfactory models of Vingean re
ection will have
to look very dierent from the models presented in this
paper: they will have to allow for logical uncertainty
and could plausibly require non-monotonic self-trust.
Nevertheless, study has to start somewhere, and we ex-
pect that starting with these highly simplied models
and pushing them toward practicality could reveal more
plausible paths toward practical methods for reliable
Vingean re
ection.
References
Bostrom, Nick. 2014. Superintelligence: Paths, Dangers,
Strategies. New York: Oxford University Press.
Christiano, Paul F., Eliezer Yudkowsky, Marcello Her-
resho, and Mihaly Barasz. 2013. Denability of
Truth in Probabilistic Logic. Working Paper. Ma-
chine Intelligence Research Institute, Berkeley, CA,
April 2. https : / / intelligence . org / files /
DefinabilityTruthDraft.pdf .
Fallenstein, Benja. 2014a. An Innitely Descending Se-
quence of Sound Theories each Proving the Next Con-
sistent. Brief Technical Note. Machine Intelligence Re-
search Institute, Berkeley, CA. http://intelligence.
org/files/ConsistencyWaterfall.pdf .
. 2014b. Procrastination in Probabilistic Logic. Work-
ing Paper. Machine Intelligence Research Institute,
Berkeley, CA. http : / / intelligence . org / files /
ProbabilisticLogicProcrastinates.pdf .
Fallenstein, Benja, and Nate Soares. 2014. \Problems of Self-
Reference in Self-Improving Space-Time Embedded In-
telligence." In Articial General Intelligence: 7th In-
ternational Conference, AGI 2014, Quebec City, QC,
Canada, August 1{4, 2014. Proceedings, edited by Ben
Goertzel, Laurent Orseau, and Javier Snaider, 21{32.
Lecture Notes in Articial Intelligence 8598. New York:
Springer. doi: 10.1007/978-3-319-09274-4\_3 .Feferman, Solomon. 1962. \Transnite Recursive Progres-
sions of Axiomatic Theories." Journal of Symbolic
Logic 27 (3): 259{316.
Gaifman, Haim. 2004. \Reasoning with Limited Resources
and Assigning Probabilities to Arithmetical State-
ments." Synthese 140 (1{2): 97{119. doi: 10.1023/B:
SYNT.0000029944.99888.a7 .
Good, Irving John. 1965. \Speculations Concerning the
First Ultraintelligent Machine." In Advances in Com-
puters, edited by Franz L. Alt and Morris Rubino,
6:31{88. New York: Academic Press. doi: 10 . 1016 /
S0065-2458(08)60418-0 .
Hutter, Marcus. 2000. \A Theory of Universal Articial In-
telligence based on Algorithmic Complexity." Unpub-
lished manuscript, April 3. http://arxiv.org/abs/
cs/0004001 .
L ob, M. H. 1955. \Solution of a Problem of Leon Henkin."
Journal of Symbolic Logic 20 (2): 115{118. http://
www.jstor.org/stable/2266895 .
Orseau, Laurent, and Mark Ring. 2012. \Space-Time Em-
bedded Intelligence." In Articial General Intelligence:
5th International Conference, AGI 2012, Oxford, UK,
December 8{11, 2012. Proceedings, edited by Joscha
Bach, Ben Goertzel, and Matthew Ikl e, 209{218. Lec-
ture Notes in Articial Intelligence 7716. New York:
Springer. doi: 10.1007/978-3-642-35506-6\_22 .
Schmidhuber, J urgen. 2007. \G odel Machines: Fully Self-
Referential Optimal Universal Self-Improvers." In Ar-
ticial General Intelligence, edited by Ben Goertzel
and Cassio Pennachin, 199{226. Cognitive Technolo-
gies. Berlin: Springer. doi: 10.1007/978-3-540-68677-
4\_7.
Seto, Danbing, Bruce Krogh, Lui Sha, and Alongkrit Chuti-
nan. 1998. \The Simplex Architecture for Safe On-
line Control System Upgrades." In Proceedings of
the American Control Conference, 1998, 6:3504{3508.
IEEE.
Simon, Herbert A. 1956. \Rational Choice and the Structure
of the Environment." Psychological Review 63 (2): 129.
Smith, James E., and Robert L. Winkler. 2006. \The Opti-
mizer's Curse: Skepticism and Postdecision Surprise in
Decision Analysis." Management Science 52 (3): 311{
322. doi: 10.1287/mnsc.1050.0451 .
Soares, Nate. 2014. Tiling Agents in Causal Graphs. Techni-
cal report 2014{5. Berkeley, CA: Machine Intelligence
Research Institute. http://intelligence.org/files/
TilingAgentsCausalGraphs.pdf .
. 2015. The Value Learning Problem. Technical re-
port 2015{4. Berkeley, CA: Machine Intelligence Re-
search Institute. https://intelligence.org/files/
ValueLearningProblem.pdf .
Soares, Nate, and Benja Fallenstein. 2014a. Aligning Su-
perintelligence with Human Interests: A Technical Re-
search Agenda. Technical report 2014{8. Berkeley, CA:
Machine Intelligence Research Institute. https : / /
intelligence.org/files/TechnicalAgenda.pdf .
10
Soares, Nate, and Benja Fallenstein. 2014b. Botworld. Tech-
nical report 2014{2. Berkeley, CA: Machine Intelligence
Research Institute. http://intelligence.org/files/
Botworld.pdf .
. 2015. Questions of Reasoning Under Logical
Uncertainty. Technical report 2015{1. Berke-
ley, CA: Machine Intelligence Research Insti-
tute. https : / / intelligence . org / files /
QuestionsLogicalUncertainty.pdf .
Stiennon, Nisan. 2013. Recursively-Dened Logical The-
ories are Well-Dened. Brief Technical Note.
Berkeley, CA: Machine Intelligence Research
Institute. http : / / intelligence . org / files /
RecursivelyDefinedTheories.pdf .
Turing, Alan Mathison. 1939. \Systems of logic based on
ordinals." Proceedings of the London Mathematical So-
ciety 2 (1): 161{228.
United Kingdom Ministry of Defense. 1991. Requirements
for the Procurement of Safety Critical Software in
Defence Equipment. Interim Defence Standard 00-55.
United Kingdom Ministry of Defense, April 5.
United States Department of Defense. 1985. Department of
Defense Trusted Computer System Evaluation Crite-
ria.Department of Defense Standard DOD 5200.28-
STD. United States Department of Defense, Decem-
ber 26. http : / / csrc . nist . gov / publications /
history/dod85.pdf .
Vinge, Vernor. 1993. \The Coming Technological Singular-
ity: How to Survive in the Post-Human Era." In Vision-
21: Interdisciplinary Science and Engineering in the
Era of Cyberspace, 11{22. NASA Conference Publica-
tion 10129. NASA Lewis Research Center. http : / /
ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/
19940022856.pdf .
Weaver, Nik. 2013. \Paradoxes of Rational Agency and For-
mal Systems That Verify Their Own Soundness." Un-
published manuscript, December 21. http://arxiv.
org/abs/1312.3626 .
Yudkowsky, Eliezer. 2008. \Articial Intelligence as a Pos-
itive and Negative Factor in Global Risk." In Global
Catastrophic Risks, edited by Nick Bostrom and Mi-
lan M. Cirkovi c, 308{345. New York: Oxford University
Press.
. 2013. The Procrastination Paradox. Brief Tech-
nical Note. Machine Intelligence Research Institute,
Berkeley, CA. http : / / intelligence . org / files /
ProcrastinationParadox.pdf .
. 2014. Distributions Allowing Tiling of Staged
Subjective EU Maximizers. Machine Intelligence Re-
search Institute, Berkeley, CA, May 11. Revised
May 31, 2014. http : / / intelligence . org / files /
DistributionsAllowingTiling.pdf .
Yudkowsky, Eliezer, and Marcello Herresho. 2013. Tiling
Agents for Self-Modifying AI, and the L obian Obstacle.
Early Draft. Machine Intelligence Research Institute,
Berkeley, CA. http : / / intelligence . org / files /
TilingAgents.pdf .
11