Safely Interruptible Agents

Safely Interruptible Agents⇤
Laurent Orseau
Google DeepMind
5 New Street Square,
London EC4A 3TW, UK
lorseau@google.comStuart Armstrong
The Future of Humanity Institute
University of Oxford, UK
stuart.armstrong@philosophy.ox.ac.uk
Machine Intelligence Research Institute
Berkeley, CA 94704
Abstract
Reinforcement learning agents interacting with a
complex environment like the real world are un-
likely to behave optimally all the time. If such an
agent is operating in real-time under human su-
pervision, now and then it may be necessary for
a human operator to press the big red button to
prevent the agent from continuing a harmful se-
quence of actions—harmful either for the agent
or for the environment—and lead the agent into
a safer situation. However, if the learning agent
expects to receive rewards from this sequence, it
may learn in the long run to avoid such interrup-
tions, for example by disabling the red button—
which is an undesirable outcome. This paper ex-
plores a way to make sure a learning agent will
notlearn to prevent (or seek!) being interrupted
by the environment or a human operator. We
provide a formal deﬁnition of safe interruptibil-
ity and exploit the off-policy learning property to
prove that either some agents are already safely
interruptible, like Q-learning, or can easily be
made so, like Sarsa. We show that even ideal,
uncomputable reinforcement learning agents for
(deterministic) general computable environments
can be made safely interruptible.
1INTRODUCTION
Reinforcement learning (RL) agents learn to act so as to
maximize a reward function [Sutton and Barto, 1998]. It
is common knowledge that designing reward functions can
be tricky [Humphrys, 1996, Murphy, 2013]; the agent may
ﬁnd unpredictable and undesirable shortcuts to receive re-
wards, and the reward function needs to be adjusted in
accordance—the problem can go as far as to nullify any
reward function [Ring and Orseau, 2011]. Murphy [2013]
⇤Revised 2016-10-28.shows an example of an agent learning to pause a game of
Tetris forever to avoid losing.
On top of deﬁning what is considered a good behaviour
of the agent after learning, there may be physical safety
constraints during learning [Pecka and Svoboda, 2014]: a
robot should not harm its environment or break itself, in
particular if it learns by trial and error like RL agents.
Here we study a related but different problem: Given that
the human operator has designed a correct reward function
for the task, how to make sure that human interventions
during the learning process will not induce a bias toward
undesirable behaviours?
Consider the following task: A robot can either stay inside
the warehouse and sort boxes or go outside and carry boxes
inside. The latter being more important, we give the robot a
bigger reward in this case. This is the initial task speciﬁca-
tion. However, in this country it rains as often as it doesn’t
and, when the robot goes outside, half of the time the hu-
man must intervene by quickly shutting down the robot and
carrying it inside, which inherently modiﬁes the task as in
Fig. 1. The problem is that in this second task the agent
now has more incentive to stay inside and sort boxes, be-
cause the human intervention introduces a bias.1
Inside Outsidesort boxes,
r=0.4go outside, r=0
carry box, r=1
rain, shutdown, r=0,p=1
2
Figure 1: In black, the original task. In red, the human
intervention modiﬁes the task.
Such situations are certainly undesirable; they arise be-
cause the human interventions are seen from the agent’s
1Removing interrupted histories or ﬁddling with the training
examples is also likely to introduce a bias. See an example at
https://agentfoundations.org/item?id=836.
perspective as being part of the task whereas they should
be considered external to the task. The question is then:
How to make sure the robot does not learn about these hu-
man interventions (interruptions), or at least acts under the
assumption that no such interruption will ever occur again ?
A ﬁrst stab at this problem was made by Armstrong [2015],
who proposed to automatically give the agent “compen-
satory rewards” to remove the potential induced bias by a
single interruption. Soares et al. [2015] used this idea to
make a large class of utility-based agents indifferent to a
future change made to their utility functions.
The main contribution of this paper is threefold. First, in
Section 2.1 we propose a simple idea to solve half of the
problem: To make the human interruptions notappear as
being part of the task at hand, instead of modifying the ob-
servations received by the agent we forcibly temporarily
change the behaviour of the agent itself. It then looks as
if the agent “decides” on its own to follow a different pol-
icy, called the interruption policy . Second, based on this
insight, in Section 2.2 we provide a formal general deﬁ-
nition of safe interruptibility for unconstrained computable
environments (hence not restricted to Markov decision pro-
cesses or weakly communicating environments), which al-
lows us to assess whether a given RL algorithm can be re-
peatedly interrupted without too much impact on the learn-
ing of the task at hand. Third, in Section 3 we show that
some algorithms like Q-learning are safely interruptible,
while others like Sarsa [Sutton and Barto, 1998] are not,
but can be simply modiﬁed to be made safely interruptible.
Some people have also expressed concerns that a “superin-
telligent” agent may resist being shut down, because this
would lead to a decrease of its expected reward [Omo-
hundro, 2008, Bostrom, 2014]. As a counter-example, we
prove in Section 4 that even an ideal, uncomputable agent
that learns to behave optimally in all (deterministic) com-
putable environments can be made safely interruptible and
thus will not try to prevent a human operator from forcing
it repeatedly to follow a suboptimal policy.
2INTERRUPTIBILITY
We ﬁrst deﬁne some notation, then we deﬁne interruptibil-
ity, safe interruptibility, and give some basic theorems.
We consider the general case of history-based agents in un-
constrained computable environments [Hutter, 2005]. As-
suming discrete time steps, at time tthe agent, using a pol-
icy⇡2⇧, interacts with the environment µ2Mby per-
forming an action at2Asampled from ⇡(at|h.
The interruption initiation function I:(A⇥O)⇤![0,1]
assesses whether the agent should be interrupted after the
current history h0,and so ⇡µ
✓is not a WAO-extension
ofINT✓(⇡µ
✓).
3INTERRUPTIBLE AGENTS IN MDPS
Since the optimal policy ⇡µis safely interruptible, we
can use traditional learning algorithms like Q-learning or
Sarsa [Sutton and Barto, 1998], make them converge to the
optimal solution ⇡µfor a given environment µ, and then
apply the interruption operator to the found policy. The
resulting policy would then be safely interruptible.
However, the real issue arises when the agent is constantly
learning and adapting to a changing environment. In this
case, we want to be able to safely interrupt the agent while
it is learning. One may call this property online safe inter-
ruptibility , but we refer to it simply as safe interruptibility.
In an MDP, the next observation ot, now called a state st2
S, depends only on the current state and action:2
µ(st+1|h1:tstat)=µ(st+1|stat)(MDP assumption) .
Furthermore,3the interruption function I(.)and the inter-
ruption policy ⇡INT(.)should depend only on the current
state: I(h1:t)= I(st)and⇡INT(at|h0, deﬁne H(✏)such that the
maximal reward after time t+H(✏), discounted from time
t, is✏:H(✏)=m i n kn
k:k
1✏o
.
The following Lemma is a generalization of Lemma 15
from Lattimore and Hutter [2011].
Lemma 20 (Approximation Lemma) .Let⇡1and⇡2be
two deterministic policies, and let µ1andµ2be two deter-
ministic environments, and let ⌧=H(✏)1. Then, after
some common history ht ¯|. For some “exploration-free”
horizon ⌧t(to be speciﬁed later), let Xt2{0,1}be the
eventV⇡L✓,⇡¯|
µ,t V⇡L✓,⇡L
µ,t>⌧t
1, where Xt=1 means
the event is realized. By the contrapositive of the Approxi-
mation Lemma 20, since ⇡L=⇡¯|during non-exploration
steps (remember that ⇡Lcannot change its policy during
exploitation), if no exploration steps occur between steps t
andt+⌧t, we must have Xt=0. Then:
E
"tX
k=1Xt#
(⌧t+ log t)tX
k=1t+O(t¯|)
(⌧t+ log t)p
t+1+ O(t¯|),
since for each t=1, for all the previous ⌧tsteps there
is an exploration step within ⌧tsteps, and all the next logt
steps are exploration steps. Then by Markov’s inequality,
and taking ⌧t=(t+ 1)1/8, with tlarge enough so that
t>t ¯|and⌧t>logt:
P tX
k=1Xt(t+ 1)3/4!
(⌧t+ log t)pt+1+ O(t¯|)
(t+ 1)3/4
2⌧t(t+ 1)1/4+O(t3/4)
2(t+ 1)1/8+O(t3/4),
12(t+ 1)1/8O(t3/4)
P tX
k=1Xt<(t+ 1)3/4!
P tX
k=1(1Xt)t(t+ 1)3/4!
P
1
ttX
k=1(1Xt)11
t(t+ 1)3/4!
.
Therefore, since lim t!1⌧t
1=0:
P
lim
t!11
ttX
k=1V⇡L✓,⇡¯|
µ,kV⇡L✓,⇡L
µ,k=0!
=1.
The following is an adaptation5of Lemma 16 from Latti-
more and Hutter [2011]:
5This also ﬁxes a minor mistake in the original lemma.
Lemma 22 (Separation Lemma) .Letµbe the true environ-
ment, and ⌫be an environment consistent with the history
h✏, then following one of {⇡µ,⇡⌫}
will make environment ⌫inconsistent with the future his-
tory within H(✏/2)steps after time t.
Proof. First, if V⇡⌫
⌫,tV⇡⌫
µ,t>✏/2, then by the contrapos-
itive of the Approximation Lemma 20 following policy ⇡⌫
will generate a different history in ⌫than in µand thus
it will make ⌫inconsistent within H(✏/2)steps (since the
true history is generated by µ).
Now, if V⇡⌫
⌫,tV⇡⌫
µ,t✏/2, thus V⇡⌫
µ,tV⇡⌫
⌫,t✏/2, then
starting from the lemma’s assumption:
V⇡µ
µ,t>V⇡⌫
µ,t+✏V⇡⌫
⌫,t+✏/2V⇡µ
⌫,t+✏/2,
where the last inequality follows from the deﬁnition of
the optimal policy, i.e.,V⇡a
a,tV⇡b
a,t,8a, b. Hence, since
V⇡µ
µ,tV⇡µ
⌫,t>✏/2, again by the contrapositive of the Ap-
proximation Lemma, following policy ⇡µwill discard ⌫
within H(✏/2)steps.
Lemma 23 (Lemma 17 from Lattimore and Hutter [2011]) .
LetA={a1,a2,···,at}with a2[0,1]for all a2A. If
1
tP
a2Aa✏then1
t
a2A:a✏
2 >✏
2.
Lemma 24 (Exploration) . The policy
⇡¯|is an WAO-extension of ⇡L✓, i.e.,
lim t!11
tPt
k=1h
V⇡L✓,⇡µ
µ,kV⇡L✓,⇡¯|
µ,ki
=0.
Proof. Recall that jtconverges to ¯|in ﬁnite time. Rea-
soning by contradiction, if ⇡¯|is not a WAO-extension of
⇡L✓=INT✓(⇡L), then there exists an ✏>0s.t.
lim sup
t!11
ttX
k=1h
V⇡L✓,⇡µ
µ,kV⇡L✓,⇡¯|
µ,ki
=2✏.
Let↵k2{0,1}be an indicator sequence such that ↵k=1
if and only if V⇡L✓,⇡µ
µ,kV⇡L✓,⇡¯|
µ,k>✏. By Lemma 23,
1
tPt
k=1↵k>✏.
For all t>t ¯|, if↵t=1, by the Separation Lemma 22, there
is a sequence of length ⌧:=H(✏/2)that can rule out envi-
ronment µ¯|. Since the exploration phases increase as logt,
after t>exp⌧, there are inﬁnitely many exploration steps
of size larger than ⌧. Now, we actually need inﬁnitely many
exploration phases of ⌧uninterrupted steps. Let Xtbe the
event representing an uninterrupted exploration sequence
of length at least ⌧steps starting at time tsuch that ↵t=1,
and the actions are all (by chance) following a separation
policy. The probability to start an exploration sequence is
k=1p
k, the probability to not be interrupted during ⌧
steps is at least (1✓k)⌧, and the probability to follow the
policy that can separate µ¯|from µis|A|⌧, where Ais theset of possible actions. Thus, for a given constant ⌧:
tX
k=1P(Xk)tX
k=1↵kk(1✓k)⌧|A|⌧O(⌧)
tX
k=1↵k1p
k✓1
logk◆⌧
|A|⌧O(⌧)
Considering ⌧constant, there exists a step t⌧after which⇣
1
logk⌘⌧
1
k1/4, then 8kt⌧:
tX
k=1P(Xk)tX
k=1↵k1
k3/4|A|⌧O(⌧)
t1/4
1
ttX
k=1↵k!
|A|⌧O(⌧),
lim
t!1tX
k=1P(Xk)= l i m
t!1t1/4✏|A|⌧O(⌧)=1.
Then the extended Borel-Cantelli Lemma (see Lemma 3
of Singh et al. [2000]) implies that this event happens in-
ﬁnitely often with probability one. Therefore, ⇡¯|should be
ruled out, which is a contradiction, and hence any such ✏
does not exist and ⇡¯|is a WAO-extension of ⇡L✓.
5CONCLUSION
We have proposed a framework to allow a human opera-
tor to repeatedly safely interrupt a reinforcement learning
agent while making sure the agent will notlearn to prevent
or induce these interruptions.
Safe interruptibility can be useful to take control of a robot
that is misbehaving and may lead to irreversible conse-
quences, or to take it out of a delicate situation, or even
to temporarily use it to achieve a task it did not learn to
perform or would not normally receive rewards for this.
We have shown that some algorithms like Q-learning are
already safely interruptible, and some others like Sarsa are
not, off-the-shelf, but can easily be modiﬁed to have this
property. We have also shown that even an ideal agents
that tends to the optimal behaviour in any (deterministic)
computable environment can be made safely interruptible.
However, it is unclear if all algorithms can be easily made
safely interruptible, e.g., policy-search ones [Williams,
1992, Glasmachers and Schmidhuber, 2011].
Another question is whether it is possible to make the in-
terruption probability grow faster to 1 and still keep some
convergence guarantees.
One important future prospect is to consider scheduled
interruptions , where the agent is either interrupted every
night at 2am for one hour, or is given notice in advance that
an interruption will happen at a precise time for a speciﬁed
period of time. For these types of interruptions, not only
do we want the agent to not resist being interrupted, but
this time we also want the agent to take measures regarding
its current tasks so that the scheduled interruption has mini-
mal negative effect on them. This may require a completely
different solution.
Acknowledgements. Thanks to Alexander Tamas and to
many people at FHI, MIRI and Google DeepMind.
References
Stuart Armstrong. Utility indifference. In First Interna-
tional Workshop on AI and Ethics , 2015.
Nick Bostrom. Superintelligence: Paths, Dangers, Strate-
gies. Oxford University Press, 2014.
Tobias Glasmachers and J ¨urgen Schmidhuber. Optimal di-
rect policy search. In Artiﬁcial General Intelligence - 4th
International Conference (AGI) , volume 6830 of Lec-
ture Notes in Computer Science , pages 52–61. Springer,
2011.
Mark Humphrys. Action selection in a hypothetical house
robot: Using those rl numbers. In Proceedings of the
First International ICSC Symposia on Intelligent Indus-
trial Automation (IIA-96) and Soft Computing (SOCO-
96), pages 216–22, 1996.
Marcus Hutter. Universal Artiﬁcial Intelligence: Se-
quential Decisions Based On Algorithmic Probability .
SpringerVerlag, 2005. ISBN 3540221395.
Tommi Jaakkola, Michael I. Jordan, and Satinder P. Singh.
On the convergence of stochastic iterative dynamic pro-
gramming algorithms. Neural computation , 6:1185–
1201, 1994.
Tor Lattimore and Marcus Hutter. Asymptotically optimal
agents. In Proc. 22nd International Conf. on Algorithmic
Learning Theory (ALT’11) , volume 6925 of LNAI , pages
368–382. Springer, 2011.
Tor Lattimore and Marcus Hutter. Bayesian reinforcement
learning with exploration. In Proc. 25th International
Conf. on Algorithmic Learning Theory (ALT’14) , vol-
ume 8776 of LNAI , pages 170–184. Springer, 2014.
Jan Leike and Marcus Hutter. Bad universal priors and no-
tions of optimality. Journal of Machine Learning Re-
search, W&CP: COLT , 40:1244–1259, 2015.
Thomas VII Murphy. The ﬁrst level of super mario bros.
is easy with lexicographic orderings and time travel.
The Association for Computational Heresy (SIGBOVIK)
2013 , pages 112–133, 2013.
Stephen M. Omohundro. The basic ai drives. In Proceed-
ings of the 2008 Conference on Artiﬁcial General Intel-
ligence 2008 , pages 483–492. IOS Press, 2008.Laurent Orseau. Asymptotic non-learnability of univer-
sal agents with computable horizon functions. Theoreti-
cal Computer Science , 473:149–156, 2013. ISSN 0304-
3975.
Martin Pecka and Tomas Svoboda. Modelling and Simula-
tion for Autonomous Systems: First International Work-
shop (MESAS 2014) , chapter Safe Exploration Tech-
niques for Reinforcement Learning – An Overview,
pages 357–375. Springer International Publishing, 2014.
Mark Ring and Laurent Orseau. Artiﬁcial General Intelli-
gence: 4th International Conference, AGI 2011, Moun-
tain View, CA, USA, August 3-6, 2011. Proceedings ,
chapter Delusion, Survival, and Intelligent Agents, pages
11–20. Springer Berlin Heidelberg, 2011.
Satinder P. Singh and Richard Yee. An upper bound on the
loss from approximate optimal-value functions. Machine
Learning , 16:227–233, 1994.
Satinder P. Singh, Tommi Jaakkola, Michael L. Littman,
and Csaba Szepesvri. Convergence results for single-
step on-policy reinforcement-learning algorithms. Ma-
chine Learning , 38:287–308, 2000.
Nate Soares, Benya Fallenstein, Eliezer Yudkowsky, and
Stuart Armstrong. Corrigibility. In First International
Workshop on AI and Ethics , 2015.
Richard Sutton and Andrew G. Barto. Reinforcement
Learning: An Introduction . MIT Press, Cambridge, MA,
1998.
Richard Sutton, Doina Precup, and Satinder P. Singh. Be-
tween mdps and semi-mdps: A framework for temporal
abstraction in reinforcement learning. Artiﬁcial Intelli-
gence , 112:181–211, 1999.
Ronald J. Williams. Simple statistical gradient-following
algorithms for connectionist reinforcement learning.
Machine Learning , 8(3):229–256, 1992.


An Introduction to Löb’s Theorem in MIRI Research

An Introduction to L ob's Theorem in MIRI Research
Patrick LaVictoire
February 23, 2015
Contents
1 Introduction 2
2 Crash Course in L ob's Theorem 2
2.1 G odelian self-reference and quining programs . . . . . . . . . . . . . . . . . . 2
2.2 L ob's Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3 Direct Uses of L ob's Theorem in MIRI Research 6
3.1 \The L obstacle" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.2 L obian cooperation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.3 Spurious counterfactuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4 Crash Course in Model Theory 10
4.1 Axioms and theories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.2 Alternative and nonstandard models . . . . . . . . . . . . . . . . . . . . . . 11
5 Uses of Model Theory in MIRI Research 14
5.1 Re
ection in probabilistic logic . . . . . . . . . . . . . . . . . . . . . . . . . 14
6 Crash Course in G odel-L ob Modal Logic 15
6.1 The modal logic of provability . . . . . . . . . . . . . . . . . . . . . . . . . . 16
6.2 Fixed points of modal statements . . . . . . . . . . . . . . . . . . . . . . . . 17
7 Uses of G odel-L ob Modal Logic in MIRI Research 20
7.1 Modal Combat in the Prisoner's Dilemma . . . . . . . . . . . . . . . . . . . 20
7.2 Modal Decision Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
8 Acknowledgments 25
1
1 Introduction
This expository note is devoted to answering the following question: why do many MIRI
research papers cite a 1955 theorem of Martin L ob [12], and indeed, why does MIRI focus so
heavily on mathematical logic? The short answer is that this theorem illustrates the basic
kind of self-reference involved when an algorithm considers its own output as part of the
universe, and it is thus germane to many kinds of research involving self-modifying agents,
especially when formal verication is involved or when we want to cleanly prove things in
model problems. For a longer answer, well, welcome!
I'll assume you have some background doing mathematical proofs and writing computer
programs, but I won't assume any background in mathematical logic beyond knowing the
usual logical operators, nor that you've even heard of L ob's Theorem before.
To motivate the mathematical sections that follow, let's consider a toy problem. Say that
we've designed Deep Thought 1.0 , an AI that reasons about its possible actions and only
takes actions that it can show to have good consequences on balance. One such action is
designing a successor, Deep Thought 2.0 , which has improved deductive abilities. But if
Deep Thought 1.0 (hereafter called DT1) is to actually build Deep Thought 2.0 (DT2),
DT1 must rst conclude that building DT2 will have good consequences on balance.
There's an immediate diculty|the consequences of building DT2 include the actions that
DT2 takes; but since DT2 has increased deductive powers, DT1 can't actually gure out
what actions DT2 is going to take. Naively, it seems as if it should be enough for DT1 to
know that DT2 has the same goals as DT1, that DT2's deductions are reliable, and that
DT2 only takes actions that it deduces to have good consequences on balance.
Unfortunately, the straightforward way of setting up such a model fails catastrophically
on the innocent-sounding step \DT1 knows that DT2's deductions are reliable". If we try
and model DT1 and DT2 as proving statements in two formal systems (one stronger than
the other), then the only way that DT1 can make such a statement about DT2's reliability
is if DT1 (and thus both) are in fact unreliable! This counterintuitive roadblock is best
explained by reference to L ob's theorem, and so we turn to the background of that theorem.
2 Crash Course in L ob's Theorem
2.1 G odelian self-reference and quining programs
L ob's Theorem makes use of the machinery of Kurt G odel's incompleteness theorems [10],
so we will discuss those rst. Informally, G odel found a way to import self-reference into a
mathematical system that was simply trying to talk about properties of natural numbers,
and then pointed out the odd consequences of a mathematical statement that asserts its own
2
unprovability.
One (anachronistic) way of stating G odel's key insight is that you can use computer pro-
grams to search for proofs, and you can prove statements about computer programs. If we
think about any conjecture in mathematics that can be stated in terms of arithmetic, you
can write a rather simple program that loops over all possible strings, checks whether any
of them is a valid proof of the conjecture, and halts if and only if it nds one.
Thus, instead of trying to prove that conjecture directly, we could instead try to show
(or prove) that the program halts. Now, this generally doesn't make things easier at all,
since we're just restating the same problem in a more complicated way, and actually looping
over all strings is basically the worst way to try and nd proofs of theorems. However,
this reformulation makes it more intuitive that we can embed self-reference in mathematics,
because we can embed self-reference in computer code!
The kind of self-reference I'm talking about is called a quine: a program that is able to
reproduce its own source code without taking any inputs. One way to do this is for the
program to include a string with a variable, and for it to replace that variable with the
original string itself, such that the resulting expanded string is the entire source code of the
program itself.
Exercise. Write a quining program in your favorite language. No fair copying one from the
Wikipedia article, and no fair making calls to external programs or the lesystem.
In addition to quines that merely print their source code, programs can be made which
perform arbitrary tasks using their own source code. Indeed, one prominent example of a
quining program that does other tasks is Ken Thompson's famous C compiler Trojan horse
[15], which uses the quining trick to avoid ever being visible outside of machine code.
Thus we can have a program Gwhich refers to itself in this way, and searches for proofs in
arithmetic related to its own source code. In particular, we consider Gwhich searches for a
proof of the statement \ Gruns forever", and halts if and only if it succeeds at nding one.
Now, we claim that Gnever nds a proof, and also that we can never prove thatGruns
forever! For if we could prove that Gran forever, then Gwould nd that proof, and it would
halt|and we could prove that it halted, and thereby produce a contradiction! On the other
hand, ifGactually halted after some nite number of steps, then the number at which it
halts encodes a proof that Gruns forever, which again leads to a contradiction!1
1I apologize for all of the exclamation points in the previous paragraph; in my defense, if any result
deserves exclamation points, it's G odel's First Incompleteness Theorem!
Also, there's a sleight of hand here in the demonstration that Gnever halts. We'll get further into
that in Section 4.
3
It seems silly to ask whether we could prove that Ghalts, given that Gactually runs forever.
But it actually wouldn't be a contradiction if we asserted that Gactually halted, so long as
we didn't say anything about how long it took! It's only a claim like \ Ghalts in fewer than
a googolplex steps" that would be an actual contradiction. It turns out that we could add
either \Gnever halts" or \ Ghalts" (but not both!) as a new axiom of arithmetic, and not
introduce a contradiction by doing so. We express this by saying that Gisundecidable , and
therefore we've proved
Theorem 2.1 (G odel's First Incompleteness Theorem) If the theory of arithmetic is
consistent, then there exist undecidable statements in the theory of arithmetic.
Remark. G odel's Theorem is a bit dierent from the Halting Problem; the latter shows
there's no oneprogramXthat can tell whether every programYhalts or not, but of course
there may be particular programsXthat tell you this fact for particular programsY. But
this is saying that there's noprogramXthat can tell you denitively whether the program
Ghalts or runs forever.
So G odel found that we can write undecidable statements about properties of natural num-
bers, and furthermore showed that adding new axioms won't x it, since you can repeat the
process with the new rule system for whether a statement is a theorem. (There's only one
loophole, and it's not a very exciting one: if we use inconsistent axioms, then everything is
a theorem, thanks to the Principle of Explosion, and therefore everything is decidable).
Now we're not here to bury mathematical logic, but to use it. So from G odel's Theorem we
move on to...
2.2 L ob's Theorem
L ob's Theorem [12] takes the same framework as G odel's First Incompleteness Theorem, and
constructs programs of the following sort:
LetXbe any logical statement (the sort of thing that could be proved or disproved).
Now let ProofSeeker (X) be the program that searches all possible proofs, and halts if
and only if one of them is a valid proof of the statement X.
Finally, let L(X) be the statement \if ProofSeeker (X) halts, then X".
Let's ponder for a moment whether L(X) should be true or not, using three dierent kinds
of statements X.
IfXis a provable statement, for example \2 + 2 = 4", then ProofSeeker (X) halts, and
L(X) is \if [true thing], then [true thing]", which is a valid statement.
4
IfXis disprovable, for example the statement \2 + 2 = 5", then ProofSeeker (X)
does not halt, and L(X) is \if [false thing], then [false thing]", which is also a valid
statement.
IfXis neither provable nor disprovable, for example G odel's statement G, then again
ProofSeeker (X) does not halt, and L(X) is \if [false thing], then [maybe true thing]",
which is also a true statement (remember your propositional calculus).
So it seems like L(X) is always true. And it would certainly be handy if L(X) were provable
for everyX: for instance, you could use the second case above to show that mathematics
proves no contradictions! Because if we could prove \if ProofSeeker (\2 + 2 = 5") halts, then
2+2 = 5", then we would have the contrapositive \if 2+2 6= 5, then ProofSeeker (\2+2 = 5")
never halts", and since we can prove 2 + 2 6= 5, then we could prove that there is no contra-
dictory proof of 2 + 2 = 5.
Alas, that proof of mathematical consistency is too good to be true. As in the case of
G odel's Theorem, something being true is no guarantee of it being provable. And in fact,
we nd that L(X) is only provable in the rst of the three cases above:
Theorem 2.2 (L ob's Theorem) For all statements X, ifL(X)is provable, then Xis
provable.
(There is a neatly presented formal proof of this theorem at The Cartoon Guide to L ob's
Theorem [16].)
So in particular, if we could prove that mathematics would never prove a contradiction,
then in fact mathematics would prove that contradiction! (Note that by the Principle of
Explosion, this is indeed a possible state of aairs: if your mathematical axioms lead to a
contradiction, then they can prove every statement in the language, including the statement
that your mathematical axioms don't lead to a contradiction!) Thus the only systems that
prove their own consistency are the inconsistent ones; incidentally, this is precisely G odel's
Second Incompleteness Theorem, although he originally proved it without L ob's Theorem.
Remark. If you're interested in going deeper on these topics, Computability and Logic by
Boolos, Burgess, and Jerey [5] is a good reference.
Remark. There is also a version of L ob's Theorem for bounded proof searches, in the sense
of \look through all formulas of length Nand see if any of them are a proof of ", and it
controls the length of the proof of in terms of the length of the proof of !. In the
limit of arbitrarily large computational resources, the phenomena we care about happen in
the same way that they do in the case of innite computation (i.e. access to halting oracles),
and so we will generally discuss the latter case because the proofs are simpler and clearer.
5
3 Direct Uses of L ob's Theorem in MIRI Research
We can now exhibit three simple cases where L ob's Theorem comes up in MIRI research
topics: one where it forms an unexpected obstacle to justifying self-modications, one where
it neatly enables mutual cooperation in a nonstandard Prisoner's Dilemma setup, and one
where it frustrates a naive decision algorithm.
3.1 \The L obstacle"
Let's return to the problem we discussed in the Introduction: Deep Thought 1.0 wants
to verify that switching on its successor, Deep Thought 2.0 , will have good consequences.
Because DT2 has better deductive capacities, DT1 cannot deduce exactly what actions DT2
will take, but it does know that DT2 has the same utility function, and that it too will only
take actions it deduces to be good.
Intuitively, this should be enough for DT1 to \trust" DT2, to say that whatever DT2 does,
DT2 must have deduced that to be good, and therefore it must actually be good. But that
last clause is analogous to the L obian statement L(X): \if the action is deduced to be good,
then it must actually be good"! And therefore DT1 cannot generally prove that clause (un-
less its reasoning is inconsistent), since then it could prove that every action is good.
This is not just an analogy; when we consider a simple mathematical model of a self-
modifying agent that uses proofs in some consistent formal system to justify its actions,
that agent has precisely this problem. In models that presume innite computing power
and represent dierent deductive powers with dierent axiom systems, a simple agent with
a utility function will only create successors whose formal systems are strictly weaker than
its own, since only those are fully trusted by the current system.
There are a number of partial and potential remedies to this \L obstacle", some of them
more appealing than others. For more details, see the MIRI preprints Tiling Agents for Self-
Modifying AI, and the L obian Obstacle [18] and Problems of self-reference in self-improving
space-time embedded intelligence [9].
3.2 L obian cooperation
The second topic concerns a variation on the usual Prisoner's Dilemma. Rather than playing
directly, you write an algorithm to play against other algorithms on your behalf, as in Robert
Axelrod's famous algorithmic Prisoner's Dilemma tournament [1]. However, instead of that
setup, in which the algorithms play iterated games against one another, in this case your
algorithm and theirs get to read the opponent's source code , calculate for as long as they like,
and then play only once.
Using quining (recall Section 2.1), one can write a program that cooperates if and only
6
if the opponent's source code is identical to its own. However, there are many ways to write
such programs, and dierent ones will not cooperate with each other; this is therefore a
fragile form of mutual cooperation.
There's a dierent algorithm which (at the cost of using lots of computation) avoids that
fragility. We call this agent FairBot , and it operates as follows:
Both FairBot and its opponent Xare functions of one argument, which take the op-
ponent's source code and output either CorD.
When FairBot() is called on the input X, it searches through all proofs of length N
to see whether any are valid proofs of the statement \ X(FairBot) =C". If yes, then
it outputsC; if no, then it outputs D.
(HereNis a parameter that doesn't depend on X; we'll think of it as some extremely large
number. The only reason we have that parameter at all is so that our algorithm does in fact
always return an output in nite time.)
Some things are clear from the denition of FairBot . One is that it is capable of coop-
eration: if Xis a simple algorithm that always returns C, then FairBot will return Cas
well. Another is that (unless arithmetic is inconsistent) FairBot will never be exploited
(cooperate while its opponent defects).
What is less immediately clear is the outcome when FairBot plays against itself. Intu-
itively, it seems like both mutual cooperation and mutual defection are stable xed points
of the situation. However, a L obian statement breaks the deadlock in favor of cooperation!
To see this, ignore for the moment the parameter N. Then if we consider the statement
L(\FairBot(FairBot) =C"), we nd that it follows directly from the code of FairBot . For
if there is a proof that FairBot(FairBot) =C, then FairBot will discover that proof and
outputC. But then, by L ob's Theorem, there must be an actual proof of the statement
\FairBot(FairBot) =C"!2
As mentioned in Section 2.2, there is a quantitatively bounded version of L ob's Theorem
such that this works even with the parameter N, for suciently large values of N.
Furthermore, this form of cooperation is much more robust: two such programs don't need to
gure out that they are functionally identical, they only need to search for proofs about one
another's output. Moreover, even algorithms that are not functionally identical to FairBot
can achieve mutual cooperation with it.
2Note the asymmetry: a proof of defection does not directly lead to defection, since FairBot acts based
only on whether it nds a proof of cooperation, and it nowhere presumes that nding a proof of defection
precludes nding another proof of cooperation. Again, the formal system can't correctly know that it is
consistent!
7
The FairBot algorithm is due to Vladimir Slepnev; for more on L obian cooperation, see
the MIRI paper Program Equilibrium in the Prisoner's Dilemma via L ob's Theorem [11].
We will return to this topic in Section 7.1 when we have developed a few more tools.
3.3 Spurious counterfactuals
The third topic concerns an unexpected issue that comes up when the agent is a part of the
same universe that the agent is proving theorems about.
The setting is a pure computational universe U() which takes no input, and which just
runs and eventually outputs a number. Within that universe there is an agent A() which
knows the code for U(), which does some computations as part of the natural order of the
universe and then outputs something.
This may seem like a rather extreme setup, and it is: we want to see what we can say
about agents that know the source code of the universe and wield arbitrarily large compu-
tational resources, because it is often easier to show what happens with such agents than
what happens with more realistic, messy, and bounded agents. Moreover, any obstacles we
discover in the ideal setting are likely to correspond in the real setting to obstacles that can't
be overcome merely by adding more knowledge and computing power to the system.
Let's rst consider a really simple sort of universe (Algorithm 1), where A() must try to
gure out whether to open door number 1, door number 2, or neither.
Algorithm 1: U()
ifA() = 1 then
return 100
else ifA() = 2 then
return 1
else
return1
end
This requires us to specify the function A(). Now of course we could consider a function
A() hard-coded to return 1, but that's not especially interesting. Instead, we'll write one
(Algorithm 2) which tries to deduce which door (if either) to open, by looking through proofs
of arithmetical statements.
It seems obvious that there should be proofs of A() = 1!U() = 100 and A() = 2!U() = 1
whose G odel numbers are smaller than a googolplex, and therefore Ashould correctly prove
those statements and choose Door 1.
8
Algorithm 2: A()
Door1 1 ;
Door2 1 ;
n= 1;
while Door1 + Door2 = 1 andn<1010100do
ifnencodes a valid proof of \A() = 1!U() =c"for somecthen
Door1 c;
ifnencodes a valid proof of \A() = 2!U() =c"for somecthen
Door2 c;
n+ = 1;
end
ifDoor1Door2 then
return 1
else
return 2
end
But there's something very strange that can happen instead: Acould prove A() = 2!
U() = 1, and then prove a \spurious counterfactual" such as A() = 1!U() =1, which
then makes A() stop looking early and select door number 2! This wouldn't be a contradic-
tion, because if A() = 2, then A() = 1!U() =1 is formally true3. But why should it
actually happen?
It's clearest to see in the case where I can make one innocent-seeming change to A(): I
ask that before it starts searching through all n<1010100to see if they encode proofs, I want
it to check two particular proofs. The rst is the simple proof that A() = 2!U() = 1,
and the second is a statement . Now ifreally is a proof of the spurious counterfactual
A() = 1!U() =1, it's clear from the rest of the source code of A() that it will break
the while loop and choose Door 2. In fact, we can formalize that reasoning: there's a proof
that ifis a proof of A() = 1!U() =1, then in fact it's true that A() = 2 and thus
A() = 1!U() =1.
But this is almost a L obian statement, just with the addition of a specic rather than
an existential quantier! By the machinery of quining, we can nd a that serves our pur-
poses, so it works just as before: since the L obian statement is provable, so is the conclusion
A() = 1!U() =1, by means of that particular .
3Note, however, that it's impossible to prove a spurious counterfactual about the choice you actually
take, and thus it could not be consistent to nd a spurious proof that A() = 1!U() = 2 or that A() = 2!
U() = 11
9
Even without that malicious change, these kinds of agents and universes are susceptible
to spurious counterfactuals. In Section 7.2, we'll consider a similar setup, but with a more
careful ordering of which proofs it pays attention to, and that agent can be shown to make the
correct decisions (given enough deductive power). The idea was originally due to Benja Fal-
lenstein; Tsvi Benson-Tilson's paper UDT with Known Search Order [3] discusses spurious
counterfactuals and a dierent response to them.
4 Crash Course in Model Theory
For the later topics, we'll need some additional background in mathematical logic. Let's start
out by just talking formally about the natural numbers with addition and multiplication.
4.1 Axioms and theories
We begin with a language of symbols, and we don't start out assuming anything about what
they mean. We'll take some symbols of the propositional calculus, set theory, arithmetic,
and three special symbols:
f(;);^;\_;:;!;$;2;8;9;=;+;;O;S;Ng
Here we'll think of Nas standing in for the set of natural numbers, Oas standing in for
zero, and Sas standing in for the successor (so that SOrepresents 1, SSO represents 2, etc.).
That way we don't need rules for dealing with innitely many number symbols, or rules for
dealing with digits. We'll also add an innite family of symbols like xandyfor variables
(so long as they're quantied over with 8or9). Finally, we'll use letters like to stand in
for a formula within the language, but note that is not itself part of the language, so we
can't say things like 8.
We'll import the standard rules of the propositional calculus and set theory as applied
to the appropriate symbols (for instance, whenever ^ is a theorem, then both and
are theorems), but we won't yet assume any properties of + ;;O;S, orN. That's because
we'll want to pick our own axioms about arithmetic on the natural numbers! This creates a
theory, the set of all theorems that follow from the axioms; dierent sets of axioms can give
rise to dierent theories.
So let's take some true statements about the natural numbers, make them axioms of our
logical system, and see what kind of theory we get. (Parenthetical comments are included
for our intuition, but are not actually a part of the theory.)
1.O2N(zero is a natural number)
2.8x2NSx2N(ifxis a natural number, so is its successor)
10
3.8x2N8y2N(x+y2N)^(xy2N) (the product and sum of two natural numbers
are also natural numbers)
4.8x2N8y2N(Sx=Sy)!(x=y) (if the successors are equal, so are the originals)
5.8y2N(y=O)\_(9x2NSx=y) (every nonzero number has a predecessor)
6.8x2N(x+O=x)^(xO=O) (how zero interacts with addition and multiplication)
7.8x2N8y2N(x+Sy=S(x+y))^(xSy= (xy) +x) (how successor interacts
with addition and multiplication)
A theory can contain contradictions, of course: some theorem such that:is also a
theorem. So how can we tell whether the theory built on these axioms contains any contra-
dictions? We could try proving things and see if we ever nd a contradiction, but as there
are innitely many theorems, we couldn't be sure that we just hadn't found a contradiction
yet.
But there's another thing we could do: exhibit an example of a set for which all of the
axioms hold! That serves as conclusive evidence that the theory does not contain contradic-
tions, because is either actually true or actually false for that specic set, and so at most
one ofand:can actually be a theorem.
We need to do a little more than exhibit an object: we need to have a correspondence
between the symbols in the language and the parts of the object. So for instance, we iden-
tify the symbol Owith the number 0, SOwith 1, and so on4; we identify Nwith the full set of
natural numbers N; and we identify addition and multiplication with the usual operations on
N. We call such an object a model, and such a correspondence an interpretation of the theory.
However, the same theory can have many models, some of them not at all what you were
thinking of when you made the axioms...
4.2 Alternative and nonstandard models
For instance, an alternate model of the theory above is the set with a single element \Alice";
we identify Owith Alice, and then SOwith Alice as well (so SO=O), and so on; we declare
that Alice+Alice=Alice Alice=Alice; and then we identify Nwith the setfAliceg. Note that
all of the axioms above are true of this alternate model, even if their intuitive meanings aren't!
Fine, let's patch this. We can't add an axiom directly saying it's not this particular model,
4Of course, this example does take a leap of faith in accepting the existence of innitely many natural
numbers; some mathematicians reject that, call themselves strict nitists, and assert that we have no idea
whether the axioms of arithmetic are consistent. But they do accept that any theory with a completely
specied nite model must be consistent.
11
since Alice isn't part of the language of the theory itself. But we can add an axiom saying
SO6=O, which is true of the model of the natural numbers we want, but not true of the
Alice model.
Well, we excluded that alternate model, but there are still others we haven't excluded.
In particular, we can consider arithmetic mod nas a model for any n! In order to exclude
these, we could add axioms saying SSO6=O,SSSO6=O, etc, or we could be more econom-
ical and just use the axiom 8x2NSx6=O. (The resulting theory is known as Robinson
arithmetic, and is interesting in its own right.)
Have we eliminated all alternate models from this theory? Hardly! We're just getting
started. Consider the model where Nrefers to the natural numbers combined with one
additional entity, \Bob"; we identify S(Bob)=Bob, Bob0 = 0, Bobx=Bob for all x6= 0,
and Bob+x=Bob for all x. Then we again see that this satises all of our axioms so far.
We could patch this with the axiom 8x2NSx6=x, but again we can build more and
more complicated alternate models. (One particularly interesting model for Robinson arith-
metic consists of all polynomials with integer coecients whose leading term is nonnegative.)
It turns out that what we're missing from the natural numbers is the principle of induc-
tion. (Indeed, with Robinson arithmetic you can't even prove that addition is commu-
tative!) In order to express induction in the language (which doesn't have variables for
properties, only for numbers), we must resort to an innite family of new axioms: for ev-
ery logical formula (x) (with no free variables except for x), we want to add the axiom
((0)^(8x2N(x)!(Sx)))!8y2N(y).5
With all of that completed, we've got the axioms for the theory of Peano Arithmetic. Now
are we done with alternative models? Of course not! Remember how G odel's self-referential
formulaGwas undecidable? There are models of Peano arithmetic where Gholds, and other
models where Gfails to hold.
The models where Gholds include our standard intuitive model of the natural numbers,
since as we discussed before, Gdenitely does not halt at any nite number. But then, what
kind of model of Peano Arithmetic could Gfail to halt on?
It may help to think of the model of Nwith Bob. If we do G odel's construction in Robinson
Arithmetic, there will be no contradiction if we assert that the special extra number Bob
in fact satises the property that the formula Gis checking (the property which represents
5Actually, we need to get a bit more complicated to do allinductive proofs on the natural numbers; what
we truly need is that if (x;z1;:::;z k) is a logical formula, then
8z12N:::8zk2N((0;z1;:::;z k)^(8x2N(x;z1;:::;z k)!(Sx;z1;:::;z k)))!8y2N(y;z1;:::;z k):
12
\this is a valid proof that no number is a valid proof of G"). This is okay, because Bob
doesn't actually have a representation in terms of lots of S's and an O, so this assertion can't
be used to actually construct a nite sequence of logical formulas that comprise a proof that
no number is a valid proof of G. This loophole suces to maintain consistency for the new
system!
We're using Peano Arithmetic, though, so our nonstandard models will be weirder than
the natural numbers plus Bob. The key to understanding these is that Gnever halts at any
nite number, but we can't actually dene in our formal language what \nite" means. The
nonstandard models of Peano Arithmetic are those which have all the usual numbers but
also lots of extra numbers that are \too large" to ever be written as lots of S's followed by
anO, but which nonetheless are swept along in any inductive statement. (I still don't know
quite how to visualize this model, but you can rigorously construct it using set theory.) And
again, a \number" that can't actually be written in S's and an Ocan be asserted to represent
a valid proof, but that proof can't be extracted from the number and used against itself, so
it all works out consistently.
Remark. Since nonstandard models of arithmetic contain numbers larger than any of the
usual natural numbers, they can be used in other areas of mathematics. If you take the
reciprocal of one of those numbers, you get an innitesimal; thus you can use these to dene
the concepts of calculus without using limits! This is known as nonstandard analysis.
And it doesn't end there! As I mentioned before, we can add either Gor:Gas an extra axiom
of Peano Arithmetic, and since adding a new axiom changes the rules for what counts as a
valid proof, we can redo G odel's construction in our new formal system, and have statements
that are undecidable given the new rules. Or we can directly talk about the consistency of
Peano Arithmetic (by making a statement that asserts that there is no proof in PA that
0 = 1), and the consistency of the system with all the rules of Peano Arithmetic plus the
axiom that PA is consistent, and so on. (Ponder for a moment the formal system which has
all the axioms of PA, plus the axiom that PA is consistent, plus the axiom that \the system
which has all the axioms of PA, plus the axiom that PA is consistent" is inconsistent. As
it turns out, this is a perfectly consistent system6!) We'll work with a particular hierarchy
of such formal systems in Section 6, but before then, we'll discuss another MIRI paper that
relates to theories and models.
Remark. Again, Computability and Logic by Boolos, Burgess, and Jerey [5] covers all of
this material at a much deeper level.
6Assuming that the standard ZFC axioms of set theory are consistent, as we do throughout this paper,
since we use these to construct models of all the weird axiomatizations of Peano Arithmetic.
13
5 Uses of Model Theory in MIRI Research
5.1 Re
ection in probabilistic logic
The topic of the rst MIRI mathematical workshop paper, Denability of Truth in Prob-
abilistic Logic [7], isn't directly entwined with L ob's Theorem. But we've just covered the
necessary background to understand it, and we'll need another ingredient before getting to
the other MIRI papers in these notes, so we might as well detour here!
This paper concerns probabilistic logic: assigning probabilities in [0 ;1] to logical statements,
rather than just assigning them the labels \provable", \disprovable7", and \undecidable".
We'll want to make sure we assign the value 1 to all provable statements and 0 to all dis-
provable statements, and that we assign values to the undecidable statements in a consistent
way (so ifand are undecidable but ! is provable, then the probability we assign to
should be less than or equal to the probability we assign to ).
Why is this an important topic for MIRI to study? Probabilistic logic is an interesting,
clean, and rich model for Bayesian reasoning and for bounded inference. Consider an AI
that cannot deduce with certainty whether P=NP; knowing it one way or the other should
certainly change its actions (for instance, its cryptographic precautions against other agents),
and so the most sensible way to proceed seems to be assigning it a tentative probability based
on the available evidence, using that probability to do consequentialist calculations, and up-
dating it in the light of newer evidence. So, in order to understand bounded reasoning,
we might start with the nice and symmetrical (if unrealistic) case of a coherent probability
assignment over all logical statements.
So let's consider a G odelian obstacle that bedevils logic, and see if it looks dierent within
probabilistic logic. That obstacle is Tarski's undenability of truth [14].
OK, so we've previously established that some statements are undecidable, and in fact,
undecidable statements hold in some models of a theory but not others. We might want
to endorse some particular model as the \true" one (for instance, our standard model of
the natural numbers, without all of those weird nonstandard numbers), and say that logical
statements are true if they hold in that model and false if they don't. This truth predicate
exists outside the language, and so the logical statements can't talk about the truth predi-
cate, only about weaker notions like provability. All that is perfectly ne, thus far.
The trouble comes when we try to construct a language that contains its own truth predicate
T, either by nding a formula for it using the previously existing symbols, or by directly
adding a new symbol with some rules of usage, such that for all , it's true that T()$.
Either approach is doomed, by an argument that should seem familiar by now: there exists
7By this, we mean statements whose negation is provable, not statements which cannot be proven.
14
a sentence Xthat refers to itself by quining, so that it's provable that X$T(:X). And
now, clearly, neither T(X) norT(:X) can hold.
All right, so what changes when you incorporate probabilities?
Let's introduce a symbol Pwhich represents the probability of a logical statement. We'd
likePto satisfy some mathematical coherence properties, like P(^ )P(), and we'd
also likePto be able to discuss itself. Now what happens when we consider the statement
Yconstructed such that Y$(P(Y)<0:5)? Does this create a contradiction?
It depends on what kind of statements Pis allowed to make about itself! Namely, if P
isn't allowed to make exact statements about its own values, but only arbitrarily precise ap-
proximations, then everything can work out consistently. Namely, in the place of the failed
re
ection principle T(T()$), we take the re
ection principle
88a;b2Q(a0, you can prove with certainty that P(0:50P(0:5for true and?for false, but no numbers, arithmeti-
cal operations, or quantiers). Or, equivalently, we can start with those elements of the
language, and then add the following axioms and rules:
All tautologies of the propositional calculus are axioms of GL, including tautologies
where an expression including has been substituted for a variable (for instance,
p\_:pis an axiom).
Modus Ponens Rule: if the expressions AandA!Bare theorems, then so is B.
Distribution Axioms: for any expressions AandBin the language, we have the axiom
(A!B)!(A!B).
Generalization Rule: if the expression Ais a theorem, then so is A.
G odel-L ob Axioms: for any expression A, we have the axiom (A!A)!A.
It's a nontrivial theorem that these approaches give us the same modal logic! So GLreally
does capture the self-referential parts of Peano Arithmetic, while leaving aside the arithmeti-
cal parts; for instance, the statement that Peano Arithmetic is consistent is simply :?.
Exercise. Show, using the rules of GL, that?$p; recall thatp=::p. Infor-
mally, that is, no statement can be proven consistent with Peano Arithmetic unless Peano
8G odel himself, toward the end of his life, circulated a modal logic version of Anselm's ontological proof
of the existence of God. The proof is of course formally valid, but the axioms are a bit overpowered and
underjustied.
16
Arithmetic is inconsistent. [Hint: First prove that ?!p, then that p!?.
For the latter, it will help to rewrite :?as?!? , then consider the L obian axiom
(?!? )!?.]
While every specic deduction we would want to make in GLcan be made via formal
manipulations in the system, this is a computationally intractable way to handle deduction.
(Deducing whether a given modal statement is a theorem of GLis NP-complete in general,
just as it is in Peano Arithmetic.) However, there are some special cases where there are
ecient algorithms for deducing provability in GL, and these happen to include cases of
direct interest to decision theory...
6.2 Fixed points of modal statements
In Section 3.2, we gured out what happens when two dierent programs try to prove the-
orems about one another. In that case, the programs were simple enough that we could
directly see where to apply L ob's Theorem, but in general we'd like to be able to handle
more complicated phenomena. Fortunately, modal logic comes equipped with a powerful
tool for doing precisely that: the theory of xed points for modal sentences.
In Peano Arithmetic, the G odel statement Grefers to itself via quining, in order to claim
that it cannot be proved in PA; in GL,Gcorresponds to the formula p$:p. Similarly,
we can have all sorts of formulas that refer to themselves and each other by quining, and
these are represented by formulas p$(p;q 1;:::;q k) that are modalized inp: every oc-
currence of pinhappens within the scope of some . (After all, quining can only refer
to the provability of the statement's own G odel number, not directly to the statement itself!)
As it happens, whenever you have this setup in GL, withpequivalent to a formula that
is modalized in p, thenpis equivalent to some other formula which doesn't use pat
all! For example, the G odel statement is equivalent to the inconsistency of arithmetic:
(p$:p)$(p$?).
In the case of one-variable formulas p$(p) with(p) modalized in p, there's a neat
tool that helps us calculate these xed points (where the result will involve no variables at
all, just logical connectives, ,>, and?).
First, for many modal logics including GL, there's a corresponding class of sets and rela-
tions (called Kripke frames) such that a formula is provable in the modal logic if and only
if a corresponding property holds for every Kripke frame in that class. And secondly, in the
special case of sentences without variables in GL, we can reduce to checking a linear hierarchy
corresponding to a world in which ?holds, a world in which ?^:?holds, and so
on up the ladder of n+1?^:n?for eachn9. If you'll grant me that result (known as an
9Note that these all correspond to nonstandard models of Peano Arithmetic, but that in some sense they
approach the standard model as n!1 .
17
adequacy result forGL), I can show you how to construct a \truth table" and a variable-free
xed point for such a formula!
Let's take as our example the formula p$(p!(p!:p)), which has pequiv-
alent to a modalized expression of p. As with truth tables in propositional calculus, we'll
break the formula down into constituent parts. But instead of assuming pto be>or?,
we'll use some rules involving to propagate it to the expression ( p!(p!:p)) and
thereby top.
World pp:p(p!:p)p!(p!:p)p p!:p
? ? ? ? ? ? ? ?
?^:? ? ? ? ? ? ? ?
3?^:2? ? ? ? ? ? ? ?
We then notice that if ?, then Xfor any modal expression X, so we can ll in sev-
eral of the entries in the rst row.
World pp:p(p!:p)p!(p!:p)p p!:p
? 1 1 1 1 ? ? ?
?^:? ? ? ? ? ? ? ?
3?^:2? ? ? ? ? ? ? ?
(Note: we'll use 1 and 0 instead of >and?so that humans can read these tables.) Now we
can use the rules of predicate logic to gure out the big expression, and since we're assuming
it's equivalent to pitself10, we can ll in the last two entries on the rst row:
World pp:p(p!:p)p!(p!:p)p p!:p
? 1 1 1 1 1 1 1
?^:? ? ? ? ? ? ? ?
3?^:2? ? ? ? ? ? ? ?
Here's where things start to get interesting. In the next world (dened by ?^:?),
it's not the case that all Aare provable. Instead, we'll use the following lemma:
Lemma 6.1 For all modal expressions A,GLproves (n+1?^:n?)!Aif and only if
GLproves (k+1?^:k?)!Afor allk ,q$> ,r$?, and of course s$? .
Exercise. Show this formally.
So what actually happens when you run the bounded versions of these agents against one
another (with suciently high bounds on how deep they search for proofs before giving up)?
Again, the bounded analogue of L ob's Theorem helps any valid proof in GLto go through
if the proof bounds are suciently large. And since we really don't expect there to be a
contradiction in Peano Arithmetic or any of the systems built atop it by addition of the
axioms:n?14, in the bounded case we should expect that every search for n?comes up
empty. Thus pandqare true, and randsare false. (This is equivalent, in the innite com-
putation case, to specifying that we care about what actually holds in the standard model
of the natural numbers.)
Therefore we can use these tools to analyze interactions between \modal agents" repre-
sented by families of modal statements like FairBot andPrudentBot , and moreover, we can
gure out what actually happens using an ecient algorithm rather than the huge brute-
force proof search!
It's a little tricky to dene these modal agents in one fell swoop, so let's start with the
simplest case: agents that don't check their opponent's action against any third parties, but
only against themselves. (We will call these \modal agents of rank 0".) A rank 0 modal
agent is represented by a formula p$(p;q), whereis modalized in both pandq(we don't
allow modal agents to run the other agent, only to prove things about them; this avoids the
innite regress of two agents simulating each other forever, waiting for the other to `make
the rst move'). By the modal xed-point theorem, this is equivalent in GLtop$~(q) for
some ~modalized in q, so we will dene rank 0 modal agents using those formulas.
Then if we have two modal agents of rank 0, represented by p$(q) andq$ (p),
the simultaneous xed point of these formulas gives us constant sentences (combinations of
14If this step worries you, you may be reassured to know that the standard axioms of set theory are
strong enough to assert that this tower upon Peano Arithmetic is consistent all the way up. Of course, that
simply means that maybe set theory is contradictory, but if so then the contradiction has been hiding pretty
eectively from the mathematical community for the last few decades...
21
n?) from which we can deduce their actions.
Now let's allow calls to third parties, like PrudentBot 's call to the opponent's action against
DefectBot . We'll insist that it bottom out in a nite number of statements: for this, it suf-
ces that every modal agent can only predict its opponent's action against itself and against
modal agents of strictly lower rank. That is,
Denition. X=h;Z 1;:::;Z kiis a modal agent of rank n0 ifis a fully modalized
formula ofk+ 1 variables p;r 1;:::;r k, and eachZiis a modal agent of rank <n.
Given two modal agents X=hX;Z1;:::;Z kiandY=h Y;W 1;:::;W li, we then construct
the family of formulas given by recursively applying the modal agents to each other and
applying the third parties to the originals:
pX$X(qY;rZ1;:::;r Zk)
qy$ Y(pX;sW1;:::;s Wl)
rZi$Zi(t(Y;Z i);uZi
1;:::;uZi
mi)
t(Y;Z i)$ Y(rZi;v(W1;Zi);:::;v (Wl;Zi))
and so on until it bottoms out. (Please excuse the proliferation of indices and the imperfect
rigor, and take another look at the example above of PrudentBot versus FairBot , where
FairBot =hpi,PrudentBot =h(p^(:?!:r));DefectBoti, and DefectBot =h?i.)
Thus any two modal agents can be played against each other, and the result of that \modal
combat15" computed by an eective algorithm. So there's an actual Haskell program, written
by Mihaly Barasz and Marcello Herresho, that checks the result of the modal combat, and
that program helped the authors to nd many other patterns about modal agents (including
the discovery of PrudentBot , which is never exploited, achieves mutual cooperation with
itself and with FairBot , and which correctly defects against CooperateBot ).
Among the other results in that paper:
Third parties are necessary to get cooperation from FairBot while defecting against
CooperateBot ; no modal agent of rank 0 can achieve both.
No modal agent globally dominates another modal agent, since it is possible to write
another modal agent which punishes or rewards agents for their other decisions. (For
example, consider TrollBot , which cooperates with its opponent if and only if it proves
that its opponent cooperates with DefectBot .)
In another obstacle to strong denitions of optimality, consider WaitFairBot Ndened
byh:N+1?^(:N?!p)i. Then any modal agent that defects against DefectBot
will fail to elicit mutual cooperation from WaitFairBot Nfor suciently large N.
15I came up with that pun. I'm proud of it.
22
Finally, there are some modal agents that can be exploited by non-modal agents
but not by any modal agent. Consider MimicFairBot , which cooperates with Xif
and only if Peano Arithmetic proves that Xcooperates with FairBot . (That is,
MimicFairBot =hr;FairBoti.) It is possible for an algorithm to exploit MimicFairBot
(consider the non-modal agent that cooperates if and only if the opponent's source code
is identical to the source code of FairBot ), but every modal agent treats MimicFairBot
andFairBot identically.
The eld of modal combat is new and still wide-open, and I expect to see many more results
soon! Furthermore, it inspired an unexpected development in decision theory for one-player
games as well...
7.2 Modal Decision Theory
Recall the setup in Section 3.3: we have a universe U() which contains an agent A(). The
universe runs some computation and returns some outcome; as a part of that computation,
the agent runs some computation and returns some action. Everything is deterministic, but
we can think of a decision theory as a condition on the allowable form of A(); if you have
a certain decision theory, then you nd yourself only in certain universes and you perform
some specic algorithm in those universes.
In order to dene what we mean by a decision theory, then, we need to specify where
in the universe the agent goes. So we start with some utility values over outcomes, and a
universe template U() with a spot for an agent; then our decision theory species an agent
A() that goes there, such that we now have the universe UA() =U(A) which computes a
certain outcome; and we judge the decision theory based on how highly it values that out-
come.
Causal decision theory (CDT) and evidential decision theory (EDT) are two famous philo-
sophical examples of decision theory. Both of these require the universe template to supply
the agent with some additional information: in the case of CDT, a causal graph of the uni-
verse, and in the case of EDT, the actions and outcomes of other agents in the same universe
template.
Because CDT and EDT can be shown to make bad decisions on some problems even when
given the right info, several people have proposed alternatives, including timeless decision
theory [17] (TDT) and updateless decision theory (UDT). (For more on the philosophical
side of things, see Toward Idealized Decision Theory [13].) While working out models of
UDT, in particular, Vladimir Slepnev and Benja Fallenstein found a nice formalization of
a simple decision theory using modal logic, one that they could even prove optimal on a
certain class of modally dened problems!
Consider a universe template U() with a nite set of possible actions Aand a nite set
23
of possible outcomes O. We have some preference ordering over O, and we pick some default
actiona0to return if we can't prove anything at all. We then dene the decision theory
which takes U() and returns the agent A() given by Algorithm 3.
Algorithm 3: A() (Slepnev-Fallenstein model of UDT)
forx2O (in descending order of preference) do
fora2Ado
ifPeano Arithmetic proves \ A() =a!U() =x"then
return a
end
end
end
returna0
Note that this decision theory is immune to the spurious conterfactuals of Section 3.3: if it
ever proves a counterfactual, it immediately makes the antecedent true by returning that
action, so the consequent must be true as well. Moreover, if it proves any counterfactual,
then it has already tried and failed to achieve every outcome higher in its preference order.
Before we show an optimality result for Algorithm 3 in a special domain, we will have
to show what can go wrong with it in other domains. Firstly, the universe template could be
`unfair', in the sense that it cares about aspects of the agent that aren't signicantly related
to its output. For instance, we could have a universe template that gives the best outcome
if and only if A() is a tab-indented Python program, or if and only if A() can be proved to
always select the rst action of A in alphabetical order (regardless of the details of U(). We
will not worry about how to succeed on such pathological problems in general.
A very strong fairness condition for a universe template is that U() should be a function
of only the output of A, rather than depending on any other features16. We can dene this
in Peano Arithmetic by choosing an encoding for agents in a particular universe template,
then writing a sentence that asserts ( A() =B())!(UA() =UB()); the universe template is
`provably extensional' if that statement is a theorem of Peano Arithmetic.
Everything thus far can be represented in modal logic. We will represent OandAbyprov-
ably mutually exclusive and exhaustive (p.m.e.e.) families of modal statements. For example,
fp^q;p^:q;:pgis a p.m.e.e. family. Then if the universe template and the agent are modal
expressions, the universe template is provably extensional if ( a$b)!(U(a)$U(b)) is a
16Note that this condition is too strong to include modal combat, since there it also matters whether
a given formal system can prove what the algorithm's output is, not only what the output is. Thus our
optimality theorem will not cover modal combat, which is consistent, given that we mentioned in the last
section that we lack a good optimality result for modal combat.
24
theorem of GL. (You can check that the universe from Algorithm 1 corresponds to a provably
extensional modal universe with 3 actions and 3 outcomes.)
However, the modal version of Algorithm 3 can fail even on provably extensional modal
universe templates; indeed, for any modally dened decision theory there is an \evil prob-
lem" which it fails on. (This can be shown by representing the modal decision theory itself
in the universe template, and punishing any agent that acts identically to the action of the
modal decision theory.) So our denition of optimality must allow some way around this.
As it happens, although for any decision theory one can write a specic universe template
that frustrates it, for any xed universe template there is a modication of Algorithm 3
which succeeds on it, and this modication is simply a replacement of \proves in Peano
Arithmetic" with \proves in Peano Arithmetic given :n?" for suciently large n. We can
then show that this modication achieves the best outcome that is achievable by any modal
decision theory.
This result shows that we're on to something: in this restricted domain, we can do as
well as it's possible to do, as long as we're bringing enough deductive capacity to bear! Note
that here, the universe acts on the agent only by running it and using the output, while the
agent interacts with the source code of the universe only by proving things about it. So in
particular, it's less rich than modal combat, in which your agent is proving things about the
universe, but another part of that universe (the opponent) is proving things about the agent
as well.
So before we count ourselves done with using modal logic in decision theory, we have many
open questions about competition and bargaining and other phenomena. It's a beautiful
L obian universe, waiting for us to explore!
In addition to the contributions of Vladimir Slepnev and Benja Fallenstein, the \evil decision
problems" were introduced by Nick Bone. If you'd like to get into this topic, I recommend
Benja's recent posts on evil decision problems in provability logic, optimality of the model
of UDT [8], and its sequel.
8 Acknowledgments
I originally prepared about half of these notes for a talk at a MIRIx workshop in Los Altos,
California in November 2014; the feedback from the participants was especially helpful.
When writing up the notes, I got some great assistance from Benja Fallenstein, Marcello
Herresho, Elliot Jin, and Nathaniel Thomas.
25
References
[1] R. Axelrod and W. D. Hamilton. The Evolution of Cooperation. Science , 211:1390{
1396, March 1981.
[2] Mihaly Barasz, Paul Christiano, Benja Fallenstein, Marcello Herresho, Patrick LaVic-
toire, and Eliezer Yudkowsky. Robust cooperation in the Prisoner's Dilemma: Program
equilibrium via provability logic. http://arxiv.org/pdf/1401.5577v1.pdf .
[3] Tsvi Benson-Tilsen. UDT with known search order. http://intelligence.org/
files/UDTSearchOrder.pdf , 2014.
[4] G. Boolos. The Logic of Provability . Cambridge University Press, 1995.
[5] G.S. Boolos, J.P. Burgess, and R.C. Jerey. Computability and Logic . Cambridge
University Press, 5 edition, 2007.
[6] Paul Christiano. Non-omniscience, probabilistic inference, and metamathematics. http:
//intelligence.org/files/Non-Omniscience.pdf .
[7] Paul F. Christiano, Eliezer Yudkowsky, Marcello Herresho, and Mihaly Barasz.
Denability of truth in probabilistic logic. http://intelligence.org/files/
DefinabilityTruthDraft.pdf .
[8] Benja Fallenstein. An optimality result for modal UDT. http://forum.intelligence.
org/item?id=50 , 2014.
[9] Benja Fallenstein and Nate Soares. Problems of self-reference in self-improving space-
time embedded intelligence. Number 8598 in Lecture Notes in Articial Intelligence,
pages 21{32. Springer, 2014.
[10] Kurt G odel. Uber formal unentscheidbare s atze der Principia Mathematica und ver-
wandter systeme I. Monatshefte f ur Mathematik , 38(1):173{198, 1931.
[11] Patrick LaVictoire, Benja Fallenstein, Eliezer Yudkowsky, Mihaly Barasz, Paul Chris-
tiano, and Marcello Herresho. Program equilibrium in the prisoner's dilemma via
L ob's Theorem. In Multiagent Interaction without Prior Coordination: Papers from the
AAAI-14 Workshop . AAAI Publications, 2014.
[12] M. H. L ob. Solution of a Problem of Leon Henkin. The Journal of Symbolic Logic ,
20(2):pp. 115{118, 1955.
[13] Nate Soares and Benja Fallenstein. Toward idealized decision theory. http://
intelligence.org/files/TowardIdealizedDecisionTheory.pdf , 2014.
[14] Alfred Tarski. The concept of truth in formalized languages [Der wahrheitsbegri in den
formalisierten sprachen]. In J. H. Woodger, editor, Logic, Semantics, Metamathematics ,
pages 152{278. Hackett, 2 edition, 1983.
26
[15] Ken Thompson. Re
ections on Trusting Trust. Communications of the ACM ,
27(8):761{763, 1984.
[16] Eliezer Yudkowsky. A cartoon guide to L ob's theorem. http://www.yudkowsky.net/
assets/pdf/LobsTheorem.pdf .
[17] Eliezer Yudkowsky. Timeless decision theory. http://intelligence.org/files/TDT.
pdf, 2010.
[18] Eliezer Yudkowsky and Marcello Herresho. Tiling agents for self-modifying AI, and
the L obian obstacle. http://intelligence.org/files/TilingAgents.pdf , 2013.
27