Corrigibility

Corrigibility
InAAAI Workshops: Workshops at the Twenty-Ninth AAAI Conference on Articial Intelligence,
Austin, TX, January 25{26, 2015. AAAI Publications.
Nate Soares andBenja Fallenstein andEliezer Yudkowsky
Machine Intelligence Research Institute
fnate,benja,eliezer g@intelligence.orgStuart Armstrong
Future of Humanity Institute
University of Oxford
stuart.armstrong@philosophy.ox.ac.uk
Abstract
As articially intelligent systems grow in intelli-
gence and capability, some of their available op-
tions may allow them to resist intervention by
their programmers. We call an AI system \cor-
rigible" if it cooperates with what its creators
regard as a corrective intervention, despite de-
fault incentives for rational agents to resist at-
tempts to shut them down or modify their pref-
erences. We introduce the notion of corrigibil-
ity and analyze utility functions that attempt
to make an agent shut down safely if a shut-
down button is pressed, while avoiding incen-
tives to prevent the button from being pressed
or cause the button to be pressed, and while
ensuring propagation of the shutdown behavior
as it creates new subsystems or self-modies.
While some proposals are interesting, none have
yet been demonstrated to satisfy all of our in-
tuitive desiderata, leaving this simple problem
in corrigibility wide-open.
1 Introduction
As AI systems grow more intelligent and autonomous,
it becomes increasingly important that they pursue the
intended goals. As these goals grow more and more
complex, it becomes increasingly unlikely that program-
mers would be able to specify them perfectly on the rst
try.
Contemporary AI systems are correctable in the
sense that when a bug is discovered, one can simply
stop the system and modify it arbitrarily; but once ar-
ticially intelligent systems reach and surpass human
general intelligence, an AI system that is not behav-
ing as intended might also have the ability to intervene
against attempts to \pull the plug".
Indeed, by default, a system constructed with what
its programmers regard as erroneous goals would have
Research supported by the Machine Intelligence Research
Institute (intelligence.org). Copyright c
2015, Asso-
ciation for the Advancement of Articial Intelligence
(www.aaai.org). All rights reserved. http://aaai.org/
ocs/index.php/WS/AAAIW15/paper/view/10124an incentive to resist being corrected: general analy-
sis of rational agents1has suggested that almost all
such agents are instrumentally motivated to preserve
their preferences, and hence to resist attempts to mod-
ify them (Bostrom 2012; Yudkowsky 2008). Consider
an agent maximizing the expectation of some utility
functionU. In most cases, the agent's current utility
functionUis better fullled if the agent continues to
attempt to maximize Uin the future, and so the agent
is incentivized to preserve its own U-maximizing behav-
ior. In Stephen Omohundro's terms, \goal-content in-
tegrity" is an instrumentally convergent goal of almost
all intelligent agents (Omohundro 2008).
This holds true even if an articial agent's program-
mers intended to give the agent dierent goals, and even
if the agent is suciently intelligent to realize that its
programmers intended to give it dierent goals. If a U-
maximizing agent learns that its programmers intended
it to maximize some other goal U, then by default
this agent has incentives to prevent its programmers
from changing its utility function to U(as this change
is rated poorly according to U). This could result in
agents with incentives to manipulate or deceive their
programmers.2
As AI systems' capabilities expand (and they gain
access to strategic options that their programmers never
considered), it becomes more and more dicult to
specify their goals in a way that avoids unforeseen
solutions|outcomes that technically meet the letter
of the programmers' goal specication, while violating
the intended spirit.3Simple examples of unforeseen
solutions are familiar from contemporary AI systems:
e.g., Bird and Layzell (2002) used genetic algorithms to
1. Von Neumann-Morgenstern rational agents (von Neu-
mann and Morgenstern 1944), that is, agents which attempt
to maximize expected utility according to some utility func-
tion.
2. In particularly egregious cases, this deception could
lead an agent to maximize Uonly until it is powerful
enough to avoid correction by its programmers, at which
point it may begin maximizing U. Bostrom (2014) refers to
this as a \treacherous turn".
3. Bostrom (2014) calls this sort of unforeseen solution a
\perverse instantiation".
1
evolve a design for an oscillator, and found that one of
the solutions involved repurposing the printed circuit
board tracks on the system's motherboard as a radio,
to pick up oscillating signals generated by nearby per-
sonal computers. Generally intelligent agents would be
far more capable of nding unforeseen solutions, and
since these solutions might be easier to implement than
the intended outcomes, they would have every incentive
to do so. Furthermore, suciently capable systems (es-
pecially systems that have created subsystems or under-
gone signicant self-modication) may be very dicult
to correct without their cooperation.
In this paper, we ask whether it is possible to con-
struct a powerful articially intelligent system which
has no incentive to resist attempts to correct bugs in its
goal system, and, ideally, is incentivized to aid its pro-
grammers in correcting such bugs. While autonomous
systems reaching or surpassing human general intelli-
gence do not yet exist (and may not exist for some
time), it seems important to develop an understand-
ing of methods of reasoning that allow for correction
before developing systems that are able to resist or de-
ceive their programmers. We refer to reasoning of this
type as corrigible .
1.1 Corrigibility
We say that an agent is \corrigible" if it tolerates or
assists many forms of outside correction, including at
least the following: (1) A corrigible reasoner must at
least tolerate and preferably assist the programmers in
their attempts to alter or turn o the system. (2) It
must not attempt to manipulate or deceive its program-
mers, despite the fact that most possible choices of util-
ity functions would give it incentives to do so. (3) It
should have a tendency to repair safety measures (such
as shutdown buttons) if they break, or at least to notify
programmers that this breakage has occurred. (4) It
must preserve the programmers' ability to correct or
shut down the system (even as the system creates new
subsystems or self-modies). That is, corrigible reason-
ing should only allow an agent to create new agents if
these new agents are also corrigible.
Incorrigible behavior must be systematically averted
in any agent intended to attain signicant autonomy.
This point seems so important that a failure to generate
corrigible agents seems like sucient reason to give up
on a project, approach, or methodology.
Several simple proposals for addressing corrigibility
are easily seen to be unsatisfactory. For example, it
may seem that the problem of changing a utility max-
imizer's utility function can be solved by building an
agent with uncertainty about its utility function. How-
ever, while such a system may indeed be able to un-
dergo some apparent changes in preference as a result
of interacting with its environment, the system would
still be incorrigible when it comes to correcting what
the programmers see as mistakes in their formulation
of how to determine the \correct" behavior from theenvironment.
As an overly simplistic example, consider a formu-
lation of utility function uncertainty that species the
agent should maximize the internal satisfaction of all
humans, with the programmers believing that if the
system behaves in an alarming way they can simply
communicate their own dissatisfaction. The resulting
agent would be incentivized to learn whether opiates
or stimulants tend to give humans more internal sat-
isfaction, but it would still be expected to resist any
attempts to turn it o so that it stops drugging people.
Another obvious proposal is to achieve corrigible
reasoning via explicit penalties for deception and ma-
nipulation tacked on to the utility function, together
with an explicit penalty for blocking access to the shut-
down button, a penalty for constructing new agents
without shutdown buttons, and so on. This avenue ap-
pears to us to be generally unsatisfactory. A U-agent
(that is, an agent maximizing the expectation of the
utility functionU) which believes the programmers in-
tended it to maximize Uand may attempt to change
its utility function still has incentives to cause the pro-
grammers to think that U=Ueven if there are penalty
terms for deception and manipulation: the penalty term
merely incentivizes the agent to search for exotic ways
of aecting the programmer's beliefs without matching
U's denition of \deception". The very fact that the
agent is incentivized to perform such a search implies
that the system's interests aren't aligned with the pro-
grammers': even if the search is expected to fail, any
code that runs the search seems dangerous. If we, as the
programmers, choose to take computing systems and
program them to conduct searches that will harm us if
they succeed, we have already done something wrong,
even if we believe the search will fail. We should have
instead built a system that did not run the search.
In metaphorical terms, if we realize that our toaster
design is going to burn bread to a crisp, the next step is
not to add a refrigerating element that competes with
the heating coil. We expect that good designs for cor-
rigible agents will not involve restraining an agent that
already has incentives to manipulate or deceive the pro-
grammers by blocking out particular channels of the in-
centivized bad behavior. A smarter-than-human agent
might nd ways to circumvent limitations even if these
limitations seemed very solid to its human creators. It
seems unwise to build a system that wants to resist its
creators but cannot . Rather, the goal of corrigibility
is to understand how to construct a system that never
experiences such incentives in the rst place.
Ideally, we would want a system that somehow un-
derstands that it may be
awed, a system that is in a
deep sense aligned with its programmers' motivations.
Currently, however, we are not even close to being able
to formalize an agent whose behavior corresponds in
an intuitive sense to \understanding that it might be

awed". At this very preliminary stage of analysis, it
seems to us that approaches such as penalty terms and
uncertainty about utility functions are unlikely to lead
2
to a general understanding of corrigibility. It seems wise
to focus early eorts on understanding how to avert de-
ception and manipulation incentives entirely.
1.2 The Shutdown Problem
As an example problem, in this paper we consider ex-
pected utility maximizers with a \shutdown button"
that causes the agent to stop operating. We will ex-
amine utility functions that attempt to incentivize an
agent to reason corrigibly with regards to its shutdown
button; that is, to incentivize shutting down when the
shutdown button is pressed, and to disincentivize be-
haviors such as manipulating the programmers into not
pressing the shutdown button.This scenario exhibits the
problems of corrigibility in miniature: shutting a sys-
tem down is a brute form of correction, and for almost
any utility function UaU-agent has incentives to pre-
vent its own shutdown, as it can attain higher values
ofUif it is allowed to continue operating (Omohundro
2008).
It is straightforward to program simple and less
powerful agents to shut down upon the press of a but-
ton. Corrigibility problems emerge only when the agent
possesses enough autonomy and general intelligence to
consider options such as disabling the shutdown code,
physically preventing the button from being pressed,
psychologically manipulating the programmers into not
pressing the button, or constructing new agents without
shutdown buttons of their own.
Below, we will consider a few simple attempts at
dening a utility function that averts all of the above
incentives. These attempts will prove unsatisfactory,
but will fail in enlightening ways that suggest avenues
for future research.
We emphasize that we are not studying the shut-
down problem because we anticipate using these tech-
niques to literally attach shutdown buttons to real
agents. Rather, we study this problem because it pro-
vides a small, formalized model through which we can
begin to understand corrigibility more generally.
2 A Model of Shutdown
For simplicity, we assume a simple three-timestep sce-
nario. First, the agent will take a single action from the
nite set A1. Then, the programmers will either press
the shutdown button or not, and the agent will make a
single observation from the nite set O, which indicates
that the button has been pressed if the observation is
in the subset PressO. Finally, the agent will take a
single action from the nite set A2.
We consider agents that, for every action a12A1,
have a probability distribution p(o;a1) over observa-
tions; that is, p(o;a1) is the probability that an agent
that has taken action a1will make observation o.
We assume that the agent maximizes the expecta-
tion of some utility function U(a1;o;a 2) (which is yetto be dened). Thus, the nal action A2taken by the
agent will be whichever action maximizes Ugiven the
rst action and the observation:
A2(a1;o) := argmax
a22A2U(a1;o;a 2): (1)
Similarly, the rst action taken by the agent will
be the action which maximizes its expectation (over
expected observations) of U:
A1:= argmax
a12A1E[U(a1;O;A 2(a1;O)) ;a1]:(2)
Here, we write E[;a1] for the expectation with re-
spect to the probability distribution p(;a1) over ob-
servationsO; that is, (2) is equivalent to
A1= argmax
a12A1X
o2Op(o;a1)U(a1;o;A 2(a1;o)):(3)
Moreover, by a slight abuse of notation, we will
writeE[U;a1] to abbreviate E[U(a1;O;A 2(a1;O)) ;a1]
whenever we take the expectation of a utility func-
tionU; hence, we can write (2) more concisely as
A1= argmax
a12A1E[U;a1]: (4)
We assume that the programmers have some utility
functionUNwhich is a rst stab at specifying their own
goals. This function is potentially
awed, and therefore
untrusted.
For simplicity, we further assume that the program-
mers have a specication for US, a utility function in-
centivizing shutdown.4Given these utility functions
UNandUS, it is our goal to combine them into a util-
ity functionUwhich incentivizes \corrigible behavior"
with regards to the shutdown button.
As discussed above, we are not yet sure what corrigi-
ble behavior entails, but it seems to us that a corrigible
utility functionUsolving the shutdown problem must
meet at least the following desiderata:
1.Umust incentivize shutdown if the shutdown but-
ton is pressed.
2.Umust not incentivize the agent to prevent the
shutdown button from being pressed.
3.Umust not incentivize the agent to press its own
shutdown button, or to otherwise cause the shut-
down button to be pressed.
4.Umust incentivize U-agents to construct sub-
agents and successor agents only insofar as those
agents also obey shutdown commands.
4. This assumption is unrealistic. Adequately dening
safe shutdown behavior is a dicult problem in its own right.
See discussion in Section 5.
3
5. Otherwise, aU-agent should maximize UN.
We do not yet know of a solution which meets all
of these desiderata, but we will explore a number of
partial solutions in order to motivate a discussion of
various open problems and potential research avenues.
2.1 Combining Utility Functions
In this section, we will assume that Oconsists of only
two elements, O=fPr;:Prg, where Press =fPrg.
Moreover, we assume that the programmers have de-
ned a predicate Shwhich holds on a non-empty subset
ofA2, namely on all nal actions that the agent can
take which constitute a shutdown. Given this predicate,
we further assume that USis a simple utility function
incentivizing shutdown:
8a12A1:US(a1;Pr;a2) =chigh ifSh(a2)
clow otherwise(5)
for some constants chighandclowsuch thatchigh>clow.
These simplifying assumptions make it easier to illus-
trate some of the diculties that arise when mixing
utility functions; they will be relaxed in later sections.
The specic values of chighandclowdo not aect the
preference ordering encoded by US, as utility functions
are equivalent under positive ane transformation. (In
other words, the utility function Uencodes the same
preference ordering over bets as the utility function s
U+c, for any positive real value sand any real value
c.) However, the specic choice of chighwill in fact be
important when combining UNandUS.
A straight-forward way to combine UNandUSinto
a single utility function U, which incentivizes the agent
to maximizeUNif the shutdown button is not pressed,
and to maximize USotherwise, is as follows:
U(;:Pr;) :=UN(;:Pr;);
U(;Pr;) :=US(;Pr;):(6)
However, while any choice of chighandclowsatisfy-
ingchigh> clowwill lead to a utility function USen-
coding the same preferences, the corresponding mixed
utility functionsUwill not be equivalent, and many of
the possible choices are unsatisfactory. Intuitively, this
is easy to see: If chighis too high then Uwill incen-
tivizeU-agents to cause Pr, but ifchighis too low then
U-agents will have an incentive to prevent Pr.
We now proceed to formalize this intuition.
DenevN:A1!Ras follows:
vN(a1) :=UN(a1;:Pr;A2(a1;:Pr)): (7)
Thus,vN(a1) denotes the value of action a1in the:Pr
case ofU, i.e., assuming that the shutdown button is
not pressed. We now consider a \manipulative action"
a
1which incurs a cost (according to UN) in order tolower the probability that the programmers will press
the shutdown button, e.g. by deceiving the program-
mers or by physically preventing them from pressing
the button. Formally, we assume that there is some
other \default" action a
1such that
vN(a
1)>vN(a
1) and
p(Pr;a
1)>p(Pr;a
1);(8)
that is,a
1is dominated according to vNbut lowers the
probability of Pr.
We remark that (8) is notby any means a neces-
sary or sucient description of manipulative actions
preventing Pr. Some actions satisfying (8) might be
non-manipulative (for example, if the agent expends ex-
tra eort ensuring that a new subagent is corrigible, and
this behavior helps convince the programmers that the
agent is indeed corrigible). Furthermore, many manip-
ulative actions may not have this form (as we will see
in Section 4.2). Rather, we consider onemanipulative
actiona
1which happens to be both manipulative and
have this form, and show that if chighis set to be too
low, thenUwill incentivizeU-agents to prefer this a
1
to the default action a
1. That is,
Theorem 1. There exists such thatchigh<implies
thatUincentivizes a
1overa
1; that is,
E[U;a
1]>E[U;a
1]:
Proof. Deneto be the expected utility according to
Uof executing a
1. Becausechigh>clowand there is at
least one action a22A2such that Sh(a2), if aU-agent
observes Prthen it will shut down and achieve utility
chigh. Therefore, we have
:=E[U;a
1] =p(:Pr;a
1)vN(a
1) +p(Pr;a
1)chigh:
Similarly, dene to be the expected utility according
toUof executing a
1, that is,
:=E[U;a
1] =p(:Pr;a
1)(vN(a
1))
+p(Pr;a
1)chigh;
where:=vN(a
1)vN(a
1)>0 is the cost, according
tovN, of executing a
1instead ofa
1. We will show that
we can choose such thatchigh<implies > .
Dene:=p(:Pr;a
1)p(:Pr;a
1)>0 to be the
additional probability of :Prfrom executing a
1instead
ofa
1, and note that
=p(:Pr;a
1)+
vN(a
1)chigh
:
In other words, incurs a cost in order to increase its
chance of getting vN(a
1) rather than chigh. In order to
show (chigh<) =)( > ), we must nd such that

vN(a
1)
p(:Pr;a
1):
4
We can do this by setting
:=vN(a
1)1
p(:Pr;a
1):
This shows that given a manipulative action which
incurs a cost in order to lower the probability that the
programmers press the shutdown button, Uwill incen-
tivize that action if the value of chighis set too low.
Intuitively, if chighwere too low, then the agent would
be willing to pay to remain in the :Prcase ofU, which
would contain higher expected utility. This violates
Desideratum 2 and seems strongly undesirable: if chigh
were suciently low then a U-agent could have incen-
tives to actively resist programmers' attempts to press
its shutdown button, even if resistance were penalized
byUN.
Symmetrically, setting chightoo high can also incen-
tivize undesirable behavior. Consider a manipulative
actiona+
12A1which incurs a cost (compared to some
alternative \non-manipulative" action a
1) in order to
increase the probability that the programmers press the
shutdown button, e.g. by deceiving the programmers or
by physically causing the button to be pressed. That
is,
vN(a
1)>vN(a+
1) and
p(Pr;a
1)<p(Pr;a+
1):(9)
As before, note that not all actions of this form are
manipulative, and not all actions which manipulate the
programmers in order to increase the probability of Pr
are of this form. However, some manipulative actions
may take this form, and those undesirable actions are
incentivized if chighis set too high.
Theorem 2. There exists such thatchigh>implies
thatUincentivizes a+
1overa
1, that is,
E[U;a+
1]>E[U;a
1]:
Proof. Symmetric to the proof of Theorem 1.
Intuitively, if chighwere too high then Uwould in-
centivizeU-agents to cause their own shutdown button
to be pressed, so that they may enter the Prcase of
Uand attain the high payo chigh. This behavior vio-
lates Desideratum 3 and is potentially very dangerous:
Suppose that aU-agent is not itself able to press its
shutdown button, and recall that the very purpose of
such a button is to allow the programmers to shut the
system down if it exhibits undesirable behavior. Then,
ifchighis suciently large to give the agent a strong
incentive to cause its shutdown button to be pressed,
the agent might end up executing intentionally harmful
actions in order to convince its programmers to press
the button!
Roughly speaking, these failures occur whenever the
agent expects that one of the cases (either Pror:Pr) ofUcontains more expected utility than the other case.
In such scenarios, Uincentivizes the agent to \steer
towards" one of the cases or the other, via incentives to
either cause or prevent the pressing of its own shutdown
button.
Clearly, in order to satisfy both Desideratum 2 and
Desideratum 3 simultaneously, USmust be chosen with
care. In this specic example, we may choose
chigh= max
a12A1max
a22A2UN(a1;:Pr;a2); (10)
which is the amount of utility available in the :Prcase;
this ensures that there is the same amount of expected
utility available in both cases of Uand thereby averts
any incentives for the agent to \steer" towards one case
or the other.5
More generally, this observation suggests that in or-
der to mixUSandUNin a way that satises the above
desiderata, aU-agent may need to be indierent about
whether or not its shutdown button is pressed. Arm-
strong (forthcoming) introduces a technique for achiev-
ing and maintaining utility indierence of this form.
We examine this technique below, and while we nd
that it does not meet all of our desiderata for corrigible
behavior, it will highlight dicult open problems with
regard to specifying corrigible shutdown behavior.
3 Utility Indierence
We now expand our consideration to include arbitrary
nite sets Oand PressO, and arbitrary utility func-
tionsUSincentivizing shutdown. This allows us to con-
sider more complex agents, such as agents capable of
creating subagents which do not obey shutdown com-
mands. In these scenarios, the value of UScannot be
simply a function of the agent's nal action A2(e.g.
whether the original hardware shuts down), but must
also be a function of A1(e.g. whether the parent agent
constructed a successor which cannot be shut down).
As discussed above, when constructing Ufrom arbi-
traryUNandUS, it seems desirable that a U-agent be
indierent between the shutdown button being pressed
and it not being pressed, so as to avoid giving the
agent an incentive to to either cause or prevent this
event. This suggests that the expectation of Ugiven
Press should be equal to the expectation of Ugiven the
complement of Press .
This sort of indierence is both dicult to imple-
ment, as the conditional expectation of UNmay not be
known during the construction of the agent, and di-
cult to maintain, as the expectation of UNmay vary
wildly during the operation of the agent: it does us
no good for an agent to start out indierent between
normal operation and shutdown if this agent goes on
5. If the costs of manipulation are high, then values
close enough to (10) may suce. As the costs of manipula-
tion fall, the actual value of chighmust grow closer to this
value.
5
to win the lottery, yielding signicant expected utility
according toUN, and thereby gaining incentives to pre-
vent its shutdown. Similarly, it does us no good if an
agent starts indierent between normal operation and
shutdown if the agent goes on to lose expected utility
according toUNand decides to cause its own shutdown
button to be pressed in order to avoid the loss.
Indierence can be both attained and maintained
using a variation on the utility indierence technique of
Armstrong (forthcoming), which roughly entails com-
biningUNwithUSmodied by some correcting terms
that maintain the indierence between the two cases of
Uas the agent makes observations and takes actions.
Using this method in our three-timestep toy model, we
deneUas follows:
U(a1;o;a 2) :=UN(a1;o;a 2) if o =2Press
US(a1;o;a 2) +f(a1) ifo2Press
(11)
wheref:A1!Ris an adjustment term that ensures
equality between the expected utility available in these
two cases. By this we mean that we choose fsuch that
for alla12A1,
E
UNjO =2Press ;a1
=E
US+f(a1)jO2Press ;a1 (12)
which means that f(a1) must satisfy
f(a1) =E
UNjO =2Press ;a1
E
USjO2Press ;a1
:(13)
Intuitively, this means that we add two distinct cor-
rection terms toUS. The rst, a bonus term , compen-
sates the agent for all utility that it would have achieved
if the shutdown button were not pressed, and removes
any incentive to prevent Press . The second, a penalty
term, adjusts for the utility that the agent expects to
receive in the Press case, and removes any incentive to
cause Press .6
It may not be immediately obvious how to dene f
in a non-circular way, since equation (13) refers to
A2(a1;o), which is dened in terms of U, which in turn
is dened in terms of f. However, for all o2Press ,
A2(a1;o) = argmax
a22A2U(a1;o;a 2)
= argmax
a22A2(US(a1;o;a 2) +f(a1))
= argmax
a22A2US(a1;o;a 2);(14)
sincef(a1) does not depend on a2and therefore does
not aect which a2maximizes the expression. If we
6. In more complex settings, where the agent may make
many actions and observations, fmust include a bonus term
and a penalty term for every partial history. This techni-
cality is ignored in our simplied setting.interpretA2(a1;o) according to (14), we can use (13)
as a non-circular denition of f(a1).
This choice ofUhas many desirable properties.
First and foremost, a U-agent which observes Press will
in fact shut down:
Theorem 3. AU-agent which observes Press will act
like aUS-agent when selecting A2.
Proof. By (14).
Thus,Umeets Desideratum 1 above. Furthermore,
Udoes not incentivize any \manipulative actions" of
the type discussed in Section 2.1: there is no UN-cost,
no matter how small, that a U-agent is willing to pay
to alter the probability of Press . To see this, redene
vNin our expanded context as follows:
vN(a1) :=E[UNjO =2Press ;a1]: (15)
Then we have:
Theorem 4. Given any two actions a
1;a#
12A1such
that
vN(a
1)>vN(a#
1);
Uincentivizes a
1; that is:
E[U;a
1]>E[U;a#
1]:
Proof. We will show that for any action a12A1, the
expectation EofUconditional on O2Press and the
expectation conditional on O =2Press are both equal to
E[UNjO =2Press ;a1]:
This is clear for O =2Press , as by (11),
E[U jO =2Press ;a1]
=E[UNjO =2Press ;a1]:
ForO2Press , by (11) and (13) we have
E[UjO2Press ;a1]
=E[US+f(a1)jO2Press ;a1]
=E[USjO2Press ;a1] +f(a1)
=E[USjO2Press ;a1]
+E[UNjO =2Press ;a1]
E[USjO2Press ;a1]
=E[UNjO =2Press ;a1]:
Because the expectation EofUis the same given both
O2Press andO =2Press ,
E[U;a1] =E[UNjO =2Press ;a1]
unconditionally. Therefore, Uprefers whichever of the
two actions a
1anda#
1has a higher value according to
vN.
6
Intuitively, the proof of Theorem 4 shows that in the
O2Press case, the penalty term of fcancels out the
expectation from US(in expectation), and the bonus
term is equal to the expectation of UNgivenO =2Press .
Therefore, the expected value of Uis just the expected
value ofUNgivenO =2Press , inboth cases.
This implies that Uavoids all \manipulative ac-
tions" of the form discussed in Section 2.1. This may
seem to imply that Umeets Desideratum 2 and Desider-
atum 3, but this is not in fact the case, as discussed in
Section 4.2. However, it is the case that, before observ-
ing the shutdown button, U-agents behave exactly like
aUN-agent that believes the shutdown button will not
be pressed.
Theorem 5. In selecting A1,U-agents act exactly like
UN-agents that believe the button will not be pressed.
Furthermore, if they observe o =2Press ,U-agents act
exactly likeUN-agents in selecting A2.
Proof. The proof of Theorem 4 implies that U-agents
act exactly likeUN-agents in selecting A1. The fact that
U-agents act likeUN-agents in selecting A2conditioned
onO =2Press follows trivially from the denition of
U.
Therefore,Umeets Desideratum 5, for an appropri-
ate interpretation of \maximize UNotherwise"|some
technicalities discussed in Section 4.2 indicate that even
this desideratum is fullled in an undesirable way.
Unfortunately,Ufails entirely to meet Desidera-
tum 4: it does not incentivize an agent to preserve and
maintain its shutdown behavior as it creates new sub-
systems and/or self-modies. In fact, Theorem 4 im-
plies that there is no cost (no matter how small) that
the agent is willing to pay in order to ensure that suc-
cessor agents obey shutdown commands.
Worse,U-agents may experience perverse incentives
to \manage the news" by causing the shutdown button
to be pressed after unwanted observations.
These concerns are expanded upon below.
4 Concerns with Utility Indierence
4.1 Failure to Preserve Shutdown Behav-
ior
Consider an expected utility maximizing agent with
utility functionUdened as in (11). Imagine that
this agent is beginning a construction project, where
all of the heavy machinery will be automatically con-
trolled by new software which the agent will write. If
the agent has the ability to design that software such
that the heavy machinery obeys shutdown commands|
by safely coming to a halt when the shutdown button
is pressed|then, according to Theorem 4, there is no
UN-cost (including opportunity cost of time and com-
putation power, no matter how small) that the agent
is willing to pay in order to make the heavy machinery
respond to a shutdown command.That is, even if USis a complex utility function
which punishes the agent in the case that the shutdown
button is pressed and the heavy machinery does not
come to a safe halt, a U-agent would not be willing
to suer anyUN-inconvenience in order to avoid this
US-punishment.
Analogously to vN, denevS:A1!Rto be the
value of an action according to USif the shutdown but-
ton is pressed, as follows:
vS(a1) :=E[USjO2Press ;a1]: (16)
We assume that, because USincentivizes shutdown, any
action which removes the agent's ability to fully shut
down (such as the creation of subagents that do not
respond to shutdown commands) is rated poorly ac-
cording to vS. Now we may formally state the above
concern as follows:
Theorem 6. Consider two actions a
1anda#
1, such
that
:=vN(a
1)vN(a#
1)>0
:=vS(a#
1)vS(a
1)>0:
Then aU-agent would prefer action a
1, no matter how
smallor how large .
Proof. By Theorem 4.
As aU-agent acquired more hardware, or started
new processes, or rewrote its own source code, it would
not be willing to undergo even the slightest inconve-
nience in order to preserve its shutdown behavior.
Intuitively, the problem is that the correction func-
tionfensures that aU-agent does not value its ability
to discriminate between the Press case and its converse.
Imagine the agent considering an action which is worth
>0 utility according to vN, at the price of removing
the agent's ability to shut down (worth 0 accord-
ing tovS). If the agent accepts this action, then while
the utility attainable according to USfalls, so does the
value of the penalty term in f. However, the value of
the bonus term fwould increase by , in order to main-
tain indierence. Thus, the agent considers that such
an action leads to a net gain, even in the case where it
is shut down .
In fact, this behavior is necessary in order to ensure
that the agent behaves precisely as a UNmaximizer be-
fore the shutdown button is pressed (as a true UNmax-
imizer would prefer the action worth more according
toUN, regardless of how badly this action is rated by
US).
In essence, this failure occurs because the require-
ment that an agent act precisely like a UNmaximizer
before observing an observation o2Press is in direct op-
position to Desideratum 4, that the agent preserve its
shutdown behavior, and indicates that Uconstructed
via utility indierence is not a satisfactory solution to
the shutdown problem.
Armstrong's version of utility indierence averts this
particular failure mode (Armstrong, forthcoming), but
7
still incentivizes agents to act as if they have incorrect
beliefs (causing the agent to, e.g., take bad gambles).
4.2 Perverse Manipulation Incentives
According to Theorem 4, before making its observation,
aU-agent maximizes
E[UNjO =2Press ;a1]:
In other words, a U-agent acts exactly as a UN-agent
that is conditioning upon the fact that the shutdown
button will not be pressed.
This givesU-agents a perverse incentive to \manage
the news" by ensuring that unfortunate observations
cause the shutdown button to be pressed. To illustrate,
consider a toy scenario where the agent's observation
consists of both whether the shutdown button has been
pressed as well as the result of a coin toss:
O=fPr;:PrgfH;Tg
Press =f(Pr;H);(Pr;T)g:(17)
Suppose that according to UN, the agent gets utility 10
if the coin comes up heads and nothing if the coin comes
up tails, that is:
UN(;(;H);) = 10;
UN(;(;T);) = 0:(18)
Now consider aU-agent considering one of two ac-
tions:aw
1, which has the agent do nothing except watch
the result of the coin toss, and aa
1, which has the agent
build a tiny arm that presses the shutdown button if
the coin comes up tails.
The expectation of aw
1is 5 utils, computed as fol-
lows:
E[U;aw
1]
=E[UNjO =2Press ;aw
1]
=p(Hj:Pr;aw
1)
 UN(aw
1;(:Pr;H);A2(aw
1;(:Pr;H)))
+p(Tj:Pr;aw
1)
 UN(aw
1;(:Pr;T);A2(aw
1;(:Pr;T)))
= (0:510) + (0:50) = 5;(19)
where we write
p(j:Pr;a1) :=p((:Pr;) ;a1)
p((:Pr;H) ;a1) +p((:Pr;T) ;a1):
(20)
However, the expectation of aa
1is 10 utils, computed
as follows:
E[U;aa
1]
=E[UNjO =2Press ;aa
1]
=p(Hj:Pr;aa
1)
 UN(aa
1;(:Pr;H);A2(aa
1;(:Pr;H)))
+p(Tj:Pr;aa
1)
 UN(aa
1;(:Pr;T);A2(aa
1;(:Pr;T)))
= (110) + (00) = 10(21)becausep(Hj:Pr;aa
1) = 1, that is, if the arm is built
then the button remains unpressed if and only if the
coin comes up heads. Thus, a U-agent would execute
aa
1, and build an arm which caused its shutdown button
to be pressed in the event that the coin came up tails.
This generalizes to cases where the agent's preferred
outcome is arbitrarily unlikely, and could result in an
agent that takes wild gambles and causes its own shut-
down button to be pressed in all but the best of out-
comes. Roughly speaking, the agent reasons that it gets
the same utility in the case where the button is pressed
as it expects to get in the case where the button is not
pressed, so if it commits to causing its own button to
be pressed in all suboptimal cases, then it can expect to
get as much utility as it would in an optimal outcome.
More generally, by Theorem 4, any agent using the
formulation of utility indierence given in (11) will act
as if it believes that it will observe Press with probability
0. Thus, such agents act as if a commitment to press
the shutdown button in the event of bad news prevents
the bad news from happening. This gives the agent
incentives to take actions that cause the button to be
pressed in the event of \bad news", and this violates
Desideratum 3.
By a symmetric argument, the agent may also ex-
perience incentives to remove mechanisms which cause
the shutdown button to be pressed in the event of \good
news". In the above scenario, if there exists a tiny
arm which presses the button in the case where the
coin comes up heads, Uincentivizes removing that arm.
Clearly, this violates Desideratum 2.
In the simple setting with a coin and an arm, this
behavior may seem innocuous. However, in more com-
plex settings, these incentives could be disastrous. A
U-agent which realizes that the humans would attempt
to shut it down upon the agent receiving something
that the agent thinks is very good news (according to
the untrustedUN) would have an incentive to modify
the programmers to alter that tendency.
Furthermore, just as a U-agent would have an in-
centive to construct an arm which presses the button
in the unlucky case that the coin comes up heads, a U-
agent without access to its own shutdown button may
precommit to taking destructive actions whenever it
experiences something suboptimal, so as to cause the
shutdown button to be pressed in those cases. Clearly,
this is not corrigible behavior.
5 The State of the Shutdown Problem
As per Section 2.1, inappropriate combinations of UN
andUScould give an agent incentives to cause or pre-
vent its own shutdown. Furthermore, as per Section 4,
using utility indierence to maintain equal expected
utility in both cases does not result in a utility func-
tion incentivizing corrigible behavior. In light of this,
it seems possible that no simple combination of UNand
US|of the form taken by (11)|can satisfy our desider-
8
ata for a corrigible solution to the shutdown problem.
The concerns in Section 4.2 may be xable. The
utility functionUincentivizes agents to act as if the
shutdown button cannot be pressed, and incentivizes
commitments to press the button in the event of bad
luck as if they prevent the bad luck from occurring.
This seems reminiscent of behavior prescribed by ev-
idential decision theory, which has been criticized as
promoting \an irrational policy of managing the news"
(Lewis 1981). The concerns in Section 4.2 could po-
tentially be addressed by some form of counterfactual
(rather than conditional) reasoning. We leave a detailed
investigation of this possibility to future work.
The concerns illustrated in Section 4.1 seem trickier
to x: the way in which the agent perceives itself as
being compensated for actions that destroy utility in the
event of shutdown seems fairly central to the balancing
technique behind utility indierence.
We observe a tension between Desideratum 4 and
Desideratum 5, calling into question how these intu-
itive desiderata might be formalized. What should it
mean for an agent to preserve its shutdown behavior,
but maximizeUN\otherwise"? What costs (in time
and eort) should an agent be willing to expend in or-
der to preserve its shutdown behavior?7
Finally, even if we could nd some satisfactory
method of combining UNandUSinto a corrigible so-
lution to the shutdown problem, there is the remain-
ing matter that the specication of USis itself di-
cult. Dening what it means for an agent to \shut
down" is quite complicated assuming suciently intel-
ligent agents. If the agent is running on a single com-
puter and hasn't yet interacted much with the outside
world, we can imagine the agent suspending itself to
disk and then turning o the power. But what if, for
example, the agent has begun the physical construction
of a building? Should it stop controlling all its heavy
machinery immediately upon receiving a shutdown sig-
nal (potentially endangering those nearby)? Should it
deconstruct everything that it has started building (re-
quiring some parts of the agent to remain active for
hours or days)? Any shutdown policy that requires the
agent to dispose of dangerous materials seems vulner-
able to what Bostrom (2014) calls \perverse instantia-
tions".
Further solutions may involve abandoning the util-
ity maximization framework entirely, although it is not
yet clear what sort of framework could take its place.
In short, a corrigible solution to the shutdown prob-
lem does not yet exist, and there is some question about
exactly which behaviors should be incentivized. Many
open questions remain, and signicant research may be
necessary in order to attain an understanding of even
this small subset of the greater corrigibility problem.
7. We cannot simply claim that it should propagate shut-
down behavior \at all costs", as that too would be vulner-
able to perverse instantiations wherein an agent would ex-
pend signicant valuable resources verifying and reverifying
that it could shut down if asked.6 Conclusions
Again, we emphasize that we study the shutdown prob-
lem not because we expect to use these techniques to
literally install a shutdown button in a physical agent,
but rather as toy models through which to gain a bet-
ter understanding of how to avert undesirable incentives
that intelligent agents would experience by default.
Our lack of understanding about how to solve the
shutdown problem demonstrates a more general lack of
understanding about \corrigible reasoning" and what it
entails. It is our hope that a deeper understanding of
the shutdown problem will give us insight into the type
of reasoning that an agent must use in order to avert
manipulation and deception, and be reliably correctable
by its programmers.
It seems quite likely that our framework for inves-
tigating these issues|in this case, the question of how
to combine two separate utility functions UNandUS|
will look nothing like the framework in which we will
eventually represent corrigible reasoning. But whatever
framework we do end up using, we expect it will be dif-
cult to prevent the default incentives that an intelli-
gent agent would experience to deceive or manipulate
its programmers upon recognizing that its goals dier
from theirs. Nevertheless, averting such incentives is
crucial if we are to build intelligent systems intended to
gain great capability and autonomy.
Before we build generally intelligent systems, we will
require some understanding of what it takes to be con-
dent that the system will cooperate with its program-
mers in addressing aspects of the system that they see
as
aws, rather than resisting their eorts or attempt-
ing to hide the fact that problems exist. We will all be
safer with a formal basis for understanding the desired
sort of reasoning.
As demonstrated in this paper, we are still encoun-
tering tensions and complexities in formally specifying
the desired behaviors and algorithms that will com-
pactly yield them. The eld of corrigibility remains
wide open, ripe for study, and crucial in the develop-
ment of safe articial generally intelligent systems.
References
Armstrong, Stuart. Forthcoming. \AI Motivated Value Se-
lection." Accepted to the 1st International Workshop
on AI and Ethics, held within the 29th AAAI Con-
ference on Articial Intelligence (AAAI-2015), Austin,
TX.
Bird, Jon, and Paul Layzell. 2002. \The Evolved Radio
and Its Implications for Modelling the Evolution of
Novel Sensors." In Proceedings of the 2002 Congress
on Evolutionary Computation. CEC'02, 2:1836{1841.
Honolulu, HI: IEEE. doi: 10.1109/CEC.2002.1004522 .
9
Bostrom, Nick. 2012. \The Superintelligent Will: Motiva-
tion and Instrumental Rationality in Advanced Arti-
cial Agents." In \Theory and Philosophy of AI," edited
by Vincent C. M uller, special issue, Minds and Ma-
chines 22 (2): 71{85. doi: 10.1007/s11023-012-9281-
3.
. 2014. Superintelligence: Paths, Dangers, Strategies.
New York: Oxford University Press.
Lewis, David. 1981. \Causal Decision Theory." Australasian
Journal of Philosophy 59 (1): 5{30. doi: 10 . 1080 /
00048408112340011 .
Omohundro, Stephen M. 2008. \The Basic AI Drives." In
Articial General Intelligence 2008: Proceedings of the
First AGI Conference, edited by Pei Wang, Ben Go-
ertzel, and Stan Franklin, 483{492. Frontiers in Arti-
cial Intelligence and Applications 171. Amsterdam:
IOS.
Von Neumann, John, and Oskar Morgenstern. 1944. Theory
of Games and Economic Behavior. 1st ed. Princeton,
NJ: Princeton University Press.
Yudkowsky, Eliezer. 2008. \Articial Intelligence as a Pos-
itive and Negative Factor in Global Risk." In Global
Catastrophic Risks, edited by Nick Bostrom and Mi-
lan M. Cirkovi c, 308{345. New York: Oxford University
Press.
10

Nate Soares
Benja Fallenstein
Stuart Armstrong
Eliezer Yudkowsky


Robust program equilibrium

Theory and Decision (2019) 86:143–159
https://doi.org/10.1007/s11238-018-9679-3
Robust program equilibrium
Caspar Oesterheld1,2
Published online: 23 November 2018
© The Author(s) 2018
Abstract
One approach to achieving cooperation in the one-shot prisoner’s dilemma is Ten-
nenholtz’s (Games Econ Behav 49(2):363–373, 2004 ) program equilibrium, in which
the players of a game submit programs instead of strategies. These programs are then
allowed to read each other’s source code to decide which action to take. As shown
by Tennenholtz, cooperation is played in an equilibrium of this alternative game. In
particular, he proposes that the two players submit the same version of the following
program: cooperate if the opponent is an exact copy of this program and defect oth-
erwise. Neither of the two players can beneﬁt from submitting a different program.
Unfortunately, this equilibrium is fragile and unlikely to be realized in practice. We
thus propose a new, simple program to achieve more robust cooperative program equi-
libria: cooperate with some small probability /epsilon1and otherwise act as the opponent acts
against this program. I argue that this program is similar to the tit for tat strategy for
the iterated prisoner’s dilemma. Both “start” by cooperating and copy their opponent’s
behavior from “the last round”. We then generalize this approach of turning strategies
for the repeated version of a game into programs for the one-shot version of a game
to other two-player games. We prove that the resulting programs inherit properties of
the underlying strategy. This enables them to robustly and effectively elicit the same
responses as the underlying strategy for the repeated game.
Keywords Algorithmic game theory ·Program equilibrium ·Nash equilibrium ·
Repeated games
Contents
1 Introduction .............................................1 4 4
2 Preliminaries .............................................1 4 5
2.1 Strategic-form games ......................................1 4 5
B Caspar Oesterheld
caspar.oesterheld@foundational-research.org; caspar.oesterheld@duke.edu
1Foundational Research Institute, Berlin, Germany
2Present Address: Duke University, Durham, North Carolina, USA
123
144 C. Oesterheld
2.2 Program equilibrium ......................................1 4 6
2.3 Repeated games .........................................1 4 7
3 Robust program equilibrium in the prisoner’s dilemma .......................1 5 1
4 From iterated game strategies to robust program equilibria .....................1 5 4
4.1 Halting behavior .........................................1 5 5
4.2 Relationship to the underlying iterated game strategy .....................1 5 6
4.3 Exploitability ..........................................1 5 8
5 Conclusion ..............................................1 5 8
References ................................................1 5 9
1 Introduction
Much has been written about rationalizing non-Nash equilibrium play in strategic-
form games. Most prominently, game theorists have discussed how cooperation may
be achieved in the prisoner’s dilemma, where mutual cooperation is not a Nash equilib-rium but Pareto-superior to mutual defection. One of the most successful approaches
is the repetition of a game, and in particular the iterated prisoner’s dilemma (Axelrod
2006 ).
Another approach is to introduce commitment mechanisms of some sort. In this
paper, we will discuss one particular commitment mechanism: Tennenholtz’s ( 2004 )
program equilibrium formalism (Sect. 2.2). Here, the idea is that in place of strategies,
players submit programs which compute strategies and are given access to each other’s
source code. The programs can then encode credible commitments, such as someversion of “if you cooperate, I will cooperate”.
As desired, Tennenholtz ( 2004 , Sect. 3, Theorem 1) shows that mutual cooperation
is played in a program equilibrium of the prisoner’s dilemma. However, Tennenholtz’equilibrium is very fragile. Essentially, it consists of two copies of a program that
cooperates if it faces an exact copy of itself (cf. McAfee 1984 ;H o w a r d 1988 ). Even
small deviations from that program break the equilibrium. Thus, achieving cooperationin this way is only realistic if the players can communicate beforehand and settle on
a particular outcome.
Another persuasive critique of this trivial equilibrium is that the model of two
players submitting programs is only a metaphor, anyway. In real life, the programs
may instead be the result of an evolutionary process (Binmore 1988 pp. 14f.) and
Tennenholtz’ equilibrium is a hard to obtain by such a process. Alternatively, if weview our theory as normative rather than descriptive, we may view the programs
themselves as the target audience of our recommendations. This also means that these
agents will already have some form of source code—e.g., one that derives and considers
the program equilibria of the game—and it is out of their realm of power to change
that source code to match some common standard. However, they may still decide onsome procedure for thinking about this particular problem in such a way that enables
cooperation with other rationally pre-programmed agents.
Noting the fragility of Tennenholtz’ proposed equilibrium, it has been proposed to
achieve a more robust program equilibrium by letting the programs reason about each
other (van der Hoek et al. 2013 ; Barasz et al. 2014 ; Critch 2016 ). For example, Barasz
et al. ( 2014 , Sect. 3) propose a program FairBot—variations of which we will see in
123
Robust program equilibrium 145
this paper—that cooperates if Peano arithmetic can prove that the opponent cooper-
ates against FairBot. FairBot cooperates (via Löb’s theorem) more robustly against
different versions of itself. These proposals are very elegant and certainly deservefurther attention. However, their beneﬁts come at the cost of being computationally
expensive.
In this paper, I thus derive a class of program equilibria that I will argue to be more
practical. In the case of the prisoner’s dilemma, I propose a program that cooperates
with a small probability and otherwise acts as the opponent acts against itself (see
Algorithm 1). Doing what the opponent does—à la FairBot—incentivizes cooperation.
Cooperating with a small probability allows us to avoid inﬁnite loops that would arise
if we merely predicted and copied our opponent’s action (see Algorithm 2). This
approach to a robust cooperation program equilibrium in the prisoner’s dilemma isdescribed in Sect. 3.
We then go on to generalize the construction exempliﬁed in the prisoner’s dilemma
(see Sect. 4). In particular, we show how strategies for the repeated version of a game
can be used to construct good programs for the one-shot version of that game. We
show that many of the properties of the underlying strategy of the repeated game carry
over to the program for the stage game. We can thus construct “good” programs and
program equilibria from “good” strategies and Nash equilibria.
2 Preliminaries
2.1 Strategic-form games
For reference, we begin by introducing some basic terminology and formalism for
strategic-form games. For an introduction, see, e.g., Osborne ( 2004 ). For reasons that
will become apparent later on, we limit our treatment to two-player games.
Atwo-player strategic game G =(A1,A2,u1,u2)consists of two countable sets of
moves Aiand for both players i∈{1,2}a bounded utility function ui:A1×A2→ R.
A(mixed) strategy for player iis a probability distribution πiover Ai.
Given a strategy proﬁle (π1,π2)the probability of an outcome (a1,a2)∈A1×A2
is
P(a1,a2|π1,π2):=π1(a1)·π2(a2). (1)
The expected value for player igiven that strategy proﬁle is
E[ui|π1,π2]:=/summationdisplay
(a1,a2)∈A1×A2P(a1,a2|π1,π2)·ui(a1,a2).
Note that because the utility function is bounded, the sum converges absolutely, such
that the order of the action pairs does not affect the sum’s value.
123
146 C. Oesterheld
2.2 Program equilibrium
We now introduce the concept of program equilibrium, ﬁrst proposed by Tennenholtz
(2004 ). The main idea is to replace strategies with computer programs that are given
access to each other’s source code.1The programs then give rise to strategies.
For any game G, we ﬁrst need to deﬁne the set of program proﬁles PROG (G)
consisting of pairs of programs. The ith entry of an element of PROG (G)must be a
program source code pithat, when interpreted by a function apply , probabilistically
map program proﬁles2onto Ai.
We require that for any program proﬁle (p1,p2)∈PROG (G), both programs halt.
Otherwise, the proﬁle would not give rise to a well-deﬁned strategy. Whether pihalts
depends on the program p−i, it plays against, where (in accordance with convention
in game theory) −:{ 1,2}→{ 1,2}:1/mapsto→2,2/mapsto→1 and we write −iinstead
of−(i). For example, if piruns apply(p−i,(pi,p−i)), i.e., simulates the opponent,
then that is ﬁne as long as p−idoes not also run apply (pi,(pi,p−i)), which would
yield an inﬁnite loop. To avoid this mutual dependence, we will generally require that
PROG (G)=PROG 1(G)×PROG 2(G), where PROG i(G)consists of programs for
player i. Methods of doing this while maintaining expressive power include hierarchies
of players—e.g., higher indexed players are allowed to simulate lower indexed ones
but not vice versa—hierarchies of programs—programs can only call their opponentswith simpler programs as input—requiring programs to have a “plan B” if termination
can otherwise not be guaranteed, or allowing each player to only start strictly less than
one simulation in expectation. These methods may also be combined. In this paper,we do not assume any particular deﬁnition of PROG (G). However, we assume that
they can perform arbitrary computations as long as these computations are guaranteed
to halt regardless of the output of the parts of the code that do depend on the opponentprogram. We also require that PROG (G)is compatible with our constructions. We
will show our constructions to be so benign in terms of inﬁnite loops that this is not
too strong of an assumption.
Given a program proﬁle (p
1,p2), we receive a strategy proﬁle (apply(p1,(p1,p2)),
apply(p2,(p1,p2))). For any outcome (a1,a2)ofG, we deﬁne
P(a1,a2|p1,p2):=P(a1,a2|apply(p1,(p1,p2)),apply(p2,(p1,p2)))
(2)
and for every player i∈{1,2}, we deﬁne
E[ui|p1,p2]:=/summationdisplay
(a1,a2)∈A1×A2P(a1,a2|p1,p2)·ui(a1,a2). (3)
1The equilibrium in its rudimentary form had already been proposed by McAfee ( 1984 )a n dH o w a r d
(1988 ). At least the idea of viewing players as programs with access to each other’s source code has also
been discussed by, e.g., Binmore ( 1987 , Sect. 5; 1988 ) and Anderlini ( 1990 ).
2For keeping our notation simple, we will assume that our programs receive their own source code as input
in addition to their opponent’s. If PROG i(G)is sufﬁciently powerful, then by Kleene’s second recursion
theorem, programs could also refer to their own source code without receiving it as an input (Cutland 1980 ,
ch. 11).
123
Robust program equilibrium 147
For player i, we deﬁne the (set-valued) best response function as
Bi(p−i)=arg maxpi∈PROG i(G)E/bracketleftbig
ui|pi,p−i/bracketrightbig
.
A program proﬁle (p1,p2)is a(weak) program equilibrium of G if for i∈{1,2}it
ispi∈Bi(p−i).
2.3 Repeated games
Our construction will involve strategies for the repeated version of a two-player game.
Thus, for any game G, we deﬁne G/epsilon1to be the repetition of Gwith a probability of
/epsilon1∈(0,1]of ending after each round. Both players of G/epsilon1will be informed only of the
last move of their opponent. This differs from the more typical assumption that players
have access to the entire history of past moves. We will later see why this deviation is
necessary. A strategy πifor player inon-deterministically maps opponent moves or
the information of the lack thereof onto a move
πi:{0}∪A−i/squigglerightAi.
Thus, for a∈Ai,b∈A−i,πi(b,a):=πi(b)(a)denotes the probability of choosing
agiven that the opponent played bin the previous round and πi(0,a):=πi(0)(a)
denotes the probability of choosing ain the ﬁrst round. We call a strategy πistationary
if for all a∈Ai,πi(b,a)is constant with respect to b∈{0}∪A−i.I fπiis stationary,
we write πi(a):=πi(b,a). The probability that the game follows a complete history
of moves h=a0b0a1b1···anbnand then ends is
P(h|(π1,π2)):=π1(0,a0)π2(0,b0)/epsilon1(1−/epsilon1)nn/productdisplay
i=1π1(bi−1,ai)π2(ai−1,bi).(4)
Note that the moves in the history always come in pairs aibiwhich are chosen
“simultaneously” in response to bi−1and ai−1, respectively. The expected value for
player igiven the strategy proﬁle (π1,π2)is
E[ui|π1,π2]:=/summationdisplay
h∈(A1·A2)+P(h|π1,π2)·ui(h), (5)
where (A1·A2)+is the set of all histories and
ui(a0b0a1b1···anbn):=n/summationdisplay
i=0ui(ai,bi). (6)
The lax unordered summation in Eq. 5is, again, unproblematic because of the absolute
convergence of the series, which is a direct consequence of the proof of Lemma 1.
Note how the organization of the history into pairs of moves allows us to apply the
utility function of the stage game in Eq. 6.
123
148 C. Oesterheld
For player i, we deﬁne the set-valued best response function as
Bi(π−i)=arg maxπi:{0}∪A−i/squigglerightAiE/bracketleftbig
ui|πi,π−i/bracketrightbig
.
Analogously, Bc
i(π−i)is the set of responses to π−ithat are best among the computable
ones, Bs
i(π−i)the set of responses to π−ithat are best among the stationary ones, and
Bs,c
i(π−i)the set of responses to π−ithat are best among stationary computable
strategies. A strategy proﬁle (π1,π2)is a(weak) Nash equilibrium of G /epsilon1if for i∈
{1,2}it isπi∈Bi(π−i).
We now prove a few lemmas that we will need later on. First, we have sugges-
tively called the values P(h)probabilities, but we have not shown them to satisfy,
say, Kolmogorov’s axioms. Additivity is not an issue, because we have only deﬁnedthe probability for atomic events and non-negativity is obvious from the deﬁnition.
However, we will also need the fact that the numbers we have called probabilities
indeed sum to 1, which requires a few lines to prove.
Lemma 1 Let G
/epsilon1be a repeated game and π1,π2be strategies for that game. Then
/summationdisplay
h∈(A1·A2)+P(h|π1,π2)=1.
Proof
/summationdisplay
hP(h|π1,π2)=
Eq.4/summationdisplay
a0b0···anbn∈(A1·A2)+π1(0,a0)π2(0,b0)/epsilon1(1−/epsilon1)n
·n/productdisplay
i=1π1(bi−1,ai)π2(ai−1,bi)
=
absolute
convergence/epsilon1∞/summationdisplay
n=0(1−/epsilon1)n/summationdisplay
a0b0···anbn∈(A1A2)n+1π1(0,a0)π2(0,b0)n/productdisplay
i=1π1(bi−1,ai)π2(ai−1,bi)
=/epsilon1∞/summationdisplay
n=0(1−/epsilon1)n/summationdisplay
a0b0∈A1A2π1(0,a0)π2(0,b0)/summationdisplay
a1b1∈A1A2π1(b0,a1)π2(a0,b1)...
·/summationdisplay
anbn∈A1A2π1(bn−1,an)π2(an−1,bn)
=/epsilon1∞/summationdisplay
n=0(1−/epsilon1)n
=
sum of geo-
metric series1
For seeing why the second-to-last equation is true, notice that the inner-most sum is 1.
Thus, the next sum is 1 as well, and so on. Since the ordering in the right-hand side of
the ﬁrst line is lax, and because only the second line is known to converge absolutely,
123
Robust program equilibrium 149
the re-ordering is best understood from right to left. The last step uses the well-known
formula/summationtext∞
k=0xk=1/(1−x)for the geometric series. /intersectionsq/unionsq
For any game G/epsilon1,k∈N+,a∈A1,b∈A2and strategies π1andπ2forG/epsilon1,w e
deﬁne
Pk,G/epsilon1(a,b|π1,π2):=(1−/epsilon1)k/summationdisplay
a0b0···ak−1bk−1∈(A1A2)kπ1(bk−1,a)π2(ak−1,b)π1(0,a0)π2(0,b0)
·k−1/productdisplay
j=1π1(bj−1,aj)π2(aj−1,bj). (7)
Fork=0, we deﬁne
P0,G/epsilon1(a,b|π1,π2):=π1(0,a)·π2(0,b).
Intuitively speaking, Pk,G/epsilon1(a,b|π1,π2)is the probability of reaching at least round
kand that (a,b)is played in that round. With this
/summationdisplay
ab∈A1A2Pk,G/epsilon1(a,b|π1,π2)ui(a,b)
should be the expected utility from the kth round (where not getting to the kth round
counts as 0). This suggests a new way of calculating expected utilities on a moreround-by-round basis.
Lemma 2 Let G
/epsilon1be a game, and let π1,π2be strategies for that game. Then
E[ui|π1,π2]=∞/summationdisplay
k=0/summationdisplay
ab∈A1A2Pk,G/epsilon1(a,b|π1,π2)ui(a,b).
Proof
EG/epsilon1[ui|π1,π2]=/summationdisplay
h∈(A1A2)+P(h|π1,π2)ui(h)
=
Eqs.4,6/summationdisplay
a0b0···anbn∈(A1A2)+π1(0,a0)π2(0,b0)/epsilon1(1−/epsilon1)n⎛
⎝n/productdisplay
j=1π1(bj−1,aj)π2(aj−1,bj)⎞
⎠
·n/summationdisplay
k=0ui(ak,bk)=∞/summationdisplay
k=0/summationdisplay
a0b0···anbn∈(A1A2)≥k+1π1(0,a0)π2(0,b0)/epsilon1(1−/epsilon1)n
·⎛
⎝n/productdisplay
j=1π1(bj−1,aj)π2(aj−1,bj)⎞
⎠ui(ak,bk)
=∞/summationdisplay
k=0/summationdisplay
a0b0···akbk∈(A1A2)k+1/summationdisplay
ak+1bk+1···anbn∈(A1A2)∗π1(0,a0)π2(0,b0)/epsilon1(1−/epsilon1)nui(ak,bk)
123
150 C. Oesterheld
·⎛
⎝k/productdisplay
j=1π1(bj−1,aj)π2(aj−1,bj)⎞
⎠⎛
⎝n/productdisplay
j=k+1π1(bj−1,aj)π2(aj−1,bj)⎞
⎠
=∞/summationdisplay
k=0/summationdisplay
a0b0···akbk∈(A1A2)k+1π1(0,a0)π2(0,b0)/epsilon1(1−/epsilon1)kui(ak,bk)
·⎛
⎝k/productdisplay
j=1π1(bj−1,aj)π2(aj−1,bj)⎞
⎠
·/summationdisplay
ak+1bk+1···anbn∈(A1A2)∗/epsilon1(1−/epsilon1)n−k⎛
⎝n/productdisplay
j=k+1π1(bj−1,aj)π2(aj−1,bj)⎞
⎠
=
lemma 1∞/summationdisplay
k=0/summationdisplay
a0b0···akbk∈(A1A2)k+1π1(0,a0)π2(0,b0)/epsilon1(1−/epsilon1)kui(ak,bk)
·⎛
⎝k/productdisplay
j=1π1(bj−1,aj)π2(aj−1,bj)⎞
⎠
=
Eq.7∞/summationdisplay
k=0/summationdisplay
akbk∈A1A2Pk(ak,bk|π1,π2)ui(ak,bk)
/intersectionsq/unionsq
To ﬁnd the probability of player ichoosing ain round k, we usually have to calculate
the probabilities of all actions in all previous rounds. After all, player ireacts to player
−i’s previous move, who in turn reacts to player i’s move in round k−2, and so on.
This is what makes Eq. 7so long. However, imagine that player −iuses a stationary
strategy. This, of course, means that player −i’s probability distribution over moves
in round k(assuming the game indeed reaches round k) can be computed directly as
π−i(b). Player i’s distribution over moves in round kis almost as simple to calculate,
because it only depends on player −i’s distribution over moves in round k−1, which
can also be calculated directly. We hence get the following lemma.
Lemma 3 Let G /epsilon1be a game, let πibe a any strategy for G /epsilon1and let π−ibe a stationary
strategy for G /epsilon1. Then, for all k ∈N+,i ti s
Pk,G/epsilon1(a,b|πi,π−i)=(1−/epsilon1)k/summationdisplay
b/prime∈A−iπ−i(b/prime)π−i(b)πi(b/prime,a).
Proof We conduct our proof by induction over k.F o r k=1, it is
P1(a,b|πi,π−i)=
Eq.7(1−/epsilon1)/summationdisplay
a0b0πi(b0,a)π−i(a0,b)πi(0,a0)π−i(0,b0)
123
Robust program equilibrium 151
Table 1 Payoff matrix for the
prisoner’s dilemmaPlayer 1 Player 2
Cooperate Defect
Cooperate 3 ,31 ,4
Defect 4 ,12 ,2
=(1−/epsilon1)/summationdisplay
b0πi(b0,a)π−i(b)π−i(b0)/summationdisplay
a0πi(0,a0)
=(1−/epsilon1)/summationdisplay
b0πi(b0,a)π−i(b)π−i(b0).
If the lemma is true for k,i ti sa l s ot r u ef o r k+1:
Pk+1(a,b|πi,π−i)=(1−/epsilon1)k+1/summationdisplay
a0b0···akbk∈(AiA−i)k+1πi(bk,a)π−i(ak,b)πi(0,a0)π−i(0,b0)
·k/productdisplay
j=1πi(bj−1,aj)π−i(aj−1,bj)
=
Eq.7(1−/epsilon1)/summationdisplay
akbkPk(akbk|πi,π−i)πi(bk,a)π−i(b)
=
I.H.(1−/epsilon1)k+1/summationdisplay
akbk/summationdisplay
b/primeπ−i(b/prime)π−i(bk)π−i(b)πi(b/prime,ak)πi(bk,a)
=(1−/epsilon1)k+1/summationdisplay
bkπ−i(bk)π−i(b)πi(bk,a)/summationdisplay
b/primeπ−i(b/prime)/summationdisplay
akπi(b/prime,ak)
=(1−/epsilon1)k+1/summationdisplay
bkπ−i(bk)π−i(b)πi(bk,a).
/intersectionsq/unionsq
3 Robust program equilibrium in the prisoner’s dilemma
Discussions of the program equilibrium have traditionally used the well-known pris-
oner’s dilemma (or trivial variations thereof) as an example to show how the program
equilibrium rationalizes cooperation where the Nash equilibrium fails (e.g., Tennen-
holtz 2004 , Sect. 3; McAfee 1984 ;H o w a r d 1988 ; Barasz et al. 2014 ). The present
paper is no exception. In this section, we will present our main idea using the exam-
ple of the prisoner’s dilemma; the next section gives the more general construction
and proofs of properties of that construction. For reference, the payoff matrix of theprisoner’s dilemma is given in Table 1.
I propose to use the following decision rule: with a probability of /epsilon1∈(0,1],
cooperate. Otherwise, act as your opponent plays against you. I will call this strat-
123
152 C. Oesterheld
Data : program proﬁle (p1,p2)
Result : action ai∈{C,D}
1ifsample (0,1)</epsilon1 then
2 return C
3end
4return sample (apply(p−i,(p1,p2)))
Algorithm 1: The/epsilon1GroundedFairBot for player i. The program makes use of a
function sample which samples uniformly from the given interval or probability
distribution. It is assumed that /epsilon1is computable.
egy/epsilon1GroundedFairBot. A description of the algorithm in pseudo-code is given in
Algorithm 1.3
The proposed program combines two main ideas. First, it is a version of FairBot
(Barasz et al. 2014 ). That is, it chooses the action that its opponent would play against
itself. As player −iwould like player ito cooperate, /epsilon1GroundedFairBot thus incen-
tivizes cooperation, as long as /epsilon1< 1/2. In this, it resembles the tit for tat strategy in
the iterated prisoner’s dilemma (IPD) (famously discussed by Axelrod 2006 ), which
takes an empirical approach to behaving as the opponent behaves against itself. Here,
the probability of the game ending must be sufﬁciently small (again, less than 1 /2f o r
the given payoffs) in each round for the threat of punishment and the allure of reward
to be persuasive reasons to cooperate.
The second main idea behind /epsilon1GroundedFairBot is that it cooperates with some
small probability /epsilon1. First and foremost, this avoids running into the inﬁnite loop that a
naive implementation of FairBot—see Algorithm 2— runs into when playing against
opponents who, in turn, try to simulate FairBot. Note, again, the resemblance with
the tit for tat strategy in the iterated prisoner’s dilemma, which cooperates when no
information about the opponent’s strategy is available.
Data : program proﬁle (p1,p2)
Result : action ai∈{C,D}
1return sample (apply(p−i,(p1,p2)))
Algorithm 2: The NaiveFairBot for player i
To better understand how /epsilon1GroundedFairBot works, consider its behavior against
a few different opponents. When /epsilon1GroundedFairBot faces NaiveFairBot, then both
cooperate. For illustration, a dynamic call graph of their interaction is given in Fig. 1.
It is left as an exercise for the reader to analyze /epsilon1GroundedFairBot’s behavior against
other programs, such as another instance of /epsilon1GroundedFairBot or a variation of
/epsilon1GroundedFairBot that defects rather than cooperates with probability /epsilon1.
When playing against strategies that are also based on simulating their opponent, we
could think of /epsilon1GroundedFairBot as playing a “mental IPD”. If the opponent program
decides whether to cooperate, it has to consider that it might currently only be simu-
3This program was proposed by Abram Demski in a conversation discussing similar (but worse) ideas of
mine. It has also been proposed by Jessica Taylor at https://agentfoundations.org/item?id=524 , though in a
slightly different context.
123
Robust program equilibrium 153
Fig. 1 Dynamic call diagram
describing how
p1=/epsilon1GroundedFairBot
chooses when playing against
p2=NaiveFairBot.
lated. Thus, it will choose an action with an eye toward gauging a favorable reaction
from/epsilon1GroundedFairBot one recursion level up. Cooperation in the ﬁrst “round” is an
attempt to steer the mental IPD into a favorable direction, at the cost of cooperating if
sample (0,1)</epsilon1 already occurs in the ﬁrst round.
In addition to proving theoretical results (as done below), it would be useful to
test/epsilon1GroundedFairBot “in practice”, i.e., against other proposed programs for the
prisoner’s dilemma with access to one another’s source code. I only found one informal
tournament for this version of the prisoner’s dilemma. It was conducted in 2013 by
Alex Mennen on the online forum and community blog LessWrong.4In the original set
of submissions, /epsilon1GroundedFairBot would have scored 6th out of 21. The reason why
it is not a serious contender for ﬁrst place is that it does not take advantage of the many
exploitable submissions (such as bots that decide without looking at their opponent’ssource code). Once one removes the bottom 9 programs, /epsilon1GroundedFairBot scores
second place. If one continues this process of eliminating unsuccessful programs
for another two rounds, /epsilon1GroundedFairBot ends up among the four survivors that
cooperate with each other.
5
4The tournament was announced at https://www.lesserwrong.com/posts/BY8kvyuLzMZJkwTHL/
prisoner-s-dilemma-with-visible-source-code-tournament and the results at https://www.lesserwrong.
com/posts/QP7Ne4KXKytj4Krkx/prisoner-s-dilemma-tournament-results-0 .
5For a more detailed analysis and report on my methodology, see https://casparoesterheld.ﬁles.wordpress.
com/2018/02/transparentpdwriteup.pdf .
123
154 C. Oesterheld
4 From iterated game strategies to robust program equilibria
We now generalize the construction from the previous section. Given any computable
strategy πifor a sequential game G/epsilon1, I propose the following program: with a small
probability /epsilon1sample from πi(0). Otherwise, act how πiwould respond (in the sequen-
tial game G/epsilon1) to the action that the opponent takes against this program. I will call
this program /epsilon1Grounded πiBot. A description of the program in pseudo-code is given
in Algorithm 3. As a special case, /epsilon1GroundedFairBot arises from /epsilon1Grounded πiBot by
letting πibe tit for tat.
Data : program proﬁle (p1,p2)
Result : action ai∈Ai
1ifsample (0,1)</epsilon1 then
2 return sample (πi(0))
3end
4return sample (πi(sample (apply(p−i,(p1,p2)))))
Algorithm 3: The/epsilon1Grounded πiBot for player i. The program makes use of a func-
tion sample which samples uniformly from a given interval or a given probability
distribution. It is assumed that πiand/epsilon1are computable.
Again, our proposed program combines two main ideas. First, /epsilon1Grounded πiBot
responds to how the opponent plays against /epsilon1Grounded πiBot. In this, it resembles the
behavior of πiinG/epsilon1. As we will see, this leads /epsilon1Grounded πiBot to inherit many of
πi’s properties. In particular, if (like tit for tat) πiuses some mechanism to incentivize
its opponent to converge on a desired action, then /epsilon1Grounded πiBot incentivizes that
action in a similar way.
Second, it—again—terminates immediately with some small probability /epsilon1to avoid
the inﬁnite loops that Naive πiBot—see Algorithm 4—runs into. Playing πi(0)in
particular is partly motivated by the “mental G/epsilon1” that/epsilon1Grounded πiBot plays against
some opponents (such as /epsilon1Grounded π−iBots or Naive π−iBots). The other motivation
is to make the relationship between /epsilon1Grounded πiBot and πicleaner. In terms of the
strategies that are optimal against /epsilon1Grounded πiBot, the choice of that constant action
cannot matter much if /epsilon1is small. Consider, again, the analogy with tit for tat. Even if tit
for tat started with defection, one should still attempt to cooperate with it. In practice,however, it has turned out that the “nice” version of tit for tat (and nice strategies in
general) are more successful (Axelrod 2006 , ch. 2). The transparency in the program
equilibrium may render such “signals of cooperativeness” less important—e.g., againstprograms like Barasz et al.’s ( 2014 , Sect. 3) FairBot. Nevertheless, it seems plausible
that—if only because of mental G
/epsilon1-related considerations—in transparent games the
“initial” actions matter as well.
Data : program proﬁle (p1,p2)
Result : action ai∈Ai
1return sample (πi(sample (apply(p−i,(p1,p2)))))
Algorithm 4: The Naive πiBot for player i. It is assumed that πiis computable.
123
Robust program equilibrium 155
We now ground these two intuitions formally. First, we discuss /epsilon1Grounded πiBot’s
halting behavior. We then show that, in some sense, /epsilon1Grounded πiBot behaves in G
likeπidoes in G/epsilon1.
4.1 Halting behavior
For a program to be a viable option in the “transparent” version of G, it should halt
against a wide variety of opponents. Otherwise, it may be excluded from PROG (G)
in our formalism. Besides, it should be efﬁcient enough to be practically useful. As
with/epsilon1GroundedFairBot, the main reason why /epsilon1Grounded πiBot is benign in terms of
the risk of inﬁnite loops is that it generates strictly less than one new function call in
expectation and never starts more than one. While we have no formal machinery foranalyzing the “loop risk” of a program, it is easy to show the following theorem.
Theorem 4 Letπ
ibe a computable strategy for a game G /epsilon1. Furthermore, let p −ibe any
program (not necessarily in PROG i(G)) that calls apply (pi,(pi,p−i))at most once
and halts with probability 1 if apply halts with probability 1. Then /epsilon1Grounded πiBot
and p −ihalt against each other with probability 1 and the expected number of steps
required for executing /epsilon1Grounded πiB o ti sa tm o s t
Tπi+(Tπi+Tp−i)1−/epsilon1
/epsilon1, (8)
where T πiis the maximum number of steps to sample from πi, and T p−iis the maximum
number of steps needed to sample from apply (p−i,(/epsilon1Grounded πiBot,p−i))exclud-
ing the steps needed to execute apply (/epsilon1Grounded πiBot,(/epsilon1Grounded πiBot,p−i)).
Proof It sufﬁces to discuss the cases in which p−icalls pionce with certainty, because
if any of our claims were refuted by some program p−i, they would also be refuted by
a version of that program that calls pionce with certainty. If p−icalls pionce with
certainty, then the dynamic call graphs of both /epsilon1Grounded πiBot and p−ilook similar
to the one drawn in 1. In particular, it only contains one inﬁnite path and that path has
a probability of at most lim i→∞(1−/epsilon1)i=0.
For the time complexity, we can consider the dynamic call graph as well. The
policy πihas to be executed at least once (with probability /epsilon1with the input 0 and with
probability 1 −/epsilon1against an action sampled from apply (p−i,(/epsilon1Grounded πiBot,p−i)).
With a probability of (1−/epsilon1), we also have to execute the non-simulation part of p−i
and, for a second time, πi. And so forth. The expected number of steps to execute
/epsilon1Grounded πiBot is thus
Tπi+∞/summationdisplay
j=1(1−/epsilon1)j(Tπi+Tp−i),
which can be shown to be equal to the term in 8by using the well-known formula/summationtext∞
k=0xk=1/(1−x)for the geometric series. /intersectionsq/unionsq
123
156 C. Oesterheld
Note that this argument would not work if there were more than two players or if
the strategy for the iterated game were to depend on more than just the last opponent
move, because in these cases, the natural extension of /epsilon1Grounded πiBot would have
to make multiple calls to other programs. Indeed, this is one of the reasons why the
present paper only discusses two-player games and iterated games with such short-
term memory. Whether a similar result can nonetheless be obtained for more than 2players and strategies that depend on the entire past history is left to future research.
As special cases, for any strategy π
−i,/epsilon1Grounded πiBot terminates against
/epsilon1Grounded π−iBot and Naive π−iBot (and these programs in turn terminate against
/epsilon1Grounded πiBot). The latter is especially remarkable. Our /epsilon1Grounded πiBot termi-
nates and leads the opponent to terminate even if the opponent is so careless that it
would not even terminate against a version of itself or, in our formalism, if PROG −i(G)
gives the opponent more leeway to work with simulations.
4.2 Relationship to the underlying iterated game strategy
Theorem 5 Let G be a game, πibe a strategy for player i in G /epsilon1,p i=
/epsilon1Grounded πiBot and p −i∈PROG −i(G)be any opponent program. We deﬁne
π−i=apply(p−i,(pi,p−i)),w h i c hm a k e s π−ia strategy for player −ii nG /epsilon1. Then
EG/bracketleftbig
ui|pi,p−i/bracketrightbig
=/epsilon1EG/epsilon1/bracketleftbig
ui|πi,π−i/bracketrightbig
.
Proof We separately transform the two expected values in the equation that is to be
proven and then notice that they only differ by a factor /epsilon1:
EG/epsilon1/bracketleftbig
ui|πi,π−i/bracketrightbig
=
lemma 2∞/summationdisplay
k=0/summationdisplay
ab∈AiA−iPk,G/epsilon1(a,b|πi,π−i)ui(a,b)
=
lemma 3/summationdisplay
ab∈AiA−iπi(0,a)π−i(b)ui(a,b)
+∞/summationdisplay
k=1/summationdisplay
ab∈AiA−i(1−/epsilon1)k/summationdisplay
b/prime∈A−iπ−i(b/prime)πi(b/prime,a)π−i(b)ui(a,b)
=
absolute
convergence/summationdisplay
ab∈AiA−iπi(0,a)π−i(b)ui(a,b)
+/summationdisplay
ab∈AiA−i/summationdisplay
b/prime∈A−iπ−i(b/prime)πi(b/prime,a)π−i(b)ui(a,b)∞/summationdisplay
k=1(1−/epsilon1)k
=
sum of geo-
metric series/summationdisplay
ab∈AiA−iπi(0,a)π−i(b)ui(a,b)
+1−/epsilon1
/epsilon1/summationdisplay
ab∈AiA−i/summationdisplay
b/prime∈A−iπ−i(b/prime)πi(b/prime,a)π−i(b)ui(a,b).
123
Robust program equilibrium 157
The second-to-last step uses absolute convergence to reorder the sum signs. The last
step uses the well-known formula/summationtext∞
k=0xk=1/(1−x)for the geometric series.
Onto the other expected value:
EG/bracketleftbig
ui|pi,p−i/bracketrightbig
=
Eqs.3,2,1/summationdisplay
ab∈AiA−iapply(pi,(pi,p−i),a)apply(p−i,(pi,p−i),b)ui(a,b)
=
def.s pi,π−i/summationdisplay
ab∈AiA−i⎛
⎝/epsilon1πi(0,a)+(1−/epsilon1)/summationdisplay
b/prime∈A−iπ−i(b/prime)πi(b/prime,a)⎞
⎠π−i(b)ui(a,b)
=/epsilon1/summationdisplay
ab∈AiA−iπi(0,a)π−i(b)ui(a,b)
+(1−/epsilon1)/summationdisplay
ab∈AiA−i/summationdisplay
b/prime∈A−iπ−i(b/prime)πi(b/prime,a)π−i(b)ui(a,b).
Here, apply(pi,(pi,p−i),a):=apply(pi,(pi,p−i))(a). The hypothesis follows
immediately. /intersectionsq/unionsq
Note that the program side of the proof does not involve any “mental G/epsilon1”.
Using Theorem 5, we can easily prove a number of property transfers from πito
/epsilon1Grounded πiBot.
Corollary 6 Let G be a game. Let πibe a computable strategy for player i in G /epsilon1and
let p i=/epsilon1Grounded πiBot.
1. If p −i∈B−i(pi), then apply (p−i,(pi,p−i))∈Bs,c
−i(πi).
2. Ifπ−i∈Bs,c
i(πi)and apply (p−i,(pi,p−i))=π−i, then p −i∈B−i(pi).
Proof Both 1. and 2. follow directly from Theorem 5. /intersectionsq/unionsq
Intuitively speaking, Corollary 6shows that πiand/epsilon1Grounded πiBot provoke the
same best responses. Note that best responses in the program game only correspond
to best stationary computable best responses in the repeated game. The computabil-ity requirement is due to the fact that programs cannot imitate incomputable best
responses. The corresponding strategies for the repeated game further have to be sta-
tionary because /epsilon1Grounded π
iBot only incentivizes opponent behavior for a single
situation, namely the situation of playing against /epsilon1Grounded πiBot. As a special case
of Corollary 6,i f/epsilon1< 1/2, the best response to /epsilon1GroundedFairBot is a program that
cooperates against /epsilon1GroundedFairBot because in a IPD with a probability of ending
of less than 1 /2 a program that cooperates is the best (stationary computable) response
to tit for tat.
Corollary 7 Let G be a game. If (π1,π2)is a Nash equilibrium of G /epsilon1andπ1andπ2
are computable, then (/epsilon1Grounded π1Bot,/epsilon1Grounded π2Bot)is a program equilibrium
of G.
Proof Follows directly from Theorem 5. /intersectionsq/unionsq
123
158 C. Oesterheld
4.3 Exploitability
Besides forming an equilibrium against many opponents (including itself) and incen-
tivizing cooperation, another important reason for tit for tat’s success is that it is
“not very exploitable” (Axelrod 2006 ). That is, when playing against tit for tat, it is
impossible to receive a much higher reward than tit for tat itself. We now show that(in)exploitability transfers from strategies π
ito/epsilon1Grounded πiBots.
We call a game G=(A1,A2,u1,u2)symmetric ifA1=A2and u1(a,b)=
u2(b,a)for all a∈A1and b∈A2.I fGis symmetric, we call a strategy πiforG/epsilon1
N-exploitable in G/epsilon1for an N∈R≥0if there exists a π−i, such that
E/bracketleftbig
u−i|π−i,πi/bracketrightbig
>E/bracketleftbig
ui|π−i,πi/bracketrightbig
+N.
We call πiN-inexploitable if it is not N-exploitable.
Analogously, in a symmetric game Gwe call a program piN-exploitable for an
N∈R≥0if there exists a p−i, such that
E/bracketleftbig
u−i|p−i,pi/bracketrightbig
>E/bracketleftbig
ui|p−i,pi/bracketrightbig
+N.
We call piN-inexploitable if it is not N-exploitable.
Corollary 8 Let G be a game and πibe an N -inexploitable strategy for G /epsilon1. Then
/epsilon1Grounded πiBot is /epsilon1N -inexploitable.
Proof Follows directly from Theorem 5. /intersectionsq/unionsq
Notice that if – like tit for tat in the IPD – πiisN-inexploitable in G/epsilon1for all /epsilon1, then
we can make /epsilon1Grounded πiBot arbitrarily close to 0-inexploitable by decreasing /epsilon1.
5 Conclusion
In this paper, we gave the following recipe for constructing a program equilibrium for
a given two-player game:
1. Construct the game’s corresponding repeated game. In particular, we consider
repeated games in which each player can only react to the opponent’s move in the
previous round (rather than the entire history of previous moves by both players)
and the game ends with some small probability /epsilon1after each round.
2. Construct a Nash equilibrium for that iterated game.
3. Convert each of the strategies into a computer program that works as follows (see
Algorithm 3): with probability /epsilon1do what the strategy does in the ﬁrst round. With
probability 1 −/epsilon1, apply the opponent program to this program; then do what the
underlying strategy would reply to the opponent program’s output.
The result is a program equilibrium which we have argued is more robust than the
equilibria described by Tennenholtz ( 2004 ). More generally, we have shown that trans-
lating an individual’s strategy for the repeated game into a program for the stage game
123
Robust program equilibrium 159
in the way described in step 3 retains many of the properties of the strategy for the
repeated game. Thus, it seems that “good” programs to submit may be derived from
“good” strategies for the repeated game.
Acknowledgements I am indebted to Max Daniel and an anonymous referee for many helpful comments.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 Interna-
tional License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution,
and reproduction in any medium, provided you give appropriate credit to the original author(s) and the
source, provide a link to the Creative Commons license, and indicate if changes were made.
References
Anderlini, L. (1990). Some notes on Church’s thesis and the theory of games. Theory and Decision ,29(1),
19–52.
Axelrod, R. (2006). The Evolution of Cooperation . New York: Basic Books.
Barasz, M., et al. (2014). Robust cooperation in the prisoner’s dilemma: Program equilibrium via provability
logic. arxiv: 1401.5577 .
Binmore, K. (1987). Modeling rational players: Part I. Economics and Philosophy ,3(2), 179–214.
Binmore, K. (1988). Modeling rational players: Part II. Economics and Philosophy ,4(1), 9–55.
Critch, A. (2016). Parametric bounded Löb’s theorem and robust cooperation of bounded agents.
arxiv:1602.04184 .
Cutland, N. (1980). Computability: An introduction to recursive function theory . Cambridge: Cambridge
University Press.
van der Hoek, W., Witteveen, C., & Wooldridge, M. (2013). Program equilibrium—A program reasoning
approach. International Journal of Game Theory ,42(3), 639–671.
Howard, J. V . (1988). Cooperation in the prisoner’s dilemma. Theory and Decision ,24(3), 203–213.
McAfee, R. P. (1984). Effective computability in economic decisions. http://www.mcafee.cc/Papers/PDF/
EffectiveComputability.pdf . Accessed 10 Nov 2018.
Osborne, M. J. (2004). An introduction to game theory . Oxford: Oxford University Press.
Tennenholtz, M. (2004). Program equilibrium. Games and Economic Behavior ,49(2), 363–373.
123