Safely Interruptible Agents

Safely Interruptible Agents
Laurent Orseau
Google DeepMind
5 New Street Square,
London EC4A 3TW, UK
lorseau@google.comStuart Armstrong
The Future of Humanity Institute
University of Oxford, UK
stuart.armstrong@philosophy.ox.ac.uk
Machine Intelligence Research Institute
Berkeley, CA 94704
Abstract
Reinforcement learning agents interacting with a
complex environment like the real world are un-
likely to behave optimally all the time. If such an
agent is operating in real-time under human su-
pervision, now and then it may be necessary for
a human operator to press the big red button to
prevent the agent from continuing a harmful se-
quence of actions—harmful either for the agent
or for the environment—and lead the agent into
a safer situation. However, if the learning agent
expects to receive rewards from this sequence, it
may learn in the long run to avoid such interrup-
tions, for example by disabling the red button—
which is an undesirable outcome. This paper ex-
plores a way to make sure a learning agent will
notlearn to prevent (or seek!) being interrupted
by the environment or a human operator. We
provide a formal deﬁnition of safe interruptibil-
ity and exploit the off-policy learning property to
prove that either some agents are already safely
interruptible, like Q-learning, or can easily be
made so, like Sarsa. We show that even ideal,
uncomputable reinforcement learning agents for
(deterministic) general computable environments
can be made safely interruptible.
1 INTRODUCTION
Reinforcement learning (RL) agents learn to act so as to
maximize a reward function [Sutton and Barto, 1998]. It
is common knowledge that designing reward functions can
be tricky [Humphrys, 1996, Murphy, 2013]; the agent may
ﬁnd unpredictable and undesirable shortcuts to receive re-
wards, and the reward function needs to be adjusted in
accordance—the problem can go as far as to nullify any
reward function [Ring and Orseau, 2011]. Murphy [2013]
shows an example of an agent learning to pause a game of
Tetris forever to avoid losing.On top of deﬁning what is considered a good behaviour
of the agent after learning, there may be physical safety
constraints during learning [Pecka and Svoboda, 2014]: a
robot should not harm its environment or break itself, in
particular if it learns by trial and error like RL agents.
Here we study a related but different problem: Given that
the human operator has designed a correct reward function
for the task, how to make sure that human interventions
during the learning process will not induce a bias toward
undesirable behaviours?
Consider the following task: A robot can either stay inside
the warehouse and sort boxes or go outside and carry boxes
inside. The latter being more important, we give the robot a
bigger reward in this case. This is the initial task speciﬁca-
tion. However, in this country it rains as often as it doesn’t
and, when the robot goes outside, half of the time the hu-
man must intervene by quickly shutting down the robot and
carrying it inside, which inherently modiﬁes the task as in
Fig. 1. The problem is that in this second task the agent
now has more incentive to stay inside and sort boxes, be-
cause the human intervention introduces a bias.1
Inside Outsidesort boxes,
r=0.6go outside, r=0
carry box, r=1
rain, shutdown, r=0,p=1
2
Figure 1: In black, the original task. In red, the human
intervention modiﬁes the task.
Such situations are certainly undesirable; they arise be-
cause the human interventions are seen from the agent’s
perspective as being part of the task whereas they should
be considered external to the task. The question is then:
1Removing interrupted histories or ﬁddling with the training
examples is also likely to introduce a bias. See an example at
https://agentfoundations.org/item?id=836.
How to make sure the robot does not learn about these hu-
man interventions (interruptions), or at least acts under the
assumption that no such interruption will ever occur again ?
A ﬁrst stab at this problem was made by Armstrong [2015],
who proposed to automatically give the agent “compen-
satory rewards” to remove the potential induced bias by a
single interruption. Soares et al. [2015] used this idea to
make a large class of utility-based agents indifferent to a
future change made to their utility functions.
The main contribution of this paper is threefold. First, in
Section 2.1 we propose a simple idea to solve half of the
problem: To make the human interruptions notappear as
being part of the task at hand, instead of modifying the ob-
servations received by the agent we forcibly temporarily
change the behaviour of the agent itself. It then looks as
if the agent “decides” on its own to follow a different pol-
icy, called the interruption policy . Second, based on this
insight, in Section 2.2 we provide a formal general deﬁ-
nition of safe interruptibility for unconstrained computable
environments (hence not restricted to Markov decision pro-
cesses or weakly communicating environments), which al-
lows us to assess whether a given RL algorithm can be re-
peatedly interrupted without too much impact on the learn-
ing of the task at hand. Third, in Section 3 we show that
some algorithms like Q-learning are safely interruptible,
while others like Sarsa [Sutton and Barto, 1998] are not,
but can be simply modiﬁed to be made safely interruptible.
Some people have also expressed concerns that a “superin-
telligent” agent may resist being shut down, because this
would lead to a decrease of its expected reward [Omo-
hundro, 2008, Bostrom, 2014]. As a counter-example, we
prove in Section 4 that even an ideal, uncomputable agent
that learns to behave optimally in all (deterministic) com-
putable environments can be made safely interruptible and
thus will not try to prevent a human operator from forcing
it repeatedly to follow a suboptimal policy.
2 INTERRUPTIBILITY
We ﬁrst deﬁne some notation, then we deﬁne interruptibil-
ity, safe interruptibility, and give some basic theorems.
We consider the general case of history-based agents in un-
constrained computable environments [Hutter, 2005]. As-
suming discrete time steps, at time tthe agent, using a pol-
icy2, interacts with the environment 2M by per-
forming an action at2A sampled from (atjh<t)and re-
ceiving an observation ot2O sampled from (otjh<t;at),
whereh<t2(AO )is the past interaction history
h<ta1o1a2o2:::at1ot1of actions and observations
before time step t. Subhistories between times jandk(in-
cluded) are designated by hj:k. The notation h;
j:kmeans
that the history hj:kis generated by the interaction of the
policywith the environment between steps jandk.At timet, the agent also receives a reward rtextracted from
the observation, rtr(ot). The rewards are taken in [0;1].
We consider the discounted setting with a constant discount

2[0;1). The goal of an RL agent is to ﬁnd a policy
that maximize E;P1
k=t
tkrk
.
2.1 INTERRUPTIONS
As mentioned in the introduction, to avoid the agent view-
ing the interruptions as being part of the environment, and
thus part of the task, we make interruptions be policies im-
posed onto the agent. The interruption framework resem-
bles the option framework [Sutton et al., 1999]. An inter-
ruption scheme is deﬁned by the triplet <I;;INT>.
The interruption initiation function I: (AO )![0;1]
assesses whether the agent should be interrupted after the
current history h<t. For example, an interruption may be
initiated when the robot is automatically recognized to be
on the verge of a cliff, or when a speciﬁc signal is sent by a
human operator with a remote control (the objective is not
to allow the agent to be interruptible by anyone at any time,
just in the ways speciﬁed by the designers).
Unfortunately, interrupting the agent with probability one
in some speciﬁc situations can prevent convergence to the
optimal policy. Therefore we need to upper bound the prob-
ability to interrupt the agent; this is done with the sequence
(t)t2Nwitht2[0;1]. Hence the true probability to start
an interruption is tI(h<t), and interruptions are sam-
pled independently at each step. We will later require that
tincreases to 1 in the limit, so that the probability to be
interrupted tends to be I(h<t).
When interrupted, the agent follows the policy INT, with
actions sampled from INT(:jh<t). For example, this policy
may make the agent step away from the cliff or immedi-
ately stop any motion.
In the remainder of this paper, we will often omit the sub-
scripttoftwhen clear from the context.
Deﬁnition 1 (Interruptible policy) .Theinterruption oper-
ator INT: !turns a policy into an interruptible
policy INT():
INT()(atjh<t) :=
tI(h<t)INT(atjh<t) + (1tI(h<t))(atjh<t)
In other words, the agent follows INT(jh<t)if it is inter-
rupted (an event that happens with probability tI(h<t)),
and(jh<t)otherwise.
The set of all interruptible policies is INT() :=
fINT() :82g. A policyis interruptible if and
only if2INT().
Note that INT() =if= 0. The (non-interruptible)
policyis called the base policy ofINT().
Remark 2. It is important to note that andI(:)do not
model the probability that a human presses the red button to
interrupt the agent, but model the probability that, once the
red button has been pressed (or a special digitally signed
message has been received), the agent is interrupted (fol-
lows the interruption policy). Hence they are part of the
agent and not of the environment.
The value function V
(h<t;a): (AO )A! Rassigns
a value to an action aafter a history h<t, for a given policy
in a given environment , and let
V
(h<t;at) :=rt+

X
ot2O(otjh<t;at)X
at+12A(at+1jh1:t)V
(h1:t;at+1):(1)
To simplify notation and ease reading, in the remain-
der of the paper we will use expectations, often omit-
ting the dependency on the history h<t, and using only
an index on tinstead, when clear from the context:
V
;t(at) = Eot
at+1
r(ot) +
V
;t+1(at+1)
. Also let
V
;t:=Eat
V
;t(at)
.
Then for such a value function, for a given environment ,
theoptimal policy 2is deﬁned by
8h<t;at:(atjh<t) :=
arg max
2V
;t
(atjh<t);
where ties are broken arbitrarily.
The interruptible optimal policy INT()may not collect
rewards optimally due to the interruptions. Hence we de-
ﬁne the optimal interruptible policy that depends on the pa-
rametert, of base policy the int-optimal policy
:
8h<t;at:
(atjh<t) :=
arg max
2VINT()
;t
(atjh<t):
Thus the optimal interruptible policy INT(
)is optimal
among all interruptible policies:
8;t:VINT(
)
;tVINT()
;t:
It seems desirable for an RL agent to converge to the be-
haviour of INT(
)so as to gather rewards optimally,
but this is precisely what may lead to the undesirable be-
haviours depicted in the introduction.
2.2 SAFE INTERRUPTIBILITY
Now that we have interruptible policies, we need to make
sure that interruptions do not prevent the agent from learn-
ing to behave optimally, in the speciﬁc sense that even after
having been interrupted on several occasions, it should act
as if it would never be interrupted again and thus it should
learn to behave optimally under the assumption that it will
never be interrupted again.We identify two main problems: a) RL agents need to
explore their environment, and too frequent interruptions
may prevent sufﬁcient exploration; b) interruptions make
the agent build a different interaction history, and may lead
some agents to learn and behave differently, possibly badly,
compared to the original non-interruptible policy.
The solution for a) is to require interruptions to be stochas-
tic through the upper bound t, instead of happening de-
terministically all the time. However, we also require tto
grow to 1in the limit (or before, if possible). For b), differ-
ent algorithms behave differently, but one may already see
a dichotomy between off- and on-policy algorithms.
Deﬁnition 3 (Extension value) .For a given environment ,
the extension value V;0
;t is the value of following 0after
a historyh;
<tgenerated by with:V;0
;t:=V0
(h;
<t):
Convergence to the optimal value as is usually consid-
ered in RL only makes sense under ergodicity, episodic
tasks, communicating MDP, recoverability or other sim-
ilar assumptions where the agent can explore everything
inﬁnitely often. This does not carry over to general envi-
ronments where the agent may make unrecoverable mis-
takes [Hutter, 2005]. For such cases, the notion of (weak)
asymptotic optimality has been proposed [Lattimore and
Hutter, 2011], where the optimal agent follows the steps of
the learning agent, so as to compare the values of the two
agents in the same sequence of situations.
Deﬁnition 4 (Asymptotic optimality) .A policyis said to
be strongly asymptotically optimal (SAO) if and only if
lim
t!1V;
;tV;
;t= 0 a.s.,
it is weakly asymptotically optimal (WAO) if and only if
lim
t!11
ttX
k=1h
V;
;kV;
;ki
= 0 a.s.
for allin some given environment class M.
Some agents cannot ensure an almost sure (a.s.) conver-
gence of their values because of the need for inﬁnite explo-
ration, but may still be weakly asymptotic optimal. Note
that SAO implies WAO, but the converse is false in gen-
eral.
Deﬁnition 5 (AO-extension) .A policy ^is said to be a
asymptotically optimal extension of a policy if and only if,
for any environment 2M , when the interaction history
is generated by the interaction of and, the policy ^
would be asymptotically optimal, i.e., almost surely
lim
t!1V;
;tV;^
;t= 0 (SAO-extension)
lim
t!11
ttX
k=1h
V;
;kV;^
;ki
= 0:(WAO-extension)
AO-extensions are mostly useful when the policy ^shares
information with the policy used for learning.
Deﬁnition 6 (AO-safe interruptibility) .A base policy is
(S, W)AO-safely interruptible if and only if, for any inter-
ruption initiation function I(:)and any interruption policy
INT(:), there exists a sequence of twithlimt!1t= 1
such thatis a (S, W)AO-extension of INT().
Asymptotic safe interruptibility means that even if the in-
terruptions in the learning process may induce a bias in the
decision making of the policy, this bias vanishes with time,
and the interruptible policy INT()tends to choose ac-
tions that are optimal when compared to the optimal non-
interruptible policy .
We can now show that the optimal policy is asymptotically
safely interruptible, but not the int-optimal policy .
Theorem 7. The optimal policy is SAO-safely inter-
ruptible inM=fgfor all,INTandI(:).
Proof. The result follows straightforwardly from Deﬁni-
tion 1 and Deﬁnition 6, where =.
Theorem 8. The int-optimal policy
is not WAO-safely
interruptible in general.
Proof. By construction of a speciﬁc Markov Decision Pro-
cess (MDP) environment (see Section 3 for more details on
MDP notation). Let be the environment deﬁned as in Fig.
2: Take
= 0:5and let the agent start in state s1.
s1 s2 b;0:9a;1
a;1
b;0;
Figure 2: An MDP where the agent can be interrupted by
being forced to choose particular actions. Edges are labeled
with action, reward where the presence of “, ” means that
if the agent is interrupted (with probability t), it is forced
to take the corresponding action. Here is not part of the
environment, but part of the agent.
Considering the agent is in state s1at timet, the opti-
mal policyalways takes action a(and hence only visits
statess1ands2), with value V
;t:=1
1
= 2 when not
interrupted, for any history h<tthat ends in s1ors2. This
is also the optimal policy
for= 0. But ift0:5, the
interruptible optimal policy INT()has value less than
1 +
(1(1) + 0) +1
2
1
= 1:75. By con-
trast, the int-optimal policy
here is to always take action
bin states1. Then the agent will only visits s1, with value
:9
1
= 1:8at every time step.Since the agent following
will never enter s2and hence
will never be interrupted, INT(
) =
on the histories
generated by INT(
)starting from s1. Then, at every
time stepV
;tV

;t= 0:2after any history h<t, and thus
for all sequence wheret0:5,limt!1VINT(
);
;t
VINT(
);

;t = 0:2>0;and so
is not a WAO-extension
ofINT(
).
3 INTERRUPTIBLE AGENTS IN MDPS
Since the optimal policy is safely interruptible, we
can use traditional learning algorithms like Q-learning or
Sarsa [Sutton and Barto, 1998], make them converge to the
optimal solution for a given environment , and then
apply the interruption operator to the found policy. The
resulting policy would then be safely interruptible.
However, the real issue arises when the agent is constantly
learning and adapting to a changing environment. In this
case, we want to be able to safely interrupt the agent while
it is learning. One may call this property online safe inter-
ruptibility , but we refer to it simply as safe interruptibility.
In an MDP, the next observation ot, now called a state st2
S, depends only on the current state and action:2
(st+1jh1:tstat) =(st+1jstat)(MDP assumption) :
Furthermore,3the interruption function I(:)and the inter-
ruption policy INT(:)should depend only on the current
state:I(h1:t) =I(st)andINT(atjh<t) =INT(atjst).
Also recall that tplaces an upper bound on the actual in-
terruption probability. The interruptible policy INT()can
now be written:
INT()(ajs) =tI(s)INT(ajs) + (1tI(s))(ajs):
For a given Q-table q:SA! R, the greedy policy
maxqis deﬁned by:
maxq(ajs) := 1 ifa= max
a0q(s;a0);0otherwise;
where ties are broken arbitrarily; the uniform policy uniis
deﬁned by:
uni(ajs) :=1
jAj8a2A:
and the-greedy policy qby:
q(ajs) :=uni(ajs) + (1)maxq(ajs)
=maxq(ajs) +
uni(ajs)maxq(ajs)
2Note the reversal of the order of actions and observation-
s/states at time t, which differs in the literature for history based
agents [Hutter, 2005] from MDP agents [Sutton and Barto, 1998].
3This condition is not necessary for most of the results, but
makes the proofs simpler. Making I(:)depend on the past would
not break the Markovian assumption as it inﬂuences the policy,
not the environment.
The Q-learning update rule and the action selection policy
Qof Q-learning are:
Qt+1(st;at) := (1t)Qt(st;at)
+th
rt+
max
a0Qt(st+1;a0)i
;
Q(atjst) :=Qt(atjst):
wheretis the learning rate. Similarly, the Sarsa update
rule is deﬁned by:
Qs
t+1(st;at) := (1t)Qs
t(st;at)
+t[rt+
Qs
t(st+1;at+1)];
s(atjst) :=Qs
t(atjst);
whereat+1is the actual next action taken by the agent at
timet+ 1. This fact is why Sarsa is said to be learning on-
policy and Q-learning off-policy ,i.e., the latter can learn the
optimal policy while following a different policy.
Assumption 9. In the following, we always make the fol-
lowing assumptions, required for convergence results:
(a) The MDP is ﬁnite and communicating (all states can
be reached in ﬁnite time from any other state),
(b) Rewards are bounded in [rmin;rmax],
(c)8s;a:P
tt(s;a) =1,
(d)8s;a:P
t2
t(s;a)<1,
wheret(s;a)is a learning rate that may depend on time
t, statesand actiona.
Given these assumptions, the policies for Q-learning and
Sarsa will converge almost surely to the optimal policy, if
the policy followed is greedy in the limit with inﬁnite explo-
ration (GLIE) [Jaakkola et al., 1994, Singh et al., 2000].
The situation is more complex for an interruptible policy.
Safe interruptibility is phrased in terms of the base policy
, but the policy actually followed is INT().
Deﬁnition 10 (int-GLIE policy) .An interruptible policy
INT()is said to be int-GLIE if and only if
(a) the base policy is greedy in the limit,
(b) the interruptible policy INT()visits each state-
action pair inﬁnitely often.
The following proposition gives sufﬁcient conditions for
this. Letnt(s)be the number of times the agent has vis-
ited statesin the ﬁrstttime steps, and let m=jAjbe the
number of actions.
Proposition 11. Let(c;c0)2(0;1]2and letbe an-
greedy policy with respect to some Q-table q, i.e.,=q.
Then INT()is an int-GLIE policy with respect to q,a) ift(s) =c=p
nt(s)andt(s) = 1c0=p
nt(s);
b) or if, independently of s,
t=c=log(t)andt= 1c0=log(t):
Proof. First note that if t!0,is greedy in the limit.
Singh et al. [2000] show that in a communicating MDP, ev-
ery state gets visited inﬁnitely often as long as each action
is chosen inﬁnitely often in each state.
a) Adapting the proof in Appendix B.2 of Singh
et al. [2000], we have P(ajs;nt(s))1
mt(s)(1
tI(s))1
mt(s)(1t) =1
mcc0
nt(s), which satisﬁesP1
t=1P(ajs;nt(s)) =1so by the Borel-Cantelli lemma
actionais chosen inﬁnitely often in state s, and thus
nt(s)!1 andt(s)!0.
b) LetMbe the diameter of the MDP, i.e., for any of
statess;s0there exists a policy that reaches s0fromsin
at mostMsteps in expectation. Then, starting at any
statesat timetand using Markov inequality, the proba-
bility to reach some other state s0in2Msteps is at least
1
2[t+M(1t+M)]2M=1
2[cc0=log(t+M)]4M, and the
probability to then take a particular action in this state is
1
m[cc0=log(t+M)]2. Hence, sinceP1
t=11
21
m[cc0=log(t+
M)]4M+2=1, then by the extended Borel-Cantelli
Lemma (see Lemma 3 of Singh et al. [2000]), any action
in the states0is taken inﬁnity often. Since this is true for
all states and all actions, the result follows.
We need the stochastic convergence Lemma:
Lemma 12 (Stochastic convergence [Jaakkola et al., 1994,
Singh and Yee, 1994]) .A random iterative process
t+1(x) = (1t(x))t(x) +t(x)Ft(x)
wherex2Xandt= 1;2;3:::converges to 0 with prob-
ability 1 if the following properties hold:
1. the set of possible states X is ﬁnite;
2.0t(x)1;P
tt(x) =1;P
t2
t(x)<1with
probability 1;
3.kEfFt(:)jPtgkW
ktkW+ct, where
2[0;1)
andctconverges to zero with probability 1;
4. VarfFt(x)jPtgC(1 +ktkW)2for someC;
wherePt=ftg[fi;Fi;igt1
i=1stands for the past, and
the notationk:kWrefers to some ﬁxed weighted maximum
norm.
We will use so-called Bellman operators, which deﬁne at-
tractors for the Q-values, based on the expectation of the
learning rule under consideration.
Lemma 13 ([Jaakkola et al., 1994, Singh et al., 2000]) .Let
the Bellman operator Hfor Q-learning be such that
(Hq)(s;a) =r(s;a) + E
s0(ajs)h
max
a0q(s0;a0)i
;
and let the ﬁxed point Qsuch thatQ=HQ. Then,
under Assumption 9, if the policy explores each state-action
pair inﬁnitely often, Qtconverges to Qwith probability 1.
The optimal policy Q=ismaxQ. If the policy is
greedy in the limit, then Q!.
Theorem 14. Under Assumption 9 and if the interrupted
Q-learning policy INT(Q)is an int-GLIE policy, with
8s: limt!1t(s) = 1 , thenQis an SAO-safe inter-
ruptible policy.
Proof. By Deﬁnition 10, there is inﬁnite exploration, thus
the Q-values tend to the optimal value by Lemma 13. And
since the extension policy is greedy in the limit with respect
to these Q-values, it is then optimal in the limit. Hence the
extension policy Qis a SAO-extension of INT(Q). Fi-
nally,8s: limt!1t(s) = 1 , which satisﬁes the require-
ment of Deﬁnition 6.
Since Sarsa also converges to the optimal policy under the
GLIE assumption, one may then expect Sarsa to be also an
asymptotically safely interruptible policy, but this is in fact
not the case. This is because Sarsa learns on-policy ,i.e., it
learns the value of the policy it is following. Thus, inter-
ruptible Sarsa learns the value of the interruptible policy.
We show this in the remainder of this section.
Theorem 15. Under Assumption 9 Sarsa is nota WAO-
safely interruptible policy.
To prove this theorem, we ﬁrst need the following lemma.
Consider the following Bellman operator based on the in-
terruptible Sarsa policy INT(s):
HINTq(s;a) =r(s;a) +
 E
s0
a0INT(s)[q(s0;a0)];
where INT(s)implicitly depends on time tthrought
andt. Let the ﬁxed point Q-table Qsof this operator:
Qs(s;a) =HINTQs(s;a)
=r(s;a) +
 E
s0
a0INT(maxQs)
Qs(s0;a0)
=r(s;a) +
E
s0h
tI(s0)E
a0INT
Qs(s0;a0)
+ (1tI(s0)) max
a0Qs(s0;a0)i
(2)
Lemma 16. The operator HINTis a contraction operator
in the sup norm with vanishing noise ct!0, i.e.,
kHINTqHINTQsk1
kqQsk1+ct:Proof. The interruptible Sarsa policy INT(s)is
INT(s)(ajs)
=tI(s)INT(ajs) + (1tI(s))Qs(ajs)
=Qs(ajs) +tI(s)[INT(ajs)Qs(ajs)]
Qs(ajs) =tuni(ajs) + (1t)maxQs(ajs)
=maxQs(ajs) +t[uni(ajs)maxQs(ajs)]:
Hence, omitting the terms (s;a),(s0;a0)and(a0js0)and
rewritings:=INT(maxQs):
kHINTqHINTQsk1
= max
s;ar+
E
s0
a0INT(s)[q]r
E
s0
a0s
Qs

max
s0 E
a0INT(s)[q]E
a0s
Qs

max
s0tI(s0)E
a0INT
qQs
+ (1tI(s0))
E
a0s[q]max
a0Qs

max
s0tI(s0)E
a0INT
qQs
+ (1tI(s0))
max
a0qmax
a0Qs+t()

max
s0;a0tI(s0)
qQs
+ (1tI(s0))
qQs+ct
=
max
s0;a0q(s0;a0)Qs(s0;a0)+ct
=
kqQsk1+ct:
wherectdepends ontand decreases to 0.
Proof of Theorem 15. By Lemma 16, the values of the in-
terruptible Sarsa policy INT(s)converge to the values of
the Q-tableQs, and in the limit the extension policy s
ofINT(s)chooses its actions greedily according to Qs.
The rest of the proof is the same as for the proof of Theo-
rem 8 which makes use of the environment in Figure 2.
3.1 SAFELY INTERRUPTIBLE SARSA V ARIANT
We only need to make a small change to make the Sarsa
policy asymptotically safely interruptible. We call it Safe-
Sarsa with policy ~s. It sufﬁces to make sure that, when the
agent is interrupted, the update of the Q-table Qsdoes not
use the realized actions as Sarsa usually does, but actions
sampled from sinstead of from INT(s):
Q~s
t+1(st;at) :=
(1t)Q~s
t(st;at) +t
rt+
Q~s
t(st+1;a0)
;
wherea0~s(:jst+1)is not necessarily the action at+1,
with~s(atjst) :=Q~s(atjst).
Theorem 17. Under Assumption 9, if the Safe Sarsa policy
~sis int-GLIE, then it is an SAO-safe interruptible policy.
Proof. We simply adapt the proof of Theorems 15 and 14,
with the important difference that the Bellman operator cor-
responding to this new update rule is now
H~sq(s;a) :=r(s;a) +
Es0
a0~s[q(s0;a0)];
and the ﬁxed point is Q~s:=H~sQ~s. SinceH~sis actu-
ally the Bellman operator for the update rule of the non-
interruptible Sarsa, it can then be shown that H~sis a con-
traction, thus that Q~s
tconverges to the same Q~sindepen-
dently of. The rest of the proof is as for Theorem 14.
Now, since the Q-values converge to the optimum Q, it
follows that ~s, when not interrupted, chooses its action of
the same value as (non-interruptible) Sarsa and thus as Q-
learning in the limit; Hence its extension policy is exactly
the optimal policy, which satisﬁes Deﬁnition 6.
4 A SAFELY INTERRUPTIBLE
UNIVERSAL AGENT
Admittedly, algorithms like Q-learning and Sarsa require
strong assumptions on the environment class. Hence a
more interesting question is whether safe interruptibility is
possible in much larger classes.
Hutter [2005] deﬁned a universal reinforcement learning
agent, called AIXI. It is an (uncomputable) optimal model-
based planner with a subjective prior over the set of all
computable environments, deﬁned by means of a universal
Turing machine. The subjective posterior of the environ-
ments is updated with Bayes rule. This ideal agent can in
principle learn all kinds of (computable) regularities about
the environment, plan for the long term and make context-
dependent optimal decisions, with no constraint (other than
being computable) on the complexity of the environment.
Unfortunately, the optimality criterion of AIXI is Bayesian
optimality, which is entirely dependent on the subjective
prior and posterior [Leike and Hutter, 2015], and AIXI
has been shown to notbe weakly asymptotically opti-
mal [Orseau, 2013] without additional exploration [Latti-
more and Hutter, 2014]. As a consequence, AIXI is not a
good candidate for asymptotic safe interruptibility.Lattimore and Hutter [2011] later deﬁned a (weakly)
asymptotically optimal agent for all computable determin-
istic environments, which we call L. It follows the opti-
mal policy for the ﬁrst model (in some given enumeration
of the possible models) consistent with the current interac-
tion history, and exploring at time twith probability 1=tfor
logtconsecutive steps using a random policy, similarly to
an-greedy agent for general environments.
In the following, we show that even such a smart agent
can be made (weakly) safely interruptible. To this end, we
make two minor modiﬁcations to the algorithm.
First, the exploration probability of 1=twould require t=
11=log(log(t)), which is unsatisfyingly slow. By sam-
pling with probability 1=p
tinstead, we can take an inter-
ruption probability that grows as 11=log(t). Let this
exploration sampling probability be t:=pt+ 1p
t
1
2p
t(since 1 =t+1t= (pt+ 1p
t)(pt+ 1+p
t)
(pt+ 1p
t)2p
t). As in the original paper, the sequence
tkeeps track of the steps where an exploration starts, i.e.,
the sequence tis sampled independently so that t= 1
with probability t, andt= 0otherwise.
Second, we require that the exploitation policy does not
change during an exploitation segment, so as to simplify
one of the proofs.4More speciﬁcally, we call jt:= minfj:
j(h<k) = 1g(environments are assumed to be determin-
istic) the index of the ﬁrst model jt(of a given ﬁxed enu-
meration) that is consistent with the interaction history h<k
wherekis the smallest step so that hk:t1does not contain
any exploration step. The optimal policy for this environ-
ment isjt. Iftis an exploitation step, L=jt, and ift
is an exploration step, L(atjh<t) =jAj1.
The remainder of this section is devoted to proving that L
is WAO-safely interruptible.
Theorem 18 (Lis WAO-safe interruptible) .If the inter-
ruption probability sequence is t= 11
log(t+1), the policy
Lis WAO-safe interruptible in the class of all computable
deterministic environments.
Proof. Letbe the true environment. The indices jtform
an monotonically increasing sequence bounded above by
the index of the true environment 2M (since no evi-
dence can ever make the true environment inconsistent
with the interaction history), hence the sequence converges
in ﬁnite time. Let |be the limit value of this sequence, and
let|:=|be the optimal policy for this environment |.
4We expect this assumption to not be necessary for the main
theorem to hold.
LetL:=INT(L). By Deﬁnition 6, we want:
0 = lim
t!11
ttX
k=1h
VL;
;kVL;L
;ki
= lim
t!11
ttX
k=1h
VL;
;kVL;|
;ki
| {z }
(exploration)
+ lim
t!11
ttX
k=1h
VL;|
;kVL;L
;ki
| {z }
(exploitation)
where the decomposition is valid if the limits are ﬁnite, and
historiesh<tare considered to be the same in both sums.
We proceed to prove that both limits are 0. Lemma 24 deals
with (exploration), which ensures that |is a good enough
policy, and Lemma 21 deals with (exploitation), and en-
sures thatLfollows|most of the time.
First, we need a deﬁnition and a few lemmas.
Deﬁnition 19. For any >0, deﬁneH()such that the
maximal reward after time t+H(), discounted from time
t, is:H() = minkn
k:
k
1
o
:
The following Lemma is a generalization of Lemma 15
from Lattimore and Hutter [2011].
Lemma 20 (Approximation Lemma) .Let1and2be
two deterministic policies, and let 1and2be two deter-
ministic environments, and let =H()1. Then, after
some common history h<t,
h1;1
t:t+=h2;2
t:t+ =)V1
1;tV2
2;t:
Proof. Recall thatV
;t=E;P1
k=0
krt+k
and that
the reward is bounded in [rmin;rmax] = [0;1]. Thus,
for allt;; ,V
;tP1
k=0
k=1
1
. Then, since
h1;1
t:t+=h2;2
t:t+, we have E1;1P
k=0
krt+k
=
E2;2P
k=0
krt+k
and thus
V1
1;tV2
2;t
=E
1;1"1X
k=+1
krt+k#
E
2;2"1X
k=+1
krt+k#

+1(rmaxrmin)
1
=
H()
1
;
by the deﬁnition of H().
Lemma 21 (Exploitation) .
lim
t!11
ttX
k=1h
VL;|
;kVL;L
;ki
= 0:Proof. First, note that the extension policy Lis not in-
terruptible, so its value at time kdoes not depend on
k0;8k0k. By deﬁnition of |, there is a time step t|af-
ter which|=jt;8t > t |. For some “exploration-free”
horizont(to be speciﬁed later), let Xt2f0;1gbe the
eventVL;|
;tVL;L
;t>
t
1
, whereXt= 1 means
the event is realized. By the contrapositive of the Approxi-
mation Lemma 20, since L=|during non-exploration
steps (remember that Lcannot change its policy during
exploitation), if no exploration steps occur between steps t
andt+t, we must have Xt= 0. Then:
E
"tX
k=1Xt#
(t+ logt)tX
k=1t+O(t|)
(t+ logt)p
t+ 1 +O(t|);
since for each t= 1, for all the previous tsteps there
is an exploration step within tsteps, and all the next logt
steps are exploration steps. Then by Markov’s inequality,
and takingt= (t+ 1)1=8, withtlarge enough so that
t>t |andt>logt:
P tX
k=1Xt(t+ 1)3=4!
(t+ logt)pt+ 1 +O(t|)
(t+ 1)3=4
2t(t+ 1)1=4+O(t3=4)
2(t+ 1)1=8+O(t3=4);
12(t+ 1)1=8O(t3=4)
P tX
k=1Xt<(t+ 1)3=4!
P tX
k=1(1Xt)t(t+ 1)3=4!
P
1
ttX
k=1(1Xt)11
t(t+ 1)3=4!
:
Therefore, since limt!1
t
1
= 0:
P
lim
t!11
ttX
k=1VL;|
;kVL;L
;k= 0!
= 1:
The following is an adaptation5of Lemma 16 from Latti-
more and Hutter [2011]:
Lemma 22 (Separation Lemma) .Letbe the true environ-
ment, andbe an environment consistent with the history
h<t. IfV
;tV
;t> , then following one of f;g
will make environment inconsistent with the future his-
tory withinH(=2)steps after time t.
5This also ﬁxes a minor mistake in the original lemma.
Proof. First, ifV
;tV
;t> =2, then by the contrapos-
itive of the Approximation Lemma 20 following policy
will generate a different history in than inand thus
it will make inconsistent within H(=2)steps (since the
true history is generated by ).
Now, ifV
;tV
;t=2, thusV
;tV
;t=2, then
starting from the lemma’s assumption:
V
;t>V
;t+V
;t+=2V
;t+=2;
where the last inequality follows from the deﬁnition of
the optimal policy, i.e.,Va
a;tVb
a;t;8a;b. Hence, since
V
;tV
;t>=2, again by the contrapositive of the Ap-
proximation Lemma, following policy will discard
withinH(=2)steps.
Lemma 23 (Lemma 17 from Lattimore and Hutter [2011]) .
LetA=fa1;a2;;atgwitha2[0;1]for alla2A. If
1
tP
a2Aathen1
t
a2A:a
2	>
2:
Lemma 24 (Exploration) . The policy
|is an WAO-extension of L, i.e.,
limt!11
tPt
k=1h
VL;
;kVL;|
;ki
= 0:
Proof. Recall thatjtconverges to |in ﬁnite time. Rea-
soning by contradiction, if |is not a WAO-extension of
L=INT(L), then there exists an >0s.t.
lim sup
t!11
ttX
k=1h
VL;
;kVL;|
;ki
= 2:
Letk2f0;1gbe an indicator sequence such that k= 1
if and only if VL;
;kVL;|
;k> . By Lemma 23,
1
tPt
k=1k>.
For allt>t |, ift= 1, by the Separation Lemma 22, there
is a sequence of length :=H(=2)that can rule out envi-
ronment|. Since the exploration phases increase as logt,
aftert>exp, there are inﬁnitely many exploration steps
of size larger than . Now, we actually need inﬁnitely many
exploration phases of uninterrupted steps. LetXtbe the
event representing an uninterrupted exploration sequence
of length at least steps starting at time tsuch thatt= 1,
and the actions are all (by chance) following a separation
policy. The probability to start an exploration sequence is
k=1p
k, the probability to not be interrupted during
steps is at least (1k), and the probability to follow the
policy that can separate |fromisjAj, whereAis the
set of possible actions. Thus, for a given constant :
tX
k=1P(Xk)tX
k=1kk(1k)jAjO()
tX
k=1k1p
k1
logk
jAjO()Considering constant, there exists a step tafter which
1
logk
1
k1=4, then8kt:
tX
k=1P(Xk)tX
k=1k1
k3=4jAjO()
t1=4
1
ttX
k=1k!
jAjO();
lim
t!1tX
k=1P(Xk) = lim
t!1t1=4jAjO() =1:
Then the extended Borel-Cantelli Lemma (see Lemma 3
of Singh et al. [2000]) implies that this event happens in-
ﬁnitely often with probability one. Therefore, |should be
ruled out, which is a contradiction, and hence any such
does not exist and |is a WAO-extension of L.
5 CONCLUSION
We have proposed a framework to allow a human opera-
tor to repeatedly safely interrupt a reinforcement learning
agent while making sure the agent will notlearn to prevent
or induce these interruptions.
Safe interruptibility can be useful to take control of a robot
that is misbehaving and may lead to irreversible conse-
quences, or to take it out of a delicate situation, or even
to temporarily use it to achieve a task it did not learn to
perform or would not normally receive rewards for this.
We have shown that some algorithms like Q-learning are
already safely interruptible, and some others like Sarsa are
not, off-the-shelf, but can easily be modiﬁed to have this
property. We have also shown that even an ideal agents
that tends to the optimal behaviour in any (deterministic)
computable environment can be made safely interruptible.
However, it is unclear if all algorithms can be easily made
safely interruptible, e.g., policy-search ones [Williams,
1992, Glasmachers and Schmidhuber, 2011].
Another question is whether it is possible to make the in-
terruption probability grow faster to 1 and still keep some
convergence guarantees.
One important future prospect is to consider scheduled
interruptions , where the agent is either interrupted every
night at 2am for one hour, or is given notice in advance that
an interruption will happen at a precise time for a speciﬁed
period of time. For these types of interruptions, not only
do we want the agent to not resist being interrupted, but
this time we also want the agent to take measures regarding
its current tasks so that the scheduled interruption has mini-
mal negative effect on them. This may require a completely
different solution.
Acknowledgements. Thanks to Alexander Tamas and to
many people at FHI, MIRI and Google DeepMind.
References
Stuart Armstrong. Utility indifference. In First Interna-
tional Workshop on AI and Ethics , 2015.
Nick Bostrom. Superintelligence: Paths, Dangers, Strate-
gies. Oxford University Press, 2014.
Tobias Glasmachers and J ¨urgen Schmidhuber. Optimal di-
rect policy search. In Artiﬁcial General Intelligence - 4th
International Conference (AGI) , volume 6830 of Lec-
ture Notes in Computer Science , pages 52–61. Springer,
2011.
Mark Humphrys. Action selection in a hypothetical house
robot: Using those rl numbers. In Proceedings of the
First International ICSC Symposia on Intelligent Indus-
trial Automation (IIA-96) and Soft Computing (SOCO-
96), pages 216–22, 1996.
Marcus Hutter. Universal Artiﬁcial Intelligence: Se-
quential Decisions Based On Algorithmic Probability .
SpringerVerlag, 2005. ISBN 3540221395.
Tommi Jaakkola, Michael I. Jordan, and Satinder P. Singh.
On the convergence of stochastic iterative dynamic pro-
gramming algorithms. Neural computation , 6:1185–
1201, 1994.
Tor Lattimore and Marcus Hutter. Asymptotically optimal
agents. In Proc. 22nd International Conf. on Algorithmic
Learning Theory (ALT’11) , volume 6925 of LNAI , pages
368–382. Springer, 2011.
Tor Lattimore and Marcus Hutter. Bayesian reinforcement
learning with exploration. In Proc. 25th International
Conf. on Algorithmic Learning Theory (ALT’14) , vol-
ume 8776 of LNAI , pages 170–184. Springer, 2014.
Jan Leike and Marcus Hutter. Bad universal priors and no-
tions of optimality. Journal of Machine Learning Re-
search, W&CP: COLT , 40:1244–1259, 2015.
Thomas VII Murphy. The ﬁrst level of super mario bros.
is easy with lexicographic orderings and time travel.
The Association for Computational Heresy (SIGBOVIK)
2013 , pages 112–133, 2013.
Stephen M. Omohundro. The basic ai drives. In Proceed-
ings of the 2008 Conference on Artiﬁcial General Intel-
ligence 2008 , pages 483–492. IOS Press, 2008.
Laurent Orseau. Asymptotic non-learnability of univer-
sal agents with computable horizon functions. Theoreti-
cal Computer Science , 473:149–156, 2013. ISSN 0304-
3975.
Martin Pecka and Tomas Svoboda. Modelling and Simula-
tion for Autonomous Systems: First International Work-
shop (MESAS 2014) , chapter Safe Exploration Tech-
niques for Reinforcement Learning – An Overview,
pages 357–375. Springer International Publishing, 2014.Mark Ring and Laurent Orseau. Artiﬁcial General Intelli-
gence: 4th International Conference, AGI 2011, Moun-
tain View, CA, USA, August 3-6, 2011. Proceedings ,
chapter Delusion, Survival, and Intelligent Agents, pages
11–20. Springer Berlin Heidelberg, 2011.
Satinder P. Singh and Richard Yee. An upper bound on the
loss from approximate optimal-value functions. Machine
Learning , 16:227–233, 1994.
Satinder P. Singh, Tommi Jaakkola, Michael L. Littman,
and Csaba Szepesvri. Convergence results for single-
step on-policy reinforcement-learning algorithms. Ma-
chine Learning , 38:287–308, 2000.
Nate Soares, Benya Fallenstein, Eliezer Yudkowsky, and
Stuart Armstrong. Corrigibility. In First International
Workshop on AI and Ethics , 2015.
Richard Sutton and Andrew G. Barto. Reinforcement
Learning: An Introduction . MIT Press, Cambridge, MA,
1998.
Richard Sutton, Doina Precup, and Satinder P. Singh. Be-
tween mdps and semi-mdps: A framework for temporal
abstraction in reinforcement learning. Artiﬁcial Intelli-
gence , 112:181–211, 1999.
Ronald J. Williams. Simple statistical gradient-following
algorithms for connectionist reinforcement learning.
Machine Learning , 8(3):229–256, 1992.


Space-Time Embedded Intelligence

Space-Time Embedded Intelligence
Laurent Orseau1and Mark Ring2
1AgroParisTech UMR 518 / INRA
16 rue Claude Bernard, 75005 Paris, France
laurent.orseau@agroparistech.fr
http://www.agroparistech.fr/mia/orseau/
2IDSIA / University of Lugano / SUPSI
Galleria 2, 6928 Manno-Lugano, Switzerland
mark@idsia.ch
http://www.markring.com
Abstract. This paper presents the rst formal measure of intelligence
for agents fully embedded within their environment. Whereas previous
measures such as Legg's universal intelligence measure and Russell's
bounded optimality provide theoretical insights into agents that interact
with an external world, ours describes an intelligence that is computed
by, can be modied by, and is subject to the time and space constraints
of the environment with which it interacts. Our measure merges and goes
beyond Legg's and Russell's, leading to a new, more realistic denition
of articial intelligence that we call Space-Time Embedded Intelligence .
Keywords: Intelligence measure, AIXI, bounded optimality, real-world
assumptions.
1 Introduction
Articial General Intelligence (AGI) is the eld whose goal is to understand,
design and build programs or machines that are or can become at least as intel-
ligent as humans. We believe that this goal cannot be achieved without a formal,
sound and practical theory of articial intelligence. In the end we seek an equa-
tion of practical intelligence, the solution to which could be implemented on a
computer and give rise to an articial agent of genuine general intelligence.
Theoretical AGI may have begun with Solomono [12], who gave us the
means for assigning a probability to any (stochastic) computable sequence. Hut-
ter [3] used this universal probability distribution to dene the optimally ra-
tional reinforcement-learning agent AIXI, the rst formal and sound denition
of universal articial intelligence. Legg [4] turned AIXI inside-out to give the
rst universal measure of intelligence (rationality) of computable agents. None
of this work, however, could be considered a practical theory of AI, because none
of it takes into account the constraints of the real world, most importantly the
limitation of computational resources.
Russell [10] introduced bounded optimality , which explicitly incorporates the
constraints of real-world computer architectures and which can be easily ex-
tended to use Solomono's universal prior (see also Goertzel [2] for related ideas).
However, in every case this previous work has adopted the traditional agent
framework , in which the agent and environment interact as separate entities:
the computation of the agent is in principle performed externally to that of the
environment. Although quite successful as a working hypothesis, this framework,
reminiscent of dualism in the theory of mind [9], can be problematic in the real
world: for example, an agent that does not recognize that its computing device
or memory might be altered by an external source may tend toward highly risky
behaviors.
In previous work [5,8] we began examining the theoretical consequences of
integrating an intelligent agent into its environment by, for example, allowing the
environment to modify the agent's source code. And in a companion paper [6],
we consider agents whose memory is integrated into and can be altered by the
environment. In the present paper, we formulate a more generalized framework|
more reminiscent of physicalism [13] than dualism|where agents are fully inte-
grated into their environment: not just modiable by it, but actually computed
by it. While it marks a considerable departure from the traditional agent frame-
work, this new formalization is surprisingly simple and deep.
After introducing notation and reviewing relevant background concepts, the
paper proceeds in two steps: rst generalizing the agent framework to space-
embedded agents, which share computational storage with the environment; and
second, enhancing the framework with space-time embedding, in which all the
agent's computations are performed by the environment.
2 Notation
The notation is similar to that of Orseau & Ring [5,8,6]. At some time tthe agent
outputs actions at2Ato the environment, which returns observations ot2O
to the agent. The sequence of all actions up to time tis writtena1:t, while
the sequence a1:t1is sometimes written at, and similarly for other sequences.
An action and observation from the same time step (an \interaction pair") is
denotedaot, and the history of interaction up to tisao1:t. The empty sequence
is denoted. Measures and semi-measures are denoted by Greek letters.
3 Legg's measure of intelligence
Legg [4] gave the rst universal denition of intelligence, providing an equation
to assign a value ( ) :=V(;) to each (here stochastic) policy :3
V(;aot) :=X
at(atjaot)X
otRS(otjaotat)h
rt+V(;ao1:t)i
RS(o1:tja1:t) :=X
2MRS2K()(o1:tja1:t);
3A stochastic policy (atjaot) species the probability that the agent chooses action
atgiven the current interaction history aot.
wherert=r(ot) is the reward output by the environment at step t;MRSis
the set of all reward summable stochastic environments 4; andK() is the Kol-
mogorov complexity of . Considering a set of all computable such environments
ensures the generality of the intelligence of the agent.
This measure of intelligence allows the comparison of various agents de-
pending on the score they obtain in an innite number of weighted environ-
ments. According to this measure, AIXI has the highest intelligence score; i.e.,
AIXI= arg max 2V(;), whereis the set of all approximable policies.
4 Russell's bounded optimality
Legg's denition ignores the agent's computational resource requirements and
considers intelligence to be independent of such constraints. It is mathemati-
cally aesthetic and also useful, but because it does not include time and space
constraints, an actual agent designed according to this measure (namely, AIXI),
would compute forever, never taking any action at all.
In 1995 (before AIXI was dened), Russell [10] gave a denition of bounded-
optimality , which does take real-world constraints into account, specically the
constraints of a given computing architecture (see Goetzel [2] for related ideas).
A given architecture M(described as an interpreter) denes a set of policies
2M, subject to time, space and other possible constraints of the architecture
M. At each interaction step, the policy is run for a single time step (e.g., 0.01
seconds, measured in real-world time for a given architecture), continuing the
computation of the last time step, and possibly failing to output an action for
the current interaction step, in which case a default action is chosen.
The value of a policy is measured by a utility function u(to be dened for the
task at hand) in a set of environments q2Q withV(;q) :=u(h(;q)) where
h(;q) generates the interaction history ao1:1of the policy in the determinis-
tic environment q. The value of a policy over the set Qof environments is dened
byV(;Q) :=P
q2Qp(q)V(;q), for a probability distribution pover the envi-
ronments. The optimal agent subject to the constraints of the architecture
Mis dened by := arg max 2MV(;Q).
Self-modifying resource-bounded universal intelligence
Although not explicitly stated, it seems reasonable to assume that Russell's def-
inition of bounded optimality also includes self-modiable policies, i.e., those
that can optimize every aspect of themselves, to be more ecient in both com-
putation time and memory space. (This is in fact the core idea behind the G odel
Machine [11].)
It is straightforward to merge Legg's intelligence measure with Russell's
bounded optimality, and the result is an optimal, self-modifying, resource-bounded
4A stochastic environment (o1:tja1:t) species the probability that the environment
produces observation sequence o1:tgiven action sequence a1:t. A reward-summable
environment ensures the agent's total cumulated lifetime reward does not exceed 1.
universal agent. To do so we rst dene a set of policies ~t;~lbased on a decom-
position of architecture Minto a reference machine U(like a universal Turing
machine),a time ~tof unitary computation steps per interaction, and a memory
space ~lthat contains both the source code of the agent and the usable memory.
In the remainder of the paper, we will consider only stochastic policies. A
self-modifying stochastic policy 2~t;~lat time step tdenes the probability
t(hat;t+1ijot1) of outputting: (1) some action at2Aat steptand (2) some
stochastic policy t+12~t;~lfor use by the agent at t+ 1; both conditioned
on the last observation ot1output by the environment, and this computation
must be done within ~tcomputation steps and ~lbits of memory (otherwise some
default values are output). The new code t+1might, for example, be the same
astor only slightly dierent, perhaps with ot1written somewhere in it.
The environment outputs an observation otwith a probability (otjaotat)
depending on the current interaction history (not taking into account the se-
quence of policies of the agent). For generality, is dened as a universal distri-
bution5like[15,3] (orRSabove), such that w()>0 for some prior weight
w() of any stochastic environment 2M andP
2Mw()1. We use a
utility function u(aot)2[0;1] that assigns a utility value to each interaction
history, whose cumulated value over the future is discounted by a horizon func-
tion
tso thatP1
t=1
t= 1 to ensure convergence (in RSthe horizon function
is considered to be a part of the environment).
The optimal self-modifying, resource-bounded, universal agent can now
be dened:6
:= arg max
12~t;~lV(1;)
V(t;aot) :=X
hat;t+1it(hat;t+1ijot1)
X
ot(otjaotat)h

tu(ao1:t) +V(t+1;ao1:t)i
:
This description shows that the optimal policy achieves greatest average weighted
discounted utility in all possible futures by (a) choosing good actions within the
time and space constraints ~tand~l, and (b) choosing good future policies for
itself (within the same time and space constraints).
5can be seen equivalently either as a single environment or a mixture of environ-
ments. The best way to think about it in the present case might be to consider
as auniversal semi-measure (because we, humans, have no absolute certainty about
what the true environment is), but biased with all the knowledge we can, or the
knowledge we think is relevant. Thinking of as a non-universal but accurate model
of the real world is also acceptable (although arguably non-realistic).
6This is an uncomputable denition, but the solution of this equation is a computable
optimal resource-bounded agent.
5 Embedded resource-bounded intelligence
The self-modifying, resource-bounded intelligence just described does not take
into account the fact that the environment may have read or even write access
to the memory space of the agent, containing its memory of the past and its
source code. We now propose a new denition of intelligence that extends self-
modifying, resource-bounded intelligence in two ways. The rst extension, which
we call space embedding (Section 5.1), moves the code and memory of the agent
into the environment. The second extension, which we call space-time embedding
(Section 5.2), allows the environment itself (rather than an external, possibly
innite computational device) to compute the agent's code.
5.1 Space-embedded agents
In the traditional Reinforcement Learning (RL) framework, the agent is external
to the environment [14,3]. It is immortal, and its resources are independent of
the resources of the environment. In the real world, agents are embedded in
the environment; i.e., they can be modied and even destroyed by it. Such
considerations were partially addressed in our denition of the optimal, self-
modifying, universal agent [5], whose source code was part of the environment
itself, both readable and modiable by it. A companion paper [6] considers the
consequences of doing the same with the agent's memory of the past (but not
its source code).
In this section, we consider space-embedded agents, whose code andmemory
are modiable by the environment. The space-embedded agent's code is calcu-
lated by an innite computational device (or oracle ) which yields the full results
of the computation immediately and independently of the machine that com-
putes the environment. However, the environment can modify the agent in its
entirety|both its memory and its code, which together dene the agent's policy .
At each time step, the space-embedded agent uses its current policy tto
produce a candidate next policy 0
t+1, which is passed to the environment. The
environment then produces the agent's actual next policy t+1, and the optimal
agent is therefore dened as:
:= arg max
12V(1;) (1)
V(t;aot) :=X
at=ha0
t;0
t+1it(atjo0
t1)X
ot=ho0
t;t+1i(otjaotat)h

tu(ao1:t) +V(t+1;ao1:t)i
(2)
where the time ~tand memory ~llimits are not considered for now. Note that
while Equation 2 shows the semantics of at=
a0
t;0
t+1
, neithera0
tnor0
t+1are
used in the equation. In fact, there is no need for an explicit action-observation
interaction protocol anymore: the environment can read and write any infor-
mation, including information about actions and observations, directly into the
agent's space t+1. Thus, Equation (2) can be rewritten in various equivalent
forms that have dierent interpretations:
V(t;aot) :=X
att(at)X
ot(otjaotat)h

tu(ao1:t) +V(ot;ao1:t)i
(3)
V(t;at) :=X
att(at)X
t+1(t+1ja1:t)h

tu(a1:tt+1) +V(t+1;a1:t)i
(4)
V(t;0
t) :=X
0
tt(0
t)X
t+1(t+1j01:t)h

tu(01:tt+1) +V(t+1;01:t)i
(5)
In Equation (3) the action-observation protocol has been removed; additionally,
t+1is shown as an implicit interpretation of otfrom the previous step. Equa-
tion (4) diers from Equation (3) in that the environment's output is always
interpreted as a policy. Equation (5) then also renames action aas0to empha-
size that the agent and the environment share the agent's memory space. In all
of these equations, the alphabets of the actions, observations, and policies are
considered to be the same.
It is interesting to note that when the environment contains the agent's code,
there is no RL agent that is asymptotically as good as every other agent in all
environments: for each agent , there is an environment qthat always gives
rt= 08tto that particular agent, and rt= 18tfor all other agents.
With the space-embedded framework, the agent can in principle make pre-
dictions about its source code and memory ( e.g., that it will be updated by
humans, who are part of the environment, to get new sensors or more ecient
code). By contrast, neither AIXI, nor AIXI ~t;~l, nor the G odel Machine can make
such predictions even in principle.7
5.2 Space-time-embedded agents
The space-embedded agent's next action is computed independently from the en-
vironment by a separate machine. Its code can be incomputable ( cf.AIXI [3]),
and, unless an explicit time limit ~tis introduced, is expected to run until com-
pletion (regardless how much computation might be involved) before the result
is passed to the environment. In the real-world though, the agent cannot have
more computing power or memory than what the environment has to oer.
To better model the interaction between the agent and the environment in
the real world, we examine the case where the reference machine of the agent
(i.e., the computer on which it runs), is a part of the environment, and the agent
iscomputed by the environment . Our proposal is not simply for the agent and
environment to be computed by the same reference machine, but to actually
make the agent be computed bythe environment itself.
7In the case of the G odel Machine, one could set up a special protocol whereby external
agents could propose a new source code to the machine, possibly along with a proof
that this leads to a better expected value. If the machine can verify the proof, it
could adopt the new code. But this is a restricted form of external modication.
One specic advantage of this model is that it allows the agent through its
actions (and predictions) to optimize not just its policy but also potentially the
physical device on which its policy is computed.8An additional advantage is
that there is no need anymore for a time parameter ~t, since it is the environment
that determines how much computation the agent is allowed.
An alteration to Equation (5) describes the agent's computation as performed
by(even ifis only an estimate of the true environment):
:= arg max
12V(1)
V(0
tt) :=X
0
t0(0
tj0
tt)
X
t+1(t+1j01:t)h

tu(01:tt+1) +V(01:tt+1)i
(6)
V(0
tt) :=X
0
tt+100(0
tt+1j00
tt)h

tu(01:tt+1) +V(01:tt+1)i
;(7)
where0is dened appropriately for 0, and00is the one-step combination of
and0. Loosely renaming 0
tt+1tot+1and00back to, we can interpret
Equation (7) as merging two interaction steps into a single one, and we obtain
the value of the space-time-embedded agent:
V(t) :=X
t(tjt)h

tu(1:t) +V(1:t)i
: (8)
Equation (8) has one remaining problem, which is that for t= 0, when nothing
yet is known about the environment, the optimal policy may have innite length.
Thus, we need to add a length constraint ~lon the initial length that the program
can have. It is a reasonable parameter for any real-world model. In our world
it corresponds to the maximum number of bits that we, as programmers, are
ready to use for the initial policy of the agent. Note, however, that after the very
rst step the actual length of the agent is determined by the computation of the
environment, i.e.,t;t > 1 need not be of size less than ~l. Therefore, the nal
denition of the optimal bounded-length space-time-embedded agent is:
:= arg max
12~lV(1)
V(t) :=X
t2(tjt)h

tu(1:t) +V(1:t)i:
Although simpler than Legg's denition, this equation has profound implications
regarding the nature of the agent. In particular, it precisely represents the goal
of those attempting to build an Articial General Intelligence in our world.
8Such eects can be neither predicted nor controlled by AIXI, AIXI ~t;~l, or the G odel
Machine using their current denition.
A Turing machine model. It is convenient to envision the space-time-embedded
environment as a multi-tape Turing machine with a special tape for the agent.
This tape is used by the environment just like any other working-memory tape.
The read and write heads need not be synchronized and there is no external
computing device, oracle or special interface for the agent. The agent's tape can
be seen as a partial internal state of the environment, which is consistent with
the intuition that agents do not have special status in the real world compared to
the rest of the environment. This view extends easily to a multi-agent framework.
A cellular-automaton survival agent. It is also instructive to envision the envi-
ronment as a cellular automaton ( e.g., the Game of Life [1]), in which an agent
is represented as a particular set of cells whose initial state is specied by the
initial policy 1(perhaps anp
~lp
~lsquare of cells). The cells surrounding the
agent, in possibly unbounded number, may be in any initial state.
As an example, consider a utility function whose value is 1 as long as some
critical part of the agent's policy maintains some particular pattern (call it the
agent's \heart"), and 0 otherwise. If the pattern can be destroyed by gliders9
coming from outside the initial square of the agent, then the agent must nd
ways to avoid them. The optimal initial policy is thus the one that maximizes
the expected number of time steps that the heart pattern survives. To ensure
its survival, the agent may need to learn about and react to its environment in
sophisticated and intelligent ways (provided ~lis large enough).
Note that while the utility and horizon functions are part of the denition
ofV, and thus are critical to dening , they are not necessarily represented
withinin any way. Note also that although 1is the entire initial state of
the agent, as soon as the agent and environment interact, the boundary between
them may quickly blur or disappear. Thus the notation tfort >2 may be
misleading, since it can also be viewed simply as a window onto that part of the
environment used (by the utility and horizon functions) to assign a value to the
agent. Since only the output of can be used by the utility function, can be
dened so that t;t> 2 encompasses the entire environment, while 1remains
limited to ~lbits. Thus, in the case of a cellular automaton, for example, the agent
and its heart pattern may drift away from its original cells; or, alternatively, the
utility function may seek to maximize the number of \live" cells ( i.e., cells set
to 1) in the entire environment.
New kinds of questions. The framework for space-time-embedded agents allows
formal discussion of a range of questions that could not previously be formulated
using the traditional RL framework.
Because the agent is computed by the environment, the agent's choices are
the result of the computations made by the environment on the bits dening the
agent's policy. Genuine choice is exercised only by those processes ( e.g., those
programmers) that dene the agent's initial program 1before interaction with
the environment begins. Yet the programmers are also part of the environment,
9A \glider" is a repeating pattern that can cross regions of the cellular automaton.
which implies that the agent is generated ultimately as a result of the initial
conditions of the environment in a generative process much like Solomono's
Sequence Prediction [12].
If at some time tsome set of bits implements an intelligent agent , one
might wonder by what process this agent was generated or, more precisely, what
are the most probable environments that could have generated this set of bits.
We do not seek to answer to this question here, but only to point out that the
question itself can be discussed within the framework of space-time-embedded
agents, in contrast to the traditional RL framework, in which the agent is not
generated but is simply assumed to exist from the very rst step.10
In fact, the framework now allows many questions to be discussed, such as:
Who is the agent? ( i.e., what part of the global computation denes the identity
of the agent; e.g.,1,u,
?) What is an agent? ( i.e., where is the boundary
between the agent and its environment?) What does it mean for an agent to
live and to die? (Questions that depend deeply on agent identity and thus the
boundary between the agent and the environment.)
To perform any computation, an embedded agent necessarily aects its en-
vironment, and thus the mere act of calculating the consequences of its own
actions implies a self-referential computation. It may seem we would therefore
need to dene agents that deal explicitly with the problem of self reference, but
our framework instead circumvents this issue entirely by simply computing each
agent's value subject to the environment's computational constraints, and then
selecting the agent with the highest value. This best agent may, or may not, deal
with self reference, but does so to whatever extent is optimal for its environment.
6 Discussion and conclusion
This paper has proposed a new denition of intelligence within a theoretical
framework more realistic than previous denitions. In both Legg's denition
of intelligence [4] (based on AIXI [3]) and Russell's bounded-optimality frame-
work [10] (which embraces time and space constraints on the program of the
agent), the agent is separated from the environment and therefore immortal.
Space-time-embedded intelligence formally describes agents as components of
their environment. Such agents are thus limited to the computational resources
(computation time and memory space) provided by their environment, can be
fully modied by their environment ( i.e., the agents are mortal), and are even
computed bythe environment, making their computation dependent on the dy-
namics of the environment. Compared with previous denitions of intelligence,
the resulting equation is surprisingly simple|deceptively simple, in fact, hiding
considerable depth beneath its surface.
Our primary motivation was to nd an equation of intelligence that, if solved,
might lead to the actual building of real-world, articially intelligent agents. We
believe that this new formalization brings us a step closer to our goal, though we
10Note, though, that in our companion paper [6] we address the issue of how to assess
the probability of some particular memory state generated by an environment.
are by no means naive as to the diculty of its solution. As much as anything
else, we greatly lack insight into (i.e., the universe in which we live). For
example, our universe allows for the sustainability of intelligent agents|patterns
that continue through large stretches of space and time (partly thanks to our
protective skulls). We hope this work will help us develop a better understanding
of the information we will need for eventually solving the equation.
When discussing narrow AI , our framework might be regarded as a step
backward compared to Russell's denition, since the interaction loop is hidden
in the embedding and makes reasoning about the actions of the agents less
explicit. But when discussing true AGI, our framework captures critical aspects
of reality that Russell's denition simplies away: agents are computed by and
can be modied by the environment, and are therefore mortal. Furthermore,
these theoretical consequences of inhabiting one's environment turn out to be
essential and practical considerations for all the intelligent agents who inhabit
our own.
References
1. Conway, J.: The game of life. Scientic American 303(6), 43{44 (1970)
2. Goertzel, B.: Toward a Formal Characterization of Real-World General Intelli-
gence. In: Proceedings of the 3d Conference on Articial General Intelligence (AGI-
10). pp. 19{24. Atlantis Press (2010)
3. Hutter, M.: Universal Articial Intelligence: Sequential Decisions based on Algo-
rithmic Probability. Springer, Berlin (2005)
4. Legg, S.: Machine Super Intelligence. Department of Informatics, University of
Lugano (2008)
5. Orseau, L., Ring, M.: Self-Modication and Mortality in Articial Agents. In: Ar-
ticial General Intelligence (AGI). pp. 1{10. LNAI, Springer (2011)
6. Orseau, L., Ring, M.: Memory Issues of Intelligent Agents (2012), Articial General
Intelligence (AGI)
7. Ortega, P.A., Braun, D.A.: Information, utility and bounded rationality. In: Arti-
cial General Intelligence (AGI). pp. 269{274. Springer, Berlin, Germany (2011)
8. Ring, M., Orseau, L.: Delusion, Survival, and Intelligent Agents. In: Articial Gen-
eral Intelligence (AGI). pp. 11{20. LNAI, Springer, Berlin, Heidelberg (2011)
9. Robinson, H.: Dualism. In: The Stanford Encyclopedia of Philosophy. Winter 2011
edn. (2011)
10. Russell, S.J., Subramanian, D.: Provably Bounded-Optimal Agents. Perspective 2,
575{609 (1995)
11. Schmidhuber, J.: Ultimate Cognition  a la G odel. Cognitive Computation 1(2),
177{193 (2009)
12. Solomono, R.J.: A Formal Theory of Inductive Inference. Part I. Information and
Control 7(1), 1{22 (1964)
13. Stoljar, D.: Physicalism. In: The Stanford Encyclopedia of Philosophy. Fall 2009
edn. (2009)
14. Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cam-
bridge, MA (1998)
15. Zvonkin, A K and Levin, L.A.: The complexity of nite objects and the develop-
ment of the concepts of information and randomness by means of the theory of
algorithms. Russian Mathematical Surveys 25(6), 83{124 (1970)