Nonparametric General
Reinforcement Learning
Jan Leike
A thesis submitted for the degree of
Doctor of Philosophy
at the
Australian National University
November 2016

©Jan Leike
This work is licensed under the
Creative Commons Attribution 4.0 International License

No reinforcement learners were harmed in the making of this thesis.
Except where otherwise indicated, this thesis is my own original work.
Jan Leike
28 November 2016

Acknowledgements
There are many without whom this thesis would not have been possible. I sincerely
hope that this page is not the way they learn how grateful I am to them. I thank in
particular ...
... ﬁrst and foremost, Marcus Hutter: he is an amazing supervisor; always very sup-
portive of my (unusual) endeavors, spent countless hours reading my drafts with
a impressive attention to detail. I am also grateful to him for forcing me to be
absolutely rigorous in my mathematical arguments, and, of course, for developing
the theory of universal AI without which this thesis would not have existed. I could
not have picked a better supervisor.
... the Australian National University for granting me scholarships that let me pursue
my academic interests unrestricted and without any ﬁnancial worries.
... Csaba Szepesvári and the University of Alberta for hosting me for three months.
... Matthias Heizmann and the University of Freiburg for hosting me while I was
traveling in Europe.
... the Machine Intelligence Research Institute for enabling me to run MIRIx research
workshops.
... CCR, UAI, Google DeepMind, ARC, MIRI, and FHI for supporting my travel.
... Tor Lattimore for numerous explanations, discussions, and pointers that left me
with a much deeper understanding of the theory of reinforcement learning.
... Laurent Orseau for interesting discussions, encouragement, and for sharing so many
intriguing ideas.
... my fellow students: Mayank Daswani, Tom Everitt, Daniel Filan, Roshan Shariﬀ,
Tian Kruger, Emily Cutts Worthington, Buck Shlegeris, Jarryd Martin, John
Aslanides, Alexander Mascolo, and Sultan Javed for so many interesting discus-
sions and for being awesome friends. I especially thank Daniel, Emily, Mayank,
and Buck for encouraging me to read more of Less Wrong and Slate Star Codex.
... ToscaLechnerforstudyingstatisticswithmedespitesomanyschedulingdiﬃculties
across all these time zones.
... Tom Sterkenburg, Christian Kamm, Alexandra Surdina, Freya Fleckenstein, Pe-
ter Sunehag, Tosca Lechner, Ines Nikolaus, Laurent Orseau, John Aslanides, and
especially Daniel Filan for proofreading parts of this thesis.
... the CSSA for being a lovely bunch that made my stay in Australia feel less isolated.
... my family for lots of love and support, and for tolerating my long absences from
Europe.

Abstract
Reinforcement learning problems are often phrased in terms of Markov decision pro-
cesses (MDPs). In this thesis we go beyond MDPs and consider reinforcement learning
in environments that are non-Markovian, non-ergodic and only partially observable.
Our focus is not on practical algorithms, but rather on the fundamental underlying
problems: How do we balance exploration and exploitation? How do we explore opti-
mally? When is an agent optimal? We follow the nonparametric realizable paradigm:
weassumethedataisdrawnfromanunknownsourcethatbelongstoaknowncountable
class of candidates.
First, we consider the passive (sequence prediction) setting, learning from data that
is not independent and identically distributed. We collect results from artiﬁcial intelli-
gence,algorithmicinformationtheory,andgametheoryandputtheminareinforcement
learning context: they demonstrate how an agent can learn the value of its own policy.
Next, we establish negative results on Bayesian reinforcement learning agents, in
particular AIXI. We show that unlucky or adversarial choices of the prior cause the
agent to misbehave drastically. Therefore Legg-Hutter intelligence and balanced Pareto
optimality, which depend crucially on the choice of the prior, are entirely subjective.
Moreover, in the class of all computable environments every policy is Pareto optimal.
This undermines all existing optimality properties for AIXI.
However, there are Bayesian approaches to general reinforcement learning that sat-
isfy objective optimality guarantees: We prove that Thompson sampling is asymptot-
ically optimal in stochastic environments in the sense that its value converges to the
value of the optimal policy. We connect asymptotic optimality to regret given a recov-
erability assumption on the environment that allows the agent to recover from mistakes.
Hence Thompson sampling achieves sublinear regret in these environments.
AIXI is known to be incomputable. We quantify this using the arithmetical hierar-
chy, and establish upper and corresponding lower bounds for incomputability. Further,
we show that AIXI is not limit computable, thus cannot be approximated using ﬁnite
computation. However there are limit computable "-optimal approximations to AIXI.
We also derive computability bounds for knowledge-seeking agents, and give a limit
computable weakly asymptotically optimal reinforcement learning agent.
Finally, our results culminate in a formal solution to the grain of truth problem: A
Bayesian agent acting in a multi-agent environment learns to predict the other agents’
policies if its prior assigns positive probability to them (the prior contains a grain of
truth). We construct a large but limit computable class containing a grain of truth and
show that agents based on Thompson sampling over this class converge to play "-Nash
equilibria in arbitrary unknown computable multi-agent environments.
ix
x
Keywords. Bayesian methods, sequence prediction, merging, general reinforcement
learning, universal artiﬁcial intelligence, AIXI, Thompson sampling, knowledge-seeking
agents, Pareto optimality, intelligence, asymptotic optimality, computability, reﬂective
oracle, grain of truth problem, Nash equilibrium.
Contents
Title Page i
Abstract ix
Contents xiii
List of Figures xv
List of Tables xvii
1 Introduction 1
1.1 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Narrow Reinforcement Learning . . . . . . . . . . . . . . . . . . . 3
1.1.2 Deep Q-Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.3 General Reinforcement Learning . . . . . . . . . . . . . . . . . . 6
1.2 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 Preliminaries 15
2.1 Measure Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2 Stochastic Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Information Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4 Algorithmic Information Theory . . . . . . . . . . . . . . . . . . . . . . 20
3 Learning 23
3.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2 Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.3 Martingales . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.4 Merging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.4.1 Strong Merging . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.4.2 Weak Merging . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.4.3 Almost Weak Merging . . . . . . . . . . . . . . . . . . . . . . . . 34
3.5 Predicting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.5.1 Dominance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.5.2 Absolute Continuity . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.5.3 Dominance with Coeﬃcients . . . . . . . . . . . . . . . . . . . . . 40
3.6 Learning with Algorithmic Information Theory . . . . . . . . . . . . . . 41
3.6.1 Solomonoﬀ Induction . . . . . . . . . . . . . . . . . . . . . . . . . 41
xi
xii Contents
3.6.2 The Speed Prior . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.6.3 Universal Compression . . . . . . . . . . . . . . . . . . . . . . . . 43
3.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4 Acting 49
4.1 The General Reinforcement Learning Problem . . . . . . . . . . . . . . . 50
4.1.1 Discounting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.1.2 Implicit Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.1.3 Typical Environment Classes . . . . . . . . . . . . . . . . . . . . 54
4.2 The Value Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.2.1 Optimal Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.2.2 Properties of the Value Function . . . . . . . . . . . . . . . . . . 58
4.2.3 On-Policy Value Convergence . . . . . . . . . . . . . . . . . . . . 59
4.3 The Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.3.1 Bayes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.3.2 Knowledge-Seeking Agents . . . . . . . . . . . . . . . . . . . . . . 63
4.3.3 BayesExp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.3.4 Thompson Sampling . . . . . . . . . . . . . . . . . . . . . . . . . 65
5 Optimality 67
5.1 Pareto Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.2 Bad Priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.2.1 The Indiﬀerence Prior . . . . . . . . . . . . . . . . . . . . . . . . 70
5.2.2 The Dogmatic Prior . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.2.3 The Gödel Prior . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.3 Bayes Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.4 Asymptotic Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.4.1 Bayes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
5.4.2 BayesExp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.4.3 Thompson Sampling . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.4.4 Almost Sure in Cesàro Average vs. in Mean . . . . . . . . . . . . 89
5.5 Regret . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
5.5.1 Sublinear Regret in Recoverable Environments . . . . . . . . . . 91
5.5.2 Regret of the Optimal Policy and Thompson sampling . . . . . . 95
5.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.6.1 The Optimality of AIXI . . . . . . . . . . . . . . . . . . . . . . . 96
5.6.2 Natural Universal Turing Machines . . . . . . . . . . . . . . . . . 97
5.6.3 Asymptotic Optimality . . . . . . . . . . . . . . . . . . . . . . . . 98
5.6.4 The Quest for Optimality . . . . . . . . . . . . . . . . . . . . . . 99
6 Computability 101
6.1 Background on Computability . . . . . . . . . . . . . . . . . . . . . . . . 103
6.1.1 The Arithmetical Hierarchy . . . . . . . . . . . . . . . . . . . . . 103
6.1.2 Computability of Real-valued Functions . . . . . . . . . . . . . . 103
Contents xiii
6.2 The Complexity of Solomonoﬀ Induction . . . . . . . . . . . . . . . . . . 105
6.3 The Complexity of AINU, AIMU, and AIXI . . . . . . . . . . . . . . . . 108
6.3.1 Upper Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.3.2 Lower Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6.4 Iterative Value Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6.5 The Complexity of Knowledge-Seeking . . . . . . . . . . . . . . . . . . . 122
6.6 A Limit Computable Weakly Asymptotically Optimal Agent . . . . . . . 122
6.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
7 The Grain of Truth Problem 127
7.1 Reﬂective Oracles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
7.1.1 Deﬁnition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
7.1.2 A Limit Computable Reﬂective Oracle . . . . . . . . . . . . . . . 131
7.1.3 Proof of Theorem 7.7 . . . . . . . . . . . . . . . . . . . . . . . . . 132
7.2 A Grain of Truth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7.2.1 Reﬂective Bayesian Agents . . . . . . . . . . . . . . . . . . . . . 135
7.2.2 Reﬂective-Oracle-Computable Policies . . . . . . . . . . . . . . . 136
7.2.3 Solution to the Grain of Truth Problem . . . . . . . . . . . . . . 137
7.3 Multi-Agent Environments . . . . . . . . . . . . . . . . . . . . . . . . . . 137
7.4 Informed Reﬂective Agents . . . . . . . . . . . . . . . . . . . . . . . . . 140
7.5 Learning Reﬂective Agents . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.6 Impossibility Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
7.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
8 Conclusion 147
Measures and Martingales 151
Bibliography 155
List of Notation 171
Index 175
xiv Contents
List of Figures
1.1 Selection of Atari 2600 video games . . . . . . . . . . . . . . . . . . . . . 13
3.1 Properties of learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.1 The dualistic agent model . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5.1 Legg-Hutter intelligence measure . . . . . . . . . . . . . . . . . . . . . . 77
5.2 Relationship between diﬀerent types of asymptotic optimality . . . . . . 80
6.1 Deﬁnition of conditional Mas a0
2-formula . . . . . . . . . . . . . . . . 106
6.2 Environment from the proof of Theorem 6.15 . . . . . . . . . . . . . . . 111
6.3 Environment from the proof of Theorem 6.16 . . . . . . . . . . . . . . . 113
6.4 Environment from the proof of Theorem 6.17 . . . . . . . . . . . . . . . 114
6.5 Environment from the proof of Proposition 6.19 . . . . . . . . . . . . . . 117
6.6 Environment from the proof of Theorem 6.22 . . . . . . . . . . . . . . . 119
6.7 Environment from the proof of Theorem 6.23 . . . . . . . . . . . . . . . 121
7.1 Answer options of a reﬂective oracle . . . . . . . . . . . . . . . . . . . . 131
7.2 The multi-agent model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
xv
xvi LIST OF FIGURES
List of Tables
1.1 Assumptions in reinforcement learning . . . . . . . . . . . . . . . . . . . 7
1.2 List of publications by chapter . . . . . . . . . . . . . . . . . . . . . . . 11
1.3 List of publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1 Examples of learning distributions . . . . . . . . . . . . . . . . . . . . . 45
3.2 Summary on properties of learning . . . . . . . . . . . . . . . . . . . . . 45
4.1 Discount functions and their eﬀective horizons . . . . . . . . . . . . . . . 53
5.1 Types of asymptotic optimality . . . . . . . . . . . . . . . . . . . . . . . 79
5.2 Compiler sizes of the UTMs of bad priors . . . . . . . . . . . . . . . . . 98
5.3 Notions of optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.1 Computability results on Solomonoﬀ’s prior . . . . . . . . . . . . . . . . 102
6.2 Computability results for diﬀerent agent models . . . . . . . . . . . . . . 102
6.3 Computability of real-valued functions . . . . . . . . . . . . . . . . . . . 104
6.4 Computability results for the iterative value function . . . . . . . . . . . 116
7.1 Terminology dictionary between reinforcement learning and game theory. 128
xvii
xviii LIST OF TABLES
Chapter 1
Introduction
Everything I did was for the glamor, the money, and the sex. — Albert Einstein
After the early enthusiastic decades, research in artiﬁcial intelligence (AI) now mainly
aims at speciﬁc domains: playing games, mining data, processing natural language,
recognizing objects in images, piloting robots, ﬁltering email, and many others (Russell
and Norvig, 2010). Progress on particular domains has been remarkable, with several
high-proﬁle breakthroughs: The chess world champion Garry Kasparov was defeated
by the computer program Deep Blue in 1997 (IBM, 2012a). In 2011 the world’s best
Jeopardy! players were defeated by the computer program Watson (IBM, 2012b). As
of 2014 Google’s self-driving cars completed over a million kilometers autonomously on
public roads (Google, 2014). Finally, in 2016 Google DeepMind’s AlphaGo beat Lee
Sedol, one of the world’s best players, at the board game Go (Google, 2016).
While these advancements are very impressive, they are highly-specialized algo-
rithms tailored to their domain of expertise. Outside that domain these algorithms
perform very poorly: AlphaGo cannot play chess, Watson cannot drive a car, and
DeepBlue cannot answer natural language queries. Solutions in one domain typically
do not generalize to other domains and no single algorithm performs well in more than
one of them. We classify these kinds of algorithms as narrow AI .
This thesis is not about narrow AI. We expect progress on narrow AI to continue
and even accelerate, taking the crown of human superiority in domain after domain.
But this is not the ultimate goal of artiﬁcial intelligence research. The ultimate goal is
to engineer a mind—to build a machine that can learn to do all tasks that humans can
do, at least as well as humans do them. We call such a machine human-level AI (HLAI)
if it performs at human level and strong AI if it surpasses human level. This thesis is
about strong AI.
The goal of developing HLAI has a long tradition in AI research and was explicitly
part of the 1956 Dartmouth conference that gave birth to the ﬁeld of AI (McCarthy
et al., 1955):
We propose that a 2 month, 10 man study of artiﬁcial intelligence be carried
out during the summer of 1956 at Dartmouth College in Hanover, New
Hampshire. The study is to proceed on the basis of the conjecture that
every aspect of learning or any other feature of intelligence can in principle
be so precisely described that a machine can be made to simulate it. An
1
2 Introduction
attempt will be made to ﬁnd how to make machines use language, form
abstractionsand concepts, solve kinds ofproblems now reserved forhumans,
and improve themselves. We think that a signiﬁcant advance can be made
in one or more of these problems if a carefully selected group of scientists
work on it together for a summer.
Inhindsightthisproposalreadsvastlyoverconﬁdent,anddisappointmentwasinevitable.
Making progress on these problems turned out to be a lot harder than promised, and
over the last decades any discussion of research targeting HLAI has been avoided by
serious researchers in the ﬁeld. This void was ﬁlled mostly by crackpots, which tainted
the reputation of HLAI research even further. However, this trend has recently been re-
verted: Chalmers (2010), Hutter (2012a), Schmidhuber (2012), Bostrom (2014), Hawk-
ing, Tegmark, Russell, and Wilczek (2014), Shanahan (2015), and Walsh (2016) are
well-known scientists discussing the prospect of HLAI seriously. Even more: the ex-
plicit motto of Google DeepMind, one of today’s leading AI research centers, is to “solve
intelligence.”
1.1 Reinforcement Learning
The best formal model for strong AI we currently have is reinforcement learning (RL).
Reinforcement learning studies algorithms that learn to act in an unknown environment
through trial and error (Sutton and Barto, 1998; Szepesvári, 2010; Wiering and van
Otterlo, 2012). Without knowing the structure of the environments or the goal, an
agent has to learn what to do through the carrot-and-stick approach: it receives a
reward in form of a numeric feedback signifying how well it is currently doing; from
this signal the agent has to ﬁgure out autonomously what to do. More speciﬁcally,
in ageneral reinforcement learning problem anagentinteracts sequentially with an
unknown environment : in every time step the agent chooses an actionand receives a
perceptconsisting of an observation and a real-valued reward. The sequence of past
actions and percepts is the history. The goal in reinforcement learning is to maximize
cumulative (discounted) rewards (this setup is described formally in Section 4.1).
A central problem in reinforcement learning is the balance between exploration and
exploitation : should the agent harvest rewards in the regions of the environment that it
currently knows (exploitation) or try discovering more proﬁtable regions (exploration)?
Exploration is costly and dangerous: it forfeits rewards that could be had right now,
and it might lead into traps from which the agent cannot recover. However, exploration
may pay oﬀ in the long run. Generally, it is not clear how to make this tradeoﬀ (see
Section 5.6).
Reinforcement learning algorithms can be categorized by whether they learn on-
policyoroﬀ-policy . Learning on-policy means learning the value of the policy that
the agent currently follows. Typically, the policy is slowly improved while learning,
likeSARSA(Sutton and Barto, 1998). In contrast, learning oﬀ-policy means following
one policy but learning the value of another policy (typically the optimal policy), like
Q-learning (Watkins and Dayan, 1992). Oﬀ-policy methods are more diﬃcult to handle
§1.1Reinforcement Learning 3
in practice (see the discussion on function approximation below) but tend to be more
data-eﬃcient since samples from an old policy do not have to be discarded.
Reinforcement learning has to be distinguished from planning . In a planning prob-
lem we are provided with the true environment and are tasked with ﬁnding an optimal
policy. Mathematically it is clear what the optimal policy is, the diﬃculty stems from
ﬁnding a reasonable solution with limited computation. Reinforcement learning is fun-
damentally more diﬃcult because the true environment is unknown and has to be
learned from observation. This enables two approaches: we could learn a model of the
trueenvironmentandthenuseplanningtechniqueswithinthatmodel; thisisthe model-
basedapproach. Alternatively, we could learn an optimal policy directly or through an
intermediate quantity (typically the value function); this is the model-free approach.
Model-based methods tend to be more data-eﬃcient but also computationally more
expensive. Therefore most algorithms used in practice ( Q-learning and SARSA) are
model-free.
1.1.1 Narrow Reinforcement Learning
In the reinforcement learning literature it is typically assumed that the environment is a
Markov decision process (MDP), i.e., the next percept only depends on the last percept
and action and is independent of the rest of the history (see Section 4.1.3). In an MDP,
percepts are usually called states. This setting is well-analyzed (Puterman, 2014; Bert-
sekas and Tsitsiklis, 1995; Sutton and Barto, 1998), and there is a variety of algorithms
that are known to learn the MDP asymptotically, such as TD learning (Sutton, 1988)
andQ-learning (Watkins and Dayan, 1992).
Moreover, for MDPs various learning guarantees have been proved in the literature.
First, there are bounds on the agent’s regret, the diﬀerence between the obtained re-
wards and the rewards of the optimal policy. Auer et al. (2009) derive the regret bound
~O(dSp
At)for ergodic MDPs where dis thediameter of the MDP (how many steps
a policy needs on average to get from one state of the MDP to any other), Sis the
number of states, Ais the number of actions, and tis the number of time steps the
algorithm runs. Second, given "and, a reinforcement learning algorithm is said to
havesample complexity C(";)iﬀ it is"-suboptimal for at most C(";)time steps with
probability at least 1(probably approximately correct, PAC). For MDPs the ﬁrst
sample complexity bounds were due to Kakade (2003). Lattimore and Hutter (2012)
use the algorithm UCRL
(Auer et al., 2009) with geometric discounting with discount
rate
and derive the currently best-known PAC bound of ~O(T=("2(1
)3) log)
whereTis the number of non-zero transitions in the MDP.
Typically, algorithms for MDPs rely on visiting every state multiple times (or even
inﬁnitelyoften), whichbecomesinfeasibleforlargestatespaces(e.g.avideogamescreen
consisting of millions of pixels). In these cases, function approximation can be used to
learn an approximation to the value function (Sutton and Barto, 1998). Linear function
approximationisknowntoconvergeforseveralon-policyalgorithms(TsitsiklisandRoy,
1997; Sutton, 1988; Gordon, 2001), but proved tricky for oﬀ-policy algorithms (Baird,
1995). A recent breakthrough was made by Mahmood et al. (2015) and Yu (2015)
4 Introduction
with their emphatic TD algorithm that converges oﬀ-policy. For nonlinear function
approximation no convergence guarantee is known.
Among the historical successes of reinforcement learning is autonomous helicopter
piloting (Kim et al., 2003) and TD-Gammon, a backgammon algorithm that learned
through self-play (Tesauro, 1995), similar to AlphaGo (Silver et al., 2016).
1.1.2 Deep Q-Networks
The current state of the art in reinforcement learning challenges itself to playing simple
video games. Video games are an excellent benchmark because they come readily with
the reward structure provided: the agent’s rewards are the change in the game score.
Without prior knowledge of any aspect of the game, the agent needs to learn to score
as many points in the game as possible from looking only at raw pixel data (sometimes
after some preprocessing).
This approach to general AI is in accordance with the deﬁnition of intelligence given
by Legg and Hutter (2007b):
Intelligence measures an agent’s ability to achieve goals in a wide range of
environments.
In reinforcement learning the deﬁnition of the goal is very ﬂexible, and provided by
the rewards. Moreover, a diverse selection of video games arguably constitutes a ‘wide
range of environments.’
A popular such selection is the Atari 2600 video game console (Bellemare et al.,
2013). There are hundreds of games released for this platform, with very diverse chal-
lenges: top-down shooting games such as Space Invaders, ball games such as Pong,
agility-based games such as Boxing or Gopher, tactical games such as Ms. Pac-Man,
and maze games such as Montezuma’s Revenge. An overview over some of the games
is given in Figure 1.1 on page 13.
Mnih et al. (2013, 2015) introduce the deep Q-network (DQN) algorithm, com-
biningQ-learning with nonlinear function approximation through convolutional neural
networks. DQN achieves 75% of the performance of a human game tester on 29 of 49
Atari games. The two innovations that made this breakthrough possible are (1) using
a not so recent target Q-function in the TD update and (2) experience replay. For ex-
perience replay, a set of recent state transitions is retained and the network is regularly
retrained on random samples from these old transitions.1
DQN rides the wave of success of deep learning (LeCun et al., 2015; Schmidhuber,
2015; Goodfellow et al., 2016). Deep learning refers to the training of artiﬁcial neural
networks with several layers. This allows them to automatically learn higher-level
abstractions from data. Deep neural networks are conceptionally simple and have been
studied since the inception of AI; only recently has computation power become cheap
enough to train them eﬀectively. Recently deep neural networks have taken the top of
the machine learning benchmarks by storm (LeCun et al., 2015, and references therein):
1The slogan for experience replay should be ‘regularly retrained randomly on retained rewards’.
§1.1Reinforcement Learning 5
These methods have dramatically improved the state-of-the-art in speech
recognition, visual object recognition, object detection and many other do-
mains such as drug discovery and genomics.
Since the introduction of DQN there have been numerous improvements on this
algorithm: increasing the gap on the Q-values of diﬀerent actions (Bellemare et al.,
2016), training in parallel (Nair et al., 2015; Mnih et al., 2016), improvements to the
experience replay mechanism (Schaul et al., 2016), generalization to continuous action
spaces (Lillicrap et al., 2016), solve the overestimation problem (van Hasselt et al.,
2016), andimprovementstotheneuralnetworkarchitecture(Wangetal.,2016). The Q-
values learned by DQN’s neural networks are intransparent to inspection; Zahavy et al.
(2016) use visualization techniques on the Q-value networks. Finally, Liang et al. (2016)
managed to reproduce DQN’s success using only linearfunction approximation (no
neural networks). The key is a selection of features similar to the ones produced by
DQN’s convolutional neural networks.
Regardless of its success, the DQN algorithm fundamentally falls short of the re-
quirements for strong AI: Q-learning with function approximation is targeted at solving
large-state (fully observable) Markov decision processes. In particular, it does not ad-
dress the following challenges.
•Partial observability. All games in the ATARI framework are fully observable (ex-
cept for Montezuma’s revenge): all information relevant to the state of the game
is visible on the screen at all times (when using the four most recent frames).
However, the real world is only partially observable. For example, when going
to the supermarket you have to remember what you wanted to buy because you
currently cannot observe which items you are missing at home. A strong AI needs
tohavememoryandbeabletorememberthingsthathappenedinthepast(rather
than only learning from it).
An obvious approach to equip DQN with memory is to use recurrent neural
networks instead of simple feedforward neural networks (Heess et al., 2015).
Hausknecht and Stone (2015) show that this enables the agent to play the games
when using only a single frame as input. However, it is currently unclear whether
recurrent neural networks are powerful enough to learn long-term dependencies
in the data (Bengio et al., 1994).
•Directed exploration. DQN fails in games with delayed rewards. For example, in
Montezuma’s Revenge the agent needs to avoid several obstacles to get to a key
before receiving the ﬁrst reward. DQN fails to score any rewards in this environ-
ment. This is not surprising: the typical approach for reinforcement learning, to
use"-exploration for which the agent chooses actions at random with a certain
probability, is insuﬃcient for exploring complex environments; the probability of
random walking into the ﬁrst reward is just too low.
Instead we need a more targeted exploration approach that aims at understanding
the environment in a structured manner. Theoretical foundations are provided
6 Introduction
byknowledge-seeking agents (Orseau, 2011, 2014a; Orseau et al., 2013). Kulkarni
et al. (2016) introduce a hierarchical approach based on intrinsic motivation to
improve DQN’s exploration and manage to score points in Montezuma’s Revenge.
However, their approach relies on quite a bit of visual preprocessing and domain
knowledge.
•Non-ergodicity. When losing in an Atari game, the agent always gets to play the
same game again. From the agent’s perspective, it has not actually failed, it just
gets transported back to the starting state. Because of this, there are no strong
incentives to be careful when exploring the environment: there can be no bad
mistakes that make recovery impossible.
However, in the real world some actions are irreversibly bad. If the robot drives
oﬀ a cliﬀ it can be fatally damaged and cannot learn from the mistake. The real
world is full of potentially fatal mistakes (e.g. crossing the street at the wrong
time) and for humans, natural reﬂexes and training by society make sure that we
are very conﬁdent of what situations to avert. This is crucial, as some mistakes
must be avoided without any training examples. Current reinforcement learning
algorithms only learn about bad states by visiting them.
•Wireheading. The goal of reinforcement learning is to maximize rewards. When
playing a video game the most eﬃcient way to get rewards is to increase the
game score. However, when a reinforcement learning algorithm is acting in the
real world, theoretically it can change its own hard- and software. In this set-
ting, the most eﬃcient way to get rewards is to modify the reward mechanism to
always provide the maximal reward (Omohundro, 2008; Ring and Orseau, 2011;
Bostrom, 2014). Consequently the agent no longer pursues the designers’ origi-
nally intended goals and instead only attempts to protect its own existence. The
namewireheading was established by analogy to a biology experiment by Olds
and Milner (1954) in which rats had a wire embedded into the reward center of
their brain that they could then stimulate by the push of a button.
Today’s reinforcement learning algorithms usually do not have access to their own
internal workings, but more importantly they are not smart enough to understand
their own architecture. They simply lack the capability to wirehead. But as we
increase their capability, wireheading will increasingly become a challenge for
reinforcement learning.
1.1.3 General Reinforcement Learning
A theory of strong AI cannot make some of the typical assumptions. Environments
are partially observable, so we are dealing with partially observable Markov decision
processes (POMDPs). The POMDP’s state space does not need to be ﬁnite. Moreover,
the environment may not allow recovery from mistakes: we do not assume ergodic-
ity or weak communication (not every POMDP state has to be reachable from every
other state). So in general, our environments are inﬁnite-state non-ergodic POMDPs.
Table 1.1 lists the assumptions that are typical but we do not make.
§1.1Reinforcement Learning 7
Assumption Description
Full observability the agent needs no memory to act optimally
Finite state the environment has only ﬁnitely many states
Ergodicity the agent can recover from any mistakes
Computability the environment is computable
Table 1.1: List of assumptions from the reinforcement learning literature. In this
thesis, we only make the computability assumption which is important for Chapter 6
and Chapter 7.
Learning POMDPs is a lot harder, and only partially successful attempts have
been made: through predictive state representations (Singh et al., 2003, 2004), and
Bayesian methods (Doshi-Velez, 2012). A general approach is feature reinforcement
learning (Hutter, 2009c,d), which aims to reduce the general reinforcement learning
problem to an MDP by aggregating histories into states. The quest for a good cost
function for feature maps remains unsuccessful thus far (Sunehag and Hutter, 2010;
Daswani, 2015). However, Hutter (2014) managed to derive strong bounds relating the
optimal value function of the aggregated MDP to the value function of the original
process even if the latter violates the Markov condition.
A full theoretical approach to the general reinforcement learning problem is given by
Hutter (2000, 2001a, 2002a, 2003, 2005, 2007a, 2012b). He introduces the Bayesian RL
agentAIXIbuilding on the theory of sequence prediction by Solomonoﬀ (1964, 1978).
Based in algorithmic information theory, Solomonoﬀ’s prior draws from famous insights
by William of Ockham, Sextus Epicurus, Alan Turing, and Andrey Kolmogorov (Rath-
mannerandHutter,2011). AIXIusesSolomonoﬀ’spriorovertheclassofallcomputable
environments and acts to maximize Bayes-expected rewards. We formally introduce
Solomonoﬀ’s theory of induction in Chapter 3 and AIXI in Section 4.3.1. See also Legg
(2008) for an accessible introduction to AIXI.
A typical optimality property in general reinforcement learning is asymptotic opti-
mality(Lattimore and Hutter, 2011): as time progresses the agent converges to achieve
the same rewards as the optimal policy. Asymptotic optimality is usually what is
meant by “ Q-learning converges” (Watkins and Dayan, 1992) or “TD learning con-
verges” (Sutton, 1988). Orseau (2010, 2013) showed that AIXI is not asymptotically
optimal. Yet asymptotic optimality in the general setting can be achieved through op-
timism (Sunehag and Hutter, 2012a,b, 2015), Thompson sampling (Section 5.4.3), or
an extra exploration component on top of AIXI (Lattimore, 2013, Ch. 5).
In our setting, learning the environment does not just involve learning a ﬁxed ﬁnite
set of parameters; the real world is too complicated to ﬁt into a template. Therefore we
fall back on the nonparametric approach where we start with an inﬁnite but countable
class of candidate environments. Our only assumption is that the true environment is
contained in this class (the realizable case ). As long as this class of environments is
large enough (such as for the class of all computable environments), this assumption is
8 Introduction
rather weak.
1.2 Contribution
The goal of this thesis is not to increase AI capability. As such, we are not trying to
improve on the state of the art, and we are not trying to derive practical algorithms.
Instead, the emphasis of this thesis is to further our understanding of general rein-
forcement learning and thus strong AI. How a future implementation of strong AI will
actually work is in the realm of speculation at this time. Therefore we should make as
few and as weak assumptions as possible.
We disregard computational constraints in order to focus on the fundamental un-
derlying problems. This is unrealistic, of course. With unlimited computation power
many traditional AI problems become trivial: playing chess, Go, or backgammon can
be solved by exhaustive expansion of the game tree. But the general RL problem does
not become trivial: the agent has to learn the environment and balance between ex-
ploration and exploitation. That being said, the algorithms that we study do have
a relationship with algorithms being used in practice and our results can and should
educate implementation.
On a high level, our insights can be viewed from three diﬀerent perspectives.
•Philosophically. Concisely, our understanding of strong AI can be summarized as
follows.
intelligence =learning +acting (1.1)
Here,intelligence refers to an agent that optimizes towards some goal in accor-
dancewiththe deﬁnition byLegg and Hutter(2007b). For learning we distinguish
two (very related) aspects: (1) arriving at accurate beliefs about the future and
(2) making accurate predictions about the future. Of course, the former implies
the latter: if you have accurate beliefs, then you can also make good predictions.
For RL accurate beliefs is what we care about because they enable us to plan
for the future. Learning is a passive process that only observes the data and
does not interfere with its generation. In particular, learning does not require
a goal. With actingwe mean the selection of actions in pursuit of some goal.
This goal can be reward maximization as in reinforcement learning, understand-
ing the environment as for knowledge-seeking agents, or something else entirely.
Together they enable an agent to learn the environment’s behavior in response to
itself (on-policy learning) and to choose a policy that furthers its goal. We dis-
cuss the formal aspects of learning in Chapter 3 and some approaches to acting
in Chapter 4.
Given inﬁnite computational resources, learning is easy and Solomonoﬀ induc-
tion provides a complete theoretical solution. However, acting is not straightfor-
ward. We show that in contrast to popular belief, AIXI, the natural extension of
Solomonoﬀ induction to reinforcement learning, does not provide the objectively
best answer to this question. We discuss some alternatives and their problems in
§1.2Contribution 9
Chapter 5. Unfortunately, the general question of how to act optimally remains
open.
AIXItl(Hutter, 2005, Ch. 7.2) is often mentioned as a computable approximation
to AIXI. But AIXI tldoes not converge to AIXI in the limit. Inspired by Hutter
search (Hutter, 2002b), it relies on an automated theorem prover to ﬁnd the
provably best policy computable in time twith a program of length l. In
contrast to AIXI, which only requires the choice of universal Turing machine,
proof search requires an axiom system that must not be too weak or too strong.
In Section 5.2.3 we discuss some of the problems with AIXI tl. Moreover, in
Corollary 6.13 we show that "-optimal AIXI is limit computable, which shows
that AIXI can be computably approximated by running this algorithm for a ﬁxed
number of time steps or until a timeout is reached. While neither AIXI tlnor
this AIXI approximation algorithm are practically feasible, the latter is a better
example for a computable strong AI.
In our view, AIXI should be taken as a descriptive rather than prescriptive model.
It is descriptive as an abstraction from an actual implementation of strong AI
where we ignore all the details of the learning algorithm and the computational
approximations of choosing how to act. It should not be viewed as a prescription
of how strong AI should be built and AIXI approximations (Veness et al., 2011,
2015) are easily outperformed by neural-network-based approaches (Mnih et al.,
2015).
•Mathematically. Some of the proof techniques we employ are novel and could
be used to analyze other algorithms. Examples include the proofs for the lower
bounds on the computability results (Section 6.3.2) and to a lesser extent the
upper bounds (Section 6.3.1), which should work analogously for a wide range of
algorithms. Furthermore, the proof of the asymptotic optimality of Thompson
sampling (Theorem 5.25) brings together a variety of mathematical tools from
measure theory, probability theory, and stochastic processes.
Next, the recoverability assumption (Deﬁnition 5.31) is a novel technical assump-
tionontheenvironmentakintoergodicityandweakcommunicationinﬁnite-state
environments. It is more general, yet mathematically simple and works for arbi-
trary environments. This assumption turns out to be what we need to prove the
connection from asymptotic optimality to sublinear regret in Section 5.5.
Moreover, we introduce the use of the recursive instead of the iterative value
function (Section 6.4). The iterative value function is the natural extension of
expectimax search to the sequential setting and was originally used by Hutter
(2005, Sec. 5.5). Yet it turned out to be an incorrect and inconvenient deﬁnition:
it does not correctly maximize expected rewards (Proposition 6.19) and it is not
limit computable (Theorem 6.22 and Theorem 6.23). However, this is only a
minor technical correction.
Finally, this work raises new mathematically intriguing questions about the prop-
erties of reﬂective oracles (Section 7.1).
10 Introduction
•Practically. One insight from this thesis is regarding the eﬀective horizon. In
practice geometric discounting is ubiquitous which has a constant eﬀective hori-
zon. However, when facing a ﬁnite horizon problem or an episodic task, some-
times the eﬀective horizon changes. One lesson from our result on Thompson
sampling (Section 5.4.3 and Section 5.5) is that you should explore for an ef-
fective horizon instead of using "-greedy. While the latter exploration method
is often used in practice, it has proved ineﬀective in environments with delayed
rewards (see Section 1.1.2).
Furthermore, our application of reinforcement learning results to game theory in
Chapter 7 reinforces this trend to solve game theory problems (Tesauro, 1995;
Bowling and Veloso, 2001; Busoniu et al., 2008; Silver et al., 2016; Heinrich and
Silver, 2016; Foerster et al., 2016, and many more). In particular, the approxima-
tion algorithm for reﬂective oracles (Section 7.1.3) could guide future applications
for computing Nash equilibria (see also Fallenstein et al., 2015b).
On a technical level, we advance the theory of general reinforcement learning. In its
center is the Bayesian reinforcement learning agent AIXI. AIXI is meant as an answer
to the question of how to do general RL disregarding computational constraints. We
analyze the computational complexity of AIXI and related agents in Chapter 6 and
show that even with an inﬁnite horizon AIXI can be computationally approximated
with a regular Turing machine (Section 6.3.1). We also derive corresponding lower
bounds for most of our upper bounds (Section 6.3.2).
Chapter 5 is about notions of optimality in general reinforcement learning. We
dispel AIXI’s status as the gold standard for reinforcement learning. Hutter (2002a)
showed that AIXI is Pareto optimal, balanced Pareto optimal, and self-optimizing.
Orseau (2013) established that AIXI does not achieve asymptotic optimality in all
computable environments (making the self-optimizing result inapplicable to this gen-
eral environment class). In Section 5.1 we show that every policy is Pareto optimal and
in Section 5.3 we show that balanced Pareto optimality is highly subjective, depending
on the choice of the prior; bad choices for priors are discussed in Section 5.2. Notable
is thedogmatic prior that locks a Bayesian reinforcement learning agent into a particu-
lar (bad) policy as long as this policy yields some rewards. Our results imply that there
are no known nontrivial and non-subjective optimality results for AIXI. We have to
regard AIXI as a relativetheory of intelligence. More generally, our results imply that
general reinforcement learning is diﬃcult even when disregarding computational costs .
But this is not the end to Bayesian methods in general RL. We show in Section 5.4
that a Bayes-inspired algorithm called Thompson sampling achieves asymptotic opti-
mality. Thompson sampling, also known as posterior sampling or theBayesian control
rulerepeatedly draws one environment from the posterior distribution and then acts as
if this was the true environment for a certain period of time (depending on the discount
function). Moreover, given a recoverability assumption on the environment and some
mild assumptions on the discount function, we show in Section 5.5 that Thompson
sampling achieves sublinear regret.
Finally, we tie these results together to solve an open problem in game theory:
§1.3Thesis Outline 11
Chapter Publication(s)
Chapter 1 -
Chapter 2 -
Chapter 3 with links to Leike and Hutter (2014a, 2015d); Filan et al. (2016)
Chapter 4 -
Chapter 5 Leike and Hutter (2015c); Leike et al. (2016a)
Chapter 6 Leike and Hutter (2015b,a, 2016)
Chapter 7 Leike et al. (2016b)
Chapter 8 -
Appendix A Leike and Hutter (2014b)
Table 1.2: List of publications by chapter.
When acting in a multi-agent environment with other Bayesian agents, each agent
needs to assign positive prior probability to the other agents’ actual policies (they need
to have a grain of truth ). Finding a reasonably large class of policies that contains the
Bayes optimal policies with respect to this class is known as the grain of truth prob-
lem(Hutter, 2009b, Q. 5j). Only small classes are known to have a grain of truth and
the literature contains several related impossibility results (Nachbar, 1997, 2005; Foster
and Young, 2001). Moreover, while AIXI assumes the environment to be computable,
our computability results on AIXI conﬁrm that it is incomputable (Theorem 6.15 and
Theorem 6.17). This asymmetry elevates AIXI above its environment computationally,
and prevents the environment from containing other AIXIs.
In Chapter 7 we give a formal and general solution to the grain of truth prob-
lem: we construct a class of policies that avoid this asymmetry. This class contains all
computable policies as well as Bayes optimal policies for every lower semicomputable
prior over the class. When the environment is unknown, our dogmatic prior from Sec-
tion 5.2 makes Bayes optimal agents fail to act optimally even asymptotically. However,
our convergence results on Thompson sampling (Section 5.4.3) imply that Thompson
samplers converge to play "-Nash equilibria in arbitrary unknown computable multi-
agent environments. While these results are purely theoretical, we use techniques from
Chapter 6 to show that they can be computationally approximated arbitrarily closely.
1.3 Thesis Outline
This thesis is based on the papers Leike and Hutter (2014a,b, 2015a,b,c, 2016); Leike
et al. (2016a,b). During my PhD, I was also involved in the publications Leike and Heiz-
mann(2014a,b,2015);Heizmannetal.(2015,2016)basedonmyresearchintermination
analysis (in collaboration with Matthias Heizmann), Daswani and Leike (2015) (co-
authored with Mayank Daswani in equal parts), Everitt et al. (2015) (co-authored with
Tom Everitt in equal parts), Filan et al. (2016) (written by Daniel Filan as part of
his honour’s thesis supervised by Marcus Hutter and me). Leike and Hutter (2016) is
12 Introduction
still under review. Leike and Hutter (2014a, 2015d) are tangential to this thesis’ main
thrust, so the results are mentioned only in passing. A list of papers written during my
PhD is given in Table 1.3 on page 14, with a corresponding chapter outline in Table 1.2.
The core of our contribution is found in chapters 5, 6, and 7.
Every thesis chapter starts with a quote. In case this is not blatantly obvious: these
are false quotes , a desperate attempt to make the thesis less dry and humorless. None
of the quotes were actually stated by the person they are attributed to (according to
our knowledge).
§1.3Thesis Outline 13
(a)Space Invaders: the player controls the
green cannon on the bottom of the screen
and ﬁres projectiles at the yellow ships at
the top. The red blobs can be used as
cover, but also ﬁred through.
(b)Pong: the player controls the green
paddle on the right of the screen and needs
to hit the white ball such that the com-
puter opponent controlling the red paddle
on the left fails to hit the ball back.
(c)Ms. Pac-Man: the player controls the
yellow mouth and needs to eat all the red
pallets in the maze. The maze is roamed
by ghosts that occasionally hunt the player
and kill her on contact unless a ‘power pill’
was consumed recently.
(d)Boxing: the player controls the white
ﬁgure on the screen and extends their arms
to throw a punch. The aim is to hit the
black ﬁgure that is controlled by the com-
puter and dodge their punches. (I’m sure
the choice of color was by accident.)
(e)Gopher: a hungry rodent attempts to
dig to the surface and steal the vegetables.
The player controls the farmer who pro-
tects them by ﬁlling the rodent’s holes.
(f)Montezuma’s Revenge: the player con-
trols the red adventurer. The aim is to
navigate a maze of deadly traps, use keys
to open doors, and collect artifacts.
Figure 1.1: A selection of Atari 2600 video games.
14 Introduction
[1] Jan Leike and Marcus Hutter. Indeﬁnitely oscillating martingales. In Algorithmic Learning
Theory, pages 321–335, 2014a
[2] Jan Leike and Matthias Heizmann. Ranking templates for linear loops. Logical Methods in
Computer Science , 11(1):1–27, March 2015
[3] Mayank Daswani and Jan Leike. A deﬁnition of happiness for reinforcement learning agents.
InArtiﬁcial General Intelligence , pages 231–240. Springer, 2015
[4] Tom Everitt, Jan Leike, and Marcus Hutter. Sequential extensions of causal and evidential
decision theory. In Algorithmic Decision Theory , pages 205–221. Springer, 2015
[5] Jan Leike and Marcus Hutter. On the computability of AIXI. In Uncertainty in Artiﬁcial
Intelligence , pages 464–473, 2015a
[6] JanLeikeandMarcusHutter. OnthecomputabilityofSolomonoﬀinductionandknowledge-
seeking. In Algorithmic Learning Theory , pages 364–378, 2015b
[7] Jan Leike and Marcus Hutter. Bad universal priors and notions of optimality. In Conference
on Learning Theory , pages 1244–1259, 2015c
[8] Jan Leike and Marcus Hutter. Solomonoﬀ induction violates Nicod’s criterion. In Algorith-
mic Learning Theory , pages 349–363. Springer, 2015d
[9] Matthias Heizmann, Daniel Dietsch, Jan Leike, Betim Musa, and Andreas Podelski. Ulti-
mate Automizer with array interpolation (competition contribution). In Tools and Algo-
rithms for the Construction and Analysis of Systems , pages 455–457. Springer, 2015
[10] Matthias Heizmann, Daniel Dietsch, Marius Greitschus, Jan Leike, Betim Musa, Claus
Schätzle, and Andreas Podelski. Ultimate Automizer with two-track proofs (competition
contribution). In Tools and Algorithms for the Construction and Analysis of Systems , pages
950–953. Springer, 2016
[11] Daniel Filan, Jan Leike, and Marcus Hutter. Loss bounds and time complexity for speed
priors. In Artiﬁcial Intelligence and Statistics , 2016
[12] Jan Leike and Marcus Hutter. On the computability of Solomonoﬀ induction and AIXI.
2016. Under review
[13] Jan Leike, Tor Lattimore, Laurent Orseau, and Marcus Hutter. Thompson sampling is
asymptotically optimal in general environments. In Uncertainty in Artiﬁcial Intelligence ,
2016a
[14] Jan Leike, Jessica Taylor, and Benya Fallenstein. A formal solution to the grain of truth
problem. In Uncertainty in Artiﬁcial Intelligence , 2016b
[15] Jan Leike and Matthias Heizmann. Geometric nontermination arguments. 2016. Under
preparation
Table 1.3: List of publications.
Chapter 2
Preliminaries
Mathematics is a waste of time. — Leonhard Euler
This chapter establishes the notation and background material that is used through-
out this thesis. Section 2.1 is about probability and measure theory, Section 2.2 is
about stochastic processes, Section 2.3 is about information theory, and Section 2.4 is
about algorithmic information theory. We defer the formal introduction to reinforce-
ment learning to Chapter 4. Additional preliminary notation and terminology is also
established in individual chapters wherever necessary. A list of notation is provided in
the appendix on page 171.
Most of the content from this chapter can be found in standard textbooks and
reference works. We recommend to consult Wasserman (2004) on statistics, Durrett
(2010) on probability theory and stochastic processes, Cover and Thomas (2006) on
information theory, Li and Vitányi (2008) on algorithmic information theory, Russell
and Norvig (2010) on artiﬁcial intelligence, Bishop (2006) and Hastie et al. (2009)
on machine learning, Sutton and Barto (1998) on reinforcement learning, and Hutter
(2005) and Lattimore (2013) on general reinforcement learning.
We understand deﬁnitions to follow natural language; e.g., when deﬁning the ad-
jective ‘continuous’, we deﬁne at the same time the noun ‘continuity’ and the adverb
‘continuously’ wherever appropriate.
Numbers. N:=f1;2;3;:::gdenotes the set of natural numbers (starting from 1),
Q:=fp=qjp2N[f0g;q2Ngdenotes the set of rational numbers, and Rdenotes the
set of real numbers. For two real numbers r1;r2, the set [r1;r2] :=fr2Rjr1rr2g
denotes the closed interval with end points r1andr2; the sets (r1;r2] := [r1;r2]nfr1g
and[r1;r2) := [r1;r2]nfr2gdenotehalf-openintervals; theset (r1;r2) := [r1;r2]nfr1;r2g
denotes an open interval.
Strings. FixXto be a ﬁnite nonempty set, called alphabet. We assume that X
contains at least two distinct elements. The set X:=S1
n=0Xnis the set of all ﬁnite
strings over the alphabet X, the setX1is the set of all inﬁnite strings over the alphabet
X, and the setX]:=X[X1is their union. The empty string is denoted by , not to
be confused with the small positive real number ". Given a string x2X], we denote
its length byjxj. For a (ﬁnite or inﬁnite) string xof lengthk, we denote with xkthe
k-th character of x, withx1:kthe ﬁrstkcharacters of x, and with x<kthe ﬁrstk1
15
16 Preliminaries
characters of x. The notation x1:1stresses that xis an inﬁnite string. We use xvy
to denote that xis a preﬁx of y, i.e.,x=y1:jxj. Our examples often (implicitly) involve
the binary alphabet f0;1g. In this case we deﬁne the functions ones;zeros :X!N
that count the number of ones and zeros in a string respectively.
Computability. A function f:X!Rislower semicomputable iﬀ the setf(x;q)2
XQjf(x)>qgis recursively enumerable. If fandfare lower semicomputable,
thenfis called computable . See Section 6.1.2 for more computability deﬁnitions.
Asymptotic Notation. Letf;g:N!R0. We usef2O(g)to denote that there
is a constant csuch thatf(t)cg(t)for allt2N. We usef2o(g)to denote that
lim supt!1f(t)=g(t) = 0. For functions on strings P;Q :X!Rwe useQPto
denote that there is a constant c>0such thatQ(x)cP(x)for allx2X. We also
useQPforPQandQ=PforQPandPQ. Note that Q=Pdoes not
implythat there is a constant csuch thatQ(x) =cP(x)for allx2X. For a sequence
(at)t2Nwith limit limt!1at=awe also write at!aast!1. If no limiting variable
is provided, we mean t!1by convention.
Other Conventions. LetAbe some set. We use #Ato denote the cardinality of
the setA, i.e., the number of elements in A, and 2Ato denote the power set of A, i.e.,
the set of all subsets of A. We use logto denote the binary logarithm and lnto denote
the natural logarithm.
2.1 Measure Theory
For a countable set
, we use
to denote the set of probability distributions over

. If
is uncountable (such as the set of all inﬁnite strings X1), we need to use the
machinery of measure theory. This section provides a concise introduction to measure
theory; see Durrett (2010) for an extensive treatment.
Deﬁnition 2.1 (-algebra).Let
be a set. The set F 2
is a-algebra over
iﬀ
(a)
2F,
(b)A2Fimplies
nA2F, and
(c) for any countable number of sets A0;A1;:::;2F, the unionS
i2NAi2F.
For a setA 2
, we deﬁne (A)to be the smallest (with respect to set inclusion)
-algebra containing A.
For the real numbers, the default -algebra (used implicitly) is the Borel-algebraB
generated by the open sets of the usual topology. Formally, B:=(f(a;b)ja;b2Rg).
A set
together with a -algebraFforms ameasurable space . The sets from the -
algebraFare called measurable sets . A function f:
1!
2between two measurable
spaces is called measurable iﬀ any preimage of an (in
2) measurable set is measurable
(in
1).
§2.1Measure Theory 17
Deﬁnition 2.2 (Probability Measure) .Let
be a measurable space with -algebra
F. Aprobability measure on the space
is a function :F! [0;1]such that
(a)(
) = 1(normalization ), and
(b)(S
i2NAi) =P
i2N(Ai)for any collection fAiji2NgFthat is pairwise
disjoint (-additivity ).
A probability measure isdeterministic iﬀ it assigns all probability mass to a single
element of
, i.e., iﬀ there is an x2
with(fxg) = 1.
We deﬁne the conditional probability (AjB)for two measurable sets A;B2F
with(B)>0as(AjB) :=(A\B)=(B).
Deﬁnition 2.3 (Random Variable) .Let
be a measurable space with probability
measure. A(real-valued) random variable is a measurable function X:
!R.
We often (but not always) denote random variables with uppercase Latin letters.
Given a-algebraF, a probability measure PonF, and anF-measurable random
variableX, theconditional expectation E[XjF]ofXgivenFis a random variable
Ysuch that (1) YisF-measurable and (2)R
AXdP =R
AYdPfor allA2F. The
conditional expectation exists and is unique up to a set of P-measure 0(Durrett, 2010,
Sec. 5.1). Intuitively, if Fdescribes the information we have at our disposal, then
E[XjF]denotes the expectation of Xgiven this information.
We proceed to deﬁne the -algebra onX1(the-algebra onX]is deﬁned analo-
gously). For a ﬁnite string x2X, thecylinder set
x:=fxyjy2X1g
is the set of all inﬁnite strings of which xis a preﬁx. Furthermore, we ﬁx the -algebras
Ft:=
fxjx2Xtg
andF1:= 1[
t=1Ft!
:
The sequence (Ft)t2Nis aﬁltration : from x=S
a2Xxafollows thatFtFt+1for
everyt2N, and allFtF1by the deﬁnition of F1.
For our purposes, the -algebraFtmeans ‘all symbols up to and including time
stept.’ So instead of conditioning an expectation on Ft, we can just as well condition
it on the sequence x1:tdrawn at time t. Hence we write E[Xjx1:t]instead of E[XjFt].
Moreover, for conditional probabilities we also write Q(xtjx<t)instead ofQ(x1:tjx<t).
In the context of probability measures, a measurable set E2F1is also called an
event. The event Ec:=X1nEdenotes the complement of E. In case the event Eis
deﬁned by a predicate Qdependent on the random variable X,E=fx2
jQ(X(x))g,
we also use the shorthand notation
P[Q(X)] :=P(fx2
jQ(X(x))g) =P(E):
We assume all sets to be measurable; when we write P(A)for some set AX1,
we understand implicitly that Abe measurable. This is not true: not all subsets of
18 Preliminaries
X1are measurable (assuming the axiom of choice). While we choose to do this for
readability purposes, note that under some axioms compatible with Zermelo-Fraenkel
set theory, notably the axiom of determinacy, all subsets of X1are measurable.
2.2 Stochastic Processes
This section introduces some notions about sequences of random variables.
Deﬁnition 2.4 (Stochastic Process) .(Xt)t2Nis astochastic process iﬀXtis a random
variable for every t2N.
A stochastic process (Xt)t2Nisnonnegative iﬀXt0for allt2N. The process is
bounded iﬀ there is a constant c2Rsuch thatjXtjcfor allt2N.
In the real numbers, a sequence (zt)t2Nconverges if and only if it is a Cauchy se-
quence, i.e., iﬀjzt+1ztj!0ast!1. For sequences of random variables convergence
is a lot more subtle and there are several diﬀerent notions of convergence.
Deﬁnition2.5 (StochasticConvergence) .LetPbeaprobabilitymeasure. Astochastic
process (Xt)t2Nconverges to the random variable X
•inP-probability iﬀ for every ">0,
P
jXtXj>"
!0ast!1 ;
•inP-meaniﬀ
EP
jXtXj
!0ast!1 ;
•P-almost surely iﬀ
Ph
lim
t!1Xt=Xi
= 1:
Almost sure convergence and convergence in mean both imply convergence in prob-
ability (Wasserman, 2004, Thm. 5.17). If the stochastic process is bounded, then con-
vergence in probability implies convergence in mean (Wasserman, 2004, Thm. 5.19).
A sequence of real numbers (at)t2Nconverges in Cesàro average toa2Riﬀ
1=tPt
k=1ak!aast! 1. The deﬁnition for sequences of random variables is
analogous.
Deﬁnition 2.6 (Martingale) .LetPbe a probability measure over (X1;F1). A
stochastic process (Xt)t2Nis aP-supermartingale ( P-submartingale) iﬀ
(a) eachXtisFt-measurable, and
(b)E[XtjFs]Xs(E[XtjFs]Xs)P-almost surely for all s;t2Nwiths<t.
AP-martingale is a process that is both a P-supermartingale and a P-submartingale.
§2.3Information Theory 19
Example 2.7 (Fair Gambling) .Suppose Mary bets on the outcome of a fair coin ﬂip.
If she predicts correctly, her wager is doubled and otherwise it is lost. Let Xtdenote
Mary’s wealth at time step t. Since the game is fair, E[Xt+1jFt] =XtwhereFt
represents the information available at time step t. Hence E[Xt] =X1, so in expectation
she never loses money regardless of her betting strategy. 3
FormartingalesthefollowingfamousconvergenceresultwasprovedbyDoob(1953).
Theorem 2.8 (Martingale Convergence; Durrett, 2010, Thm. 5.2.9) .If(Xt)t2Nis a
nonnegative supermartingale, then it converges almost surely to a limit XwithE[X]
E[X1].
By Theorem 2.8 the martingale from Example 2.7 representing Mary’s wealth con-
verges almost surely, regardless of her betting strategy. Either she refrains from betting
at some point (assuming she cannot place smaller and smaller bets) or she cannot play
anymore because her wealth is 0. Is there a lesson to learn here about gambling?
2.3 Information Theory
This section introduces the notions of entropy and two notions of distance between
probability measures: KL-divergence andtotal variation distance .
Deﬁnition 2.9 (Entropy) .Let
be a countable set. For a probability distribution
p2
, theentropyofpis deﬁned as
Ent(p) :=X
x2
:p(x)>0p(x) logp(x):
Deﬁnition 2.10 (KL-Divergence) .LetP;Qbe two measures and let m2Nbe a
lookahead time step. The Kullback-Leibler-divergence (KL-divergence ) ofPandQ
between time steps tandmis deﬁned as
KLm(P;Qjx<t) :=X
xt:m2Xmt+1P(x1:mjx<t) logP(x1:mjx<t)
Q(x1:mjx<t):
Moreover, we deﬁne KL1(P;Qjx<t) := limm!1KLm(P;Qjx<t).
KL-divergence is also known as relative entropy . KL-divergence is always nonneg-
ative by Gibbs’ inequality, but it is not a distance since it is not symmetric. If the
alphabetXis ﬁnite, then KLm(P;Qjx)is always ﬁnite. However, KL1(P;Qjx)may
be inﬁnite.
Deﬁnition 2.11 (Total Variation Distance) .LetP;Qbe two measures and let 1
m1be a lookahead time step. The total variation distance betweenPandQ
between time steps tandmis deﬁned as
Dm(P;Qjx) := sup
AXmP(Ajx<t)Q(Ajx<t):
20 Preliminaries
Total variation distance is always bounded between 0and1sincePandQare prob-
ability measures. Moreover, in contrast to KL-divergence total variation distance satis-
ﬁes the axioms of distance: symmetry ( D(P;Q) =D(Q;P)), identity of indiscernibles
(D(P;Q) = 0if and only if P=Q), and the triangle inequality ( D(P;Q) +D(Q;R)
D(P;R)).
The following lemma shows that total variation distance can be used to bound
diﬀerences in expectation.
Lemma 2.12 (Total Variation Bound on the Expectation) .For a random variable X
with 0X1and two probability measures PandQ
EP[X]EQ[X]D(P;Q):
KL-divergence and total variation distance are linked by the following inequality.
Lemma 2.13 (Pinsker’s inequality; Tsybakov, 2008, Lem. 2.5i) .For all probability
measuresPandQonX1, for everyx2X, and for every m2N
Dm(P;Qjx)r
1
2KLm(P;Qjx)
2.4 Algorithmic Information Theory
Auniversal Turing machine (UTM) is a Turing machine that can simulate all other
Turing machines. Formally, a Turing machine Uis a UTM iﬀ for every Turing machine
Tthere is a binary string p(calledprogram) such that U(p;x) =T(x)for allx2X,
i.e., the output of Uwhen run on (p;x)is the same as the output of Twhen run on x.
We assume the set of programs on Uis preﬁx-free. The Kolmogorov complexity K(x)
of a stringxis the length of the shortest program on Uthat prints xand then halts:
K(x) := minfjpjjU(p) =xg:
Amonotone Turing machine is a Turing machine with a one-way read-only input
tape, a one-way write-only output tape, and a read/write work tape. Monotone Turing
machines sequentially read symbols from their input tape and write to their output
tape. Interpreted as a function, a monotone Turing machine Tmaps a string xto the
longest string that Twrites to the output tape while reading xand no more from the
input tape (Li and Vitányi, 2008, Ch. 4.5.2).
We also use Uto denote a universal monotone Turing machine (programs on the
universal monotone Turing machine do not have to be preﬁx-free). The monotone Kol-
mogorov complexity Km(x)denotes the length of the shortest program on the monotone
machineUthat prints a string starting with x(Li and Vitányi, 2008, Def. 4.5.9):
Km(x) := minfjpjjxvU(p)g: (2.1)
Since monotone complexity does not require the machine to halt, there is a constant c
such thatKm(x)K(x) +cfor allx2X.
§2.4Algorithmic Information Theory 21
The following notion of a (semi)measure is particular to algorithmic information
theory.
Deﬁnition 2.14 (Semimeasure; Li and Vitányi, 2008, Def. 4.2.1) .Asemimeasure over
the alphabetXis a function :X![0;1]such that
(a)()1, and
(b)(x)P
a2X(xa)for allx2X.
A semimeasure is a (probability) measure iﬀ equalities hold in (a) and (b) for all x2X.
Semimeasures are not probability measures in the classical measure theoretic sense.
However, semimeasures correspond canonically to classical probability measures on the
probability space X]=X[X1whose-algebra is generated by the cylinder sets (Li
and Vitányi, 2008, Ch. 4.2 and Hay, 2007).
Lower semicomputable semimeasures correspond naturally to monotone Turing ma-
chines (Li and Vitányi, 2008, Thm. 4.5.2): for a monotone Turing machine T, the
semimeasure Tmaps a string xto the probability that Toutputs something starting
withxwhen fed with fair coin ﬂips as input (and vice versa). Hence we can enumerate
all lower semicomputable semimeasures 1;2;:::by enumerating all monotone Tur-
ing machines. We deﬁne the Kolmogorov complexity K()of a lower semicomputable
semimeasure as the Kolmogorov complexity of the index of in this enumeration.
We often mix the (semi)measures of algorithmic information theory with concepts
from probability theory. For convenience, we identify a ﬁnite string x2Xwith its
cylinder set x. Then(x)in the algorithmic information theory sense coincides with
(x)in the measure theory sense if we use the identiﬁcation of semimeasures with
probability measures above.
Example 2.15 (Lebesgue Measure) .TheLebesgue measure oruniform measure is
deﬁned as
(x) := (#X)jxj: 3
The following deﬁnition turns a semimeasure into a measure, preserving the predic-
tive ratio(xa)=(xb)fora;b2X.
Deﬁnition 2.16 (Solomonoﬀ Normalization) .TheSolomonoﬀ normalization normof
a semimeasure is deﬁned as norm() := 1and for allx2Xanda2X,
norm(xa) :=norm(x)(xa)P
b2X(xb): (2.2)
By deﬁnition, normis a measure. Moreover, normdominatesaccording to the
following lemma.
Lemma 2.17 (norm).norm(x)(x)for allx2Xand all semimeasures .
22 Preliminaries
Proof.We use induction on the length of x: ifx=thennorm() = 1 =(), and
otherwise
norm(xa) =norm(x)(xa)P
b2X(xb)(x)(xa)P
b2X(xb)(x)(xa)
(x)=(xa):
The ﬁrst inequality holds by induction hypothesis and the second inequality uses the
fact thatis a semimeasure.
Chapter 3
Learning
The problem of induction is essentially solved. — David Hume
Machine learning refers to the process of learning models of and/or making predictions
about (large) sets of data points that are typically independent and identically dis-
tributed (i.i.d.); see Bishop (2006) and Hastie et al. (2009). In this chapter we do not
makethei.i.d.assumption. Instead, weaimmoregenerallyatthetheoreticalfundamen-
tals of the sequence prediction problem : how will a sequence of symbols generated by
an unknown stochastic process be continued? Given a ﬁnite string x<t=x1x2:::xt1
of symbols, what is the next symbol xt? How likely does a given property hold for the
entire sequence x1:1? Arguably, any learning or prediction problem can be phrased in
this fashion: anything that can be stored on a computer can be turned into a sequence
of bits.
We distinguish two major elements of learning. First, the process of converging
to accurate beliefs, called merging. Second, the process of making accurate forecasts
about the next symbol, called predicting . These two notions are not distinct: if you
have accurate beliefs about the unseen data, then you can make good predictions, but
not necessarily vice versa (see Example 3.41). We discuss diﬀerent notions of merging
in Section 3.4 and state bounds on the prediction regret in Section 3.5.
In the general reinforcement learning problem we target in this thesis, the environ-
ment is unknown and the agent needs to learn it. The literature on non-i.i.d. learning
has focused on predicting individual symbols and bounds on the number of prediction
errors (Hutter, 2001b, 2005; Cesa-Bianchi and Lugosi, 2006), and the results on merg-
ing are from the game theory literature (Blackwell and Dubins, 1962; Kalai and Lehrer,
1994; Lehrer and Smorodinsky, 1996). However, we argue that merging is the essential
property for general AI. In order to make good decisions, the agent needs to have ac-
curate beliefs about what its actions will entail. On a technical level, merging leads to
on-policy value convergence (Section 4.2.3), the fact that the agents learns to estimate
the values for its own policy correctly.
The setup we consider is the realizable case : we assume that the data is generated
by an unknown probability distribution that belongs to a known (countable) class of
distributions. In contrast, the nonrealizable case allows no assumptions on the under-
lying process that generates the data. A well-known approach to the nonrealizable case
isprediction with expert advice (Cesa-Bianchi and Lugosi, 2006), which we do not con-
23
24 Learning
sider here. Generally, the nonrealizable case is harder, but Ryabko (2011) argues that
for some problems, both cases coincide.
After introducing the formal setup in Section 3.1, we discuss several examples for
learning distributions and notions that relate the learning distribution with the process
generating the data in Section 3.2. In Section 3.3 we connect these notions to the theory
of martingale processes.
Section 3.6 connects the results from the ﬁrst sections to the learning framework
developed by Solomonoﬀ (1964, 1978), Hutter (2001b, 2005, 2007b), and Schmidhuber
(2002) (among others). This framework relies on results from algorithmic information
theory and computability theory to learn any computable distribution quickly and
eﬀectively. It is incomputable (see Section 6.2), but can serve as a gold standard for
learning.
Most of this chapter echoes the literature. We collect results from economics and
computer science that previously had not been assembled in one place. We provide
proofs that connect the various properties (Proposition 3.23, Proposition 3.16, and
Proposition 3.37), and we ﬁll in a few gaps in the picture: the prediction bounds for
absolute continuity (Section 3.5.2) and the improved regret bounds for nonuniform
measures (Theorem 3.48 and Theorem 3.51). Section 3.7 summarizes the results in
Table 3.2 on page 45 as well as Figure 3.1 on page 46.
3.1 Setup
For the rest of this chapter, ﬁx PandQto be two probability measures over the
measurablespaceofinﬁnitesequences (X1;F1). Wethinkof Pasthetrue distribution
from which the data sequence x1:1is drawn, and of Qas our belief distribution or
learning algorithm. In other words, we use the distribution Qto learn a string drawn
from the distribution P.
LetHdenote a hypothesis , i.e., any measurable set from F1. Ourprior belief in
the hypothesis HisQ(H). In each time step t, we make one observation xt2X. Our
historyx<t=x1x2:::xt1is the sequence of all previous observations. We update our
belief in accordance with Bayesian learning; our posterior belief in the hypothesis His
Q(Hjx1:t) =Q(H\x1:t)
Q(x1:t):
The observation xtconﬁrms the hypothesis HiﬀQ(Hjx1:t)>Q(Hjx<t)(the belief
inHincreases), and the observation xtdisconﬁrms the hypothesis HiﬀQ(Hjx1:t)<
Q(Hjx<t)(the belief in Hdecreases). If Q(Hjx1:t) = 0, thenHisrefutedorfalsiﬁed.
When we assign a prior belief of 0to a hypothesis H, this means that we think
thatHis impossible; it is refuted from the beginning. If Q(H) = 0, then the posterior
Q(Hjx<t) = 0, so no evidence whatsoever can change our mind that His impossible.
This is bad if the hypothesis His actually true.
To be able to learn we need to make some assumptions on the learning distribution
Q: we need to have an open mind about anything that might actually happen, i.e.,
§3.2Compatibility 25
Q(H)>0on any hypothesis HwithP(H)>0. This property is called absolute
continuity . We discuss this and other notions of compatibility of PandQin Section 3.2.
We motivate this chapter with the following example.
Example 3.1 (The Black Ravens; Rathmanner and Hutter, 2011, Sec. 7.4) .If we live
in a world in which all ravens are black, how can we learn this fact? Since at every time
step we have observed only a ﬁnite subset of the (possibly inﬁnite) set of all ravens,
how can we conﬁdently state anything about all ravens?
We formalize this problem in line with Rathmanner and Hutter (2011, Sec. 7.4) and
Leike and Hutter (2015d). We deﬁne two predicates, blackness Band ravenness R.
There are four possible observations: a black raven BR, a non-black raven BR, a black
non-ravenBR, and a non-black non-raven BR. Therefore our alphabet consists of four
symbols corresponding to each of the possible observations, X:=fBR;BR;BR;BRg.
We are interested in the hypothesis ‘all ravens are black’. Formally, it corresponds
to the measurable set
H:=fx2X1jxt6=BR8tg=fBR;BR;BRg1; (3.1)
the set of all inﬁnite strings in which the symbol BRdoes not occur.
Ifweobserveanon-blackraven, xt=BR,thehypothesis Hisrefutedsince H\x1:t=
;and this implies Q(Hjx1:t) = 0. In this case, our inquiry regarding His settled.
The interesting case is when the hypothesis His in fact true ( P(H) = 1), i.e.,Pdoes
not generate any non-black ravens. The property we desire is that in a world in which
all ravens are black, we arrive at this belief: P(H) = 1impliesQ(Hjx<t)!1as
t!1. 3
3.2 Compatibility
In this section we deﬁne dominance ,absolute continuity ,dominance with coeﬃcients ,
weak dominance , andlocal absolute continuity , in decreasing order of their strength.
These notions make the relationship of the two probability measures PandQprecise.
We also give examples for various choices for the learning algorithm Q.
In our examples, we frequently rely on the following process.
Example 3.2 (Bernoulli Process) .AssumeX=f0;1g. For a real number r2[0;1]
we deﬁne the Bernoulli process with parameter ras the measure
Bernoulli (r)(x) :=rones(x)(1r)zeros(x):
Note that Bernoulli (1=2) =, the Lebesgue measure from Example 2.15. 3
Deﬁnition 3.3 (Dominance) .The measure Qdominates P(QP) iﬀ there is a
constantc>0such thatQ(x)cP(x)for all ﬁnite strings x2X.
Dominance is also called having a grain of truth (Lehrer and Smorodinsky, 1996,
Def. 2a and Kalai and Lehrer, 1993); we discuss this property in the context of game
theory in Chapter 7.
26 Learning
Example 3.4 (Bayesian mixture) .LetMbe a countable set of probability measures
on(X1;F1)and letw2Mbe a prior overM. Ifw(P)>0for allP2M, the
priorwis called positiveoruniversal . Then the Bayesian mixture :=P
P2Mw(P)P
dominates each P2M. 3
The Bayesian mixture is a mathematically simple yet very powerful concept. It is
very easy to derive from a countable set of distributions, and it has been considered
extensivelyintheliterature(Solomonoﬀ,1964;Jaynes,2003;Hutter,2005, ...). Ryabko
(2009)showsthatevenforuncountablyinﬁniteclasses, iftherearegoodpredictors, then
a Bayesian mixture over a countable subclass asymptotically also does well.
Example 3.5 (Solomonoﬀ Prior) .Solomonoﬀ (1964) deﬁnes a distribution MoverX]
that assigns to a string xthe probability that the universal monotone Turing machine
Uoutputsxwhen fed with fair coin ﬂips on the input tape. Formally,
M(x) :=X
p:xvU(p)2jpj(3.2)
wherepisabinarystring.1Thefunction Misalowersemicomputablesemimeasure,but
not computable and not a measure (Li and Vitányi, 2008, Lem. 4.5.3); see Section 6.2
for the computability properties of M. More importantly, Mdominates every lower
semicomputable semimeasure (Li and Vitányi, 2008, Thm. 4.5.1).
Solomonoﬀ’s prior Mhas a number of appealing philosophical properties. In line
with Ockham’s razor it favors simple environments over complex ones: Turing machines
that have a short program on the UTM Uhave a higher contribution in the sum (3.2).
In line with Epicurus’ principle it never discards possible explanations: every program
that produces the string xcontributes to the sum. See Rathmanner and Hutter (2011)
for a discussion on the philosophical underpinnings of Solomonoﬀ’s prior. 3
Wood et al. (2011) show that the Solomonoﬀ prior Mcan equivalently be deﬁned as
a Bayesian mixture over all lower semicomputable semimeasures with a prior w(P)/
2K(P). (If we use w(P) = 2K(P)we get a semiprior becauseP
P2M2K(P)can be
less than 1. This prior also carries the name Solomonoﬀ prior .)
Deﬁnition 3.6 (Absolute Continuity) .The measure Pisabsolutely continuous with
respect toQ(QP) iﬀQ(A) = 0impliesP(A) = 0for all measurable sets A.
Remark 3.7 (Absolute Continuity ;Dominance) .Absolute continuity is strictly
weaker than dominance: let X:=f0;1gand deﬁne a probability measure Pthat
assigns probability 2=3to1and probability 1=3to0until seeing the ﬁrst 0, thenP
behaves like the Lebesgue measure . Formally,
P(x1:t) :=(2
3tifx1:t= 1t;and2
3n1
3(xn+2:t)if9n0:1n0vx1:t:
1We use the name Solomonoﬀ prior for both a distribution over X1and a distribution over a com-
putably enumerable set M. Maybe Mshould better be called Solomonoﬀ mixture to avoid confusion.
§3.2Compatibility 27
Since(1t)=P(1t) = (3=4)t!0ast!1, there is no constant csuch that(x)=P(x)>
c>0for all ﬁnite strings x2X, hencedoes not dominate P. ButPis absolutely
continuous with respect to becauseP-almost surely we draw a 0eventually, and
thenPbehaves like . HenceP-almost surely =P6!0. The claim now follows from
Proposition 3.23b. 3
The idea of Remark 3.7 is to ‘punch a hole into ’ at the inﬁnite string 11. This
inﬁnite string has probability 0, hence this hole does not break absolute continuity.
But it breaks dominance on this inﬁnite string. Analogously we could punch countably
many holes into a probability measure without breaking absolute continuity.
Deﬁnition 3.8 (Weak Dominance) .The measure Qweakly dominates P(QWP)
iﬀ
lim
t!11
tlogQ(x1:t)
P(x1:t)= 0withP-probability 1:
Lehrer and Smorodinsky (1996, Rem. 8) point out that for any PandQ,
lim sup
t!11
tlogQ(x1:t)
P(x1:t)0P-almost surely ;
so crucial is whether the lim infis also 0.
Remark 3.9 (Weak Dominance) .The measure Qweakly dominates Pif and only if
P-almost surely log(P(x)=Q(x))2o(t). 3
Ryabko and Hutter (2007, 2008) consider the following deﬁnition. It is analogous
to Deﬁnition 3.3, except that the constant cis allowed to depend on time.
Deﬁnition 3.10 (Dominance with Coeﬃcients; Ryabko and Hutter, 2008, Def. 2) .The
measureQdominatesPwith coeﬃcients f(QP=f) iﬀQ(x)P(x)=f(jxj)for all
x2X.
IfQdominatesPwith coeﬃcients fandfgrows subexponentially ( f2o(exp)),
thenQweakly dominates Pby Remark 3.9.
Example 3.11 (Speed Prior) .Schmidhuber (2002) deﬁnes a variant of Solomonoﬀ’s
priorMthat penalizes programs by their running time, called the speed prior . Consider
the speed prior
SKt(x) :=X
p:xvU(p)2jpj
t(U;p;x )
wheret(U;p;x )is the number of time steps the Turing machine Utakes to produce x
from the program p. For any deterministic measure Pcomputable in time qwe have
SKt(x)P(x)=q(jxj). Therefore SKtdominatesPwith coeﬃcients O(q). Ifqis a
polynomial ( Pis computable in polynomial time), then it grows subexponentially and
thusSKtweakly dominates P. 3
Thesemimeasureloss SKt(x)P
a2XSKt(xa)inthespeedpriorisquitesubstantial:
since it takes at least nsteps to output a string of length n,M(x)jxjSKt(x).
28 Learning
Example 3.12 (Laplace Rule) .TheLaplace rule Lis deﬁned by
L(xtjx<t) :=#fi<tjxi=xtg
t+ #X:
ForX=f0;1gandr2[0;1]themeasure LdominatesBernoulli (r)withcoeﬃcients
f(t) =t#X+1(Ryabko and Hutter, 2008, Prop. 3). 3
Deﬁnition 3.13 (Local Absolute Continuity) .The measure Pislocally absolutely
continuous with respect to Q(QLP) iﬀQ(x) = 0impliesP(x) = 0for all ﬁnite
stringsx2X.
The notable diﬀerence between local absolute continuity and absolute continuity
is that Deﬁnition 3.6 talks about arbitrary measurable sets while Deﬁnition 3.13 only
talks about ﬁnite strings. The former is a much stronger property.
For example, every measure is locally absolutely continuous with respect to the
Lebesgue measure since (x)>0for all ﬁnite strings x2X.
Local absolute continuity is an extremely weak property. If it is not satisﬁed, we
have to be very careful when using Qfor prediction: then there is a positive probability
that we have to condition on a probability zero event.
Example 3.14 (The Minimum Description Length Principle; Grünwald, 2007) .Let
Mbe a countable set of probability measures on (X1;F1)and letK:M! [0;1]
be a function such thatP
P2M2K(P)1calledregularizer . Following notation from
Hutter (2009a), we deﬁne for each x2Xtheminimal description length model as
MDLx:= arg min
P2MflogP(x) +K(P)g:
logP(x)is the (arithmetic) code length of xgiven model P, andK(P)is a complexity
penalty for P. Given data x2X,MDLxis the measure P2Mthat minimizes the
total code length of data and model.
Note that the Lebesgue measure is not locally absolutely continuous with respect
to the MDL distribution Q(x) := MDLx(x): for somex2Xthe minimum description
P2Mmay assign probability zero to a continuation xy2X. 3
Remark 3.15 (MDL is Inductively Inconsistent; Leike and Hutter, 2014a, Cor. 13) .
The MDL estimator for countable classes as deﬁned in Example 3.14 is inductively
inconsistent: the selected model P2 Mcan change inﬁnitely often and thus the
limit limt!1MDLx<tmay not exist. This can be a major obstacle for using MDL for
prediction, since the model used for prediction has to be changed over and over again,
incurring the corresponding computational cost. 3
The following proposition establishes the relationship between our notions of com-
patibility; see also Figure 3.1 on page 46.
Proposition 3.16 (Relationships between Compatibilities) .
(a) IfQP, thenQP.
§3.3Martingales 29
(b) IfQP, thenQWP.
(c) IfQP, thenQdominatesPwith coeﬃcients ffor a constant function f.
(d) IfQdominatesPwith coeﬃcients fandf2o(exp), thenQWP.
(e) IfQWP, thenQLP.
Proof.(a) From Proposition 3.23 (a) and (b).
(b) From Proposition 3.23b and Kalai and Lehrer (1994, Prop. 3a).
(c) Follows immediately from the deﬁnitions.
(d) From Remark 3.9.
(e) Follows immediately from the deﬁnitions.
Note that the converse of Proposition 3.16d is false: in Remark 3.7 we deﬁned
a measure Pthat is absolutely continuous with respect to (and hence is weakly
dominated by ), but the coeﬃcients for P=grow exponentially on the string 1t.
This inﬁnite string has P-probability 0, but dominance with coeﬃcients demands the
inequalityQP=fto hold for all strings.
Remark 3.17 (Local Absolute Continuity ;Absolute Continuity) .DeﬁneP:=
Bernoulli (2=3)andQ:=Bernoulli (1=3). Both measures PandQare nonzero on
all cylinder sets: Q(x)3jxj>0andP(x)3jxj>0for everyx2X. Therefore
Qis locally absolutely continuous with respect to P. However, Qisnotabsolutely
continuous with respect to P: deﬁne
A:=
x2X!lim sup
t!11
tones(x1:t)1
2
:
The setAisF1-measurable since A=T1
n=1S
x2UnxwithUn:=fx2Xjjxj
nandones(x)jxj=2g, the set of all ﬁnite strings of length at least nthat have at
least as many zeros as ones. We have that P(A) = 0andQ(A) = 1, henceQis not
absolutely continuous with respect to P. 3
3.3 Martingales
The following two theorems state the connection between probability measures on inﬁ-
nite strings and martingales. For two probability measures PandQthe quotient Q=P
is a nonnegative P-martingale if Qis locally absolutely continuous with respect to P.
Conversely, for every nonnegative P-martingale there is a probability measure QLP
such that the martingale is P-almost surely a multiple of Q=P.
30 Learning
Theorem 3.18 (Measures7!Martingales; Doob, 1953, II§7 Ex. 3) .LetQandPbe
two probability measures on (X1;F1)such thatQis locally absolutely continuous with
respect toP. Then the stochastic process (Xt)t2N,
Xt(x) :=Q(x1:t)
P(x1:t)
is a nonnegative P-martingale with E[Xt] = 1.
Theorem3.19 (Martingales7!Measures) .LetPbe a probability measure on (X1;F1)
and let (Xt)t2Nbe a nonnegative P-martingale with E[Xt] = 1. There is a probability
measureQon(X1;F1)that is locally absolutely continuous with respect to Pand for
allx2X1and allt2NwithP(x1:t)>0,
Xt(x) =Q(x1:t)
P(x1:t):
The proofs for Theorem 3.18 and Theorem 3.19 are provided in the Appendix.
Example 3.20 (The Posterior Martingale) .Suppose we are interested in a hypothesis
HX1(such as the proposition ‘all ravens are black’ in Example 3.1). If Q(H) =P
P2Mw(P)P(H)is a Bayesian mixture over a set of probability distributions M
with prior weights w2M(see Example 3.4), then the posterior belief Q(Hjx) =P
P2Mw(Pjx)P(Hjx). The weights w(Pjx)are called posterior weights , and
satisfy the identity
w(Pjx) =w(P)P(x)
Q(x)(3.3)
since
Q(Hjx) =Q(H\x)
Q(x)
=1
Q(x)X
P2Mw(P)P(H\x)
=X
P2Mw(P)P(x)P(H\x)
Q(x)P(x)
=X
P2Mw(Pjx)P(Hjx):
According to Theorem 3.18 the posterior weight w(Pjx)is aQ-martingale with
expectation w(P). In particular, this means that the posterior weights converge Q-
almostsurelybythemartingaleconvergencetheorem(Theorem2.8). Since Qdominates
P, by Proposition 3.16a Pis absolutely continuous with respect to Qand hence the
posterior also converges P-almost surely. 3
Remark 3.21 (Martingales and Absolute Continuity) .While Theorem 3.18 trivially
also holds if Qis absolutely continuous with respect to P, Theorem 3.19 does not imply
thatQis absolutely continuous with respect to P.
§3.3Martingales 31
LetPandQbe deﬁned as in Remark 3.17. Consider the process X0(x) := 1,
Xt+1(x) :=(
2Xt;ifxt+1= 0;and
1
2Xt;ifxt+1= 1:
The process (Xt)t2Nis a nonnegative P-martingale since every XtisFt-measurable and
forx=y1:twe have
E[Xt+1jFt](y) =P(x0jx)2Xt(y) +P(x1jx)1
2Xt(y)
=1
32Xt(y) +2
31
2Xt(y) =Xt(y):
Moreover,
Q(x) =1
3ones(x)2
3zeros(x)=2
3ones(x)1
3zeros(x)2ones(x)2zeros(x)=P(x)Xt(y):
HenceXt(y) =Q(y1:t)=P(y1:t)P-almost surely. The measure Qis uniquely deﬁned
by its values on the cylinder sets, and as shown in Remark 3.17, Qis not absolutely
continuous with respect to P. 3
Theorem 3.22 (Radon-Nikodym Derivative) .IfQP, then there is a function
dP=dQ :X1![0;1)called the Radon-Nikodym derivative such that
Z
fdP =Z
fdP
dQdQ
for all measurable functions f.
This function dP=dQcan be seen as a density function of Pwith respect to the
background measure Q. Moreover, dP=dQis the limit of the martingale P=Q(Durrett,
2010, Sec. 5.3.3) which exists Q-almost surely according to Theorem 2.8.
ThefollowingpropositioncharacterizesthenotionsofcompatibilityfromSection3.2
in terms of the martingale Q=P.
Proposition 3.23 (Martingales and Compatibility) .The following relationships hold
betweenQ,P, and theP-martingale Yt:=Q(x1:t)=P(x1:t).
(a)QPif and only if Ytc>0for allt2N.
(b)QPif and only if P-almost surely Yt6!0ast!1.
(c)QdominatesPwith coeﬃcients fif and only if Yt1=f(t)for allt.
(d)QWPif and only if P-almost surely log(Yt+1=Yt)!0in Cesàro average.
(e)QLPif and only if P-almost surely Yt>0for allt2N.
Proof. (Yt)t2Nis aP-martingale according to Theorem 3.18.
(a)Q(x)cP(x)withc>0for allx2Xis equivalent to Q(x)=P(x)c>0for all
x2X.
32 Learning
(b) Proved by Hutter (2009a, Lem. 3i).
(c) Analogously to the proof of (a).
(d) IfQweakly dominates P, we getlogYt2o(t)according to Remark 3.9. To-
gether with Y0= 1we getlogYt=Pt1
k=0log(Yk+1=Yk)2o(t), therefore
t1Pt1
k=0log(Yk+1=Yk)!0ast!1. Conversely, if the Cesàro average con-
verges to 0, thent1logYt!0, hencelogYt2o(t).
(e) Letx2 Xbe any ﬁnite string. If QLPandP(x)>0, thenQ(x)>0,
and henceQ(x)=P(x)>0. Conversely, if P(x)>0thenYtis well-deﬁned, so if
Yjxj(x)>0thenQ(x)>0.
From Proposition 3.23b and Theorem 3.22 we get that QPif and only if the
Radon-Nikodym derivative dQ=dPis positive on a set of P-measure 1.
3.4 Merging
IfQis capable of learning, it should use the sequence xdrawn from Pto change its
opinions more in the direction of P. More precisely, we want Q(jx<t)P(jx<t)
for larget. In the rest of this chapter, we make this notion of closeness precise and
discuss diﬀerent conditions on Qthat are suﬃcient for learning.
Strong merging implies that the belief of anyhypothesis merges. This is very strong,
ashypothesescantalkabout tail events : eventsthatareindependentofanyﬁniteinitial
part of the inﬁnite sequence (such as the event Ain Remark 3.17). Weak merging only
considers hypothesis about the next couple of symbols, and almost weak merging allows
Qto deviate from Pin a vanishing fraction of the time. Much of this section is based
on Kalai and Lehrer (1994) and Lehrer and Smorodinsky (1996).
3.4.1 Strong Merging
Deﬁnition 3.24 (Strong Merging) .Qmerges strongly with PiﬀD1(P;Qjx<t)!0
ast!1P-almost surely.
The following theorem is the famous merging of opinions theorem by Blackwell and
Dubins (1962).
Theorem 3.25 (Absolute Continuity )Strong Merging; Blackwell and Dubins, 1962) .
IfPis absolutely continuous with respect to Q, thenQmerges strongly with P.
Example 3.26 (The Black Ravens 2; Rathmanner and Hutter, 2011, Sec. 7.4) .Recall
the black raven problem from Example 3.1. Let Qbe a learning distribution that
dominates the true distribution P, such as a Bayesian mixture (Example 3.4). By
Proposition 3.16a we get QP, and hence Qmerges strongly to Pby Theorem 3.25.
Thus we get as t!1thatP-almost surelyjQ(Hjx<t)P(Hjx<t)j!0for the
hypothesis Hthat ‘all ravens are black’ deﬁned in (3.1). Thus if all ravens are black
in the real world ( P(H) = 1),Qlearns this asymptotically ( Q(Hjx<t)!1). This is
§3.4Merging 33
the solution we desired: the learning distribution Qconverges to a true belief about an
inﬁnite set by only looking from a ﬁnite (but growing) number of data points. 3
The following is the converse of Theorem 3.25.
Theorem 3.27 (Strong Merging^Local Absolute Continuity )Absolute Continuity;
Kalai and Lehrer, 1994, Thm. 2) .IfQis locally absolutely continuous with respect to
PandQmerges strongly with P, thenPis absolutely continuous with respect to Q.
The following result shows that local absolute continuity is not required for strong
merging: recall that according to Example 3.14 the MDL distribution is not locally
absolutely continuous with respect to every Pfrom the classM.
Theorem 3.28 (Strong Merging for MDL; Hutter, 2009a, Thm. 1) .IfP2M, then
D1(P;MDLxjx)!0asjxj!1P-almost surely.
LetMbe a (possibly uncountable) set of probability measures on (X1;F1).
Ryabko (2010, Thm. 4) shows that if there is a Qthat merges strongly with every
P2M, then there is a Bayesian mixture over a countable subset of Mthat also
merges strongly with every P2M.
3.4.2 Weak Merging
In Deﬁnition 3.24 the supremum ranges over all measurable sets A2F1which includes
tail events. Instead, we may restrict the supremum to the next few symbols. This is
known as weak merging .
Deﬁnition 3.29 (Weak Merging) .Qweakly merges with Piﬀ for every d2N,
Dt+d(Q;Pjx<t)!0ast!1P-almost surely.
The following lemma gives an equivalent formulation of weak merging.
Lemma 3.30 (Lehrer and Smorodinsky, 1996, Rem. 5) .Qweakly merges with Pif
and only if Dt(Q;Pjx<t)!0ast!1P-almost surely.
Unfortunately, weak dominance is not suﬃcient for weak merging (Lehrer and
Smorodinsky, 1996, Ex. 10). We need the following stronger condition, that turns
out to be (almost) necessary. In the following, let Yt:=Q(x1:t)=P(x1:t)denote the
P-martingale from Proposition 3.23.
Theorem 3.31 (Kalai and Lehrer, 1994, Prop. 5a) .IfP-almost surely Yt+1=Yt!1,
thenQmerges weakly with P.
Example 3.32 (Laplace Rule 2) .Suppose we use the Laplace rule from Example 3.12
to predict a Bernoulli (r)process. By the strong law of large numbers, L(xtjx<t)!r
almost surely. Therefore we can use Theorem 3.31 to conclude that Lmerges weakly
with Bernoulli (r)for allr2[0;1]. (Note that strongly merging with every Bernoulli
process is impossible; Ryabko, 2010, p. 7) 3
34 Learning
The following is a converse to Theorem 3.31.
Theorem 3.33 (Kalai and Lehrer, 1994, Prop. 5b) .IfQmerges weakly with P, then
Yt+1=Yt!1inP-probability.
Unfortunately, weak dominance is not enough to guarantee weak merging.
Example3.34 (WeakDominance ;WeakMerging; RyabkoandHutter,2007,Prop.7) .
LetX=f0;1gand letfbe any arbitrarily slowly monotone growing function with
f(t)!1. DeﬁneP(11) := 1, the sequence (ti)i2Nsuch thatf(ti+1)2f(ti), and
Q(xtjx<t) :=8
>><
>>:1
2ift=tifor somei2N;
1ift6=tiandxt= 1;and
0otherwise:
NowQdominatesPwith coeﬃcients fby construction and Qweakly dominates Pif
fgrows subexponentially. However, jQ(1j1t)P(1j1t)j1=2for inﬁnitely many
t2N. 3
3.4.3 Almost Weak Merging
The following deﬁnition is due to Lehrer and Smorodinsky (1996, Def. 10).
Deﬁnition 3.35 (Almost Weak Merging) .Qalmost weakly merges with Piﬀ for every
d2N
1
ttX
k=1Dt+d(Q;Pjx<t)!0ast!1P-almost surely :
There is also an analogue of Lemma 3.30 for almost weakly merging in the sense
that we can equivalently set d= 0(Lehrer and Smorodinsky, 1996, Rem. 6).
Remark 3.36 (Weak Merging and Merging in KL-Divergence) .From Lemma 2.13
follows that weak merging is implied by KLd(P;Qjx)!0P-almost surely and
almost weak merging is implied byPt
k=1KL1(P;Qjx<k)2o(t)P-almost surely,
i.e.,KLt(P;Q)2o(t)(Ryabko and Hutter, 2008, Lem. 1). The converse is generally
false. 3
The following proposition relates the three notions of merging.
Proposition 3.37 (Strong Merging )Weak Merging)Almost Weak Merging) .If
Qmerges strongly with P, thenQmerges weakly with P. IfQmerges weakly with P,
thenQmerges almost weakly with P.
Proof.Follows immediately from the deﬁnitions.
Theorem 3.38 (Weak Dominance )Almost Weak Merging; Lehrer and Smorodinsky,
1996, Thm. 4) .IfQweakly dominates P, thenQmerges almost weakly with P.
§3.5Predicting 35
From Theorem 3.38 we get that the speed prior (Example 3.11) merges almost
weakly with any probability distribution estimable in polynomial time.
We also have the following converse to Theorem 3.38.
Theorem 3.39 (Almost Weak Merging )Weak Dominance; Lehrer and Smorodinsky,
1996, Cor. 7) .IfQis locally absolutely continuous with respect to P,Qmerges almost
weakly with P, andP-almost surely lim inft!1Yt+1=Yt>0, thenQweakly dominates
P.
3.5 Predicting
In Section 3.4 we wanted Qto acquire the correct beliefs about P. In this section, we
exploit the accuracy of our beliefs for predicting individual symbols. We derive bounds
on the number of errors Qmakes when trying to predict a string drawn from P.
Sincethedatadrawnfrom Pisstochastic, wecannotexpecttomakeaﬁnitenumber
of errors. Even the perfect predictor that knows Pgenerally makes an inﬁnite number
of errors. For example, trying to predict the Lebesgue measure (Example 2.15), in
expectation we make half an error in every time step. So instead we are asking about
the asymptotic error rate of a predictor based on Qcompared to a predictor based on
P, theprediction regret .
LetxR
tbe thet-th symbol predicted by the probability measure Raccording to the
maximum likelihood estimator:
xR
t:2arg max
a2XR(x<tajx<t): (3.4)
Theinstantaneous error of aR-based predictor is deﬁned as
eR
t:=(
0ifxt=xR
t;and
1otherwise:
and thecumulative error is
ER
t:=tX
k=1eR
k:
Note that both etandEtare random variables.
Deﬁnition 3.40 (Prediction Regret) .In time step ttheprediction regret isEQ
tEP
t
and theexpected prediction regret isEh
EQ
tEP
ti
.
More generally, we could also follow Hutter (2001b) and phrase predictive perfor-
mance in terms of loss: given a loss function `:XX! Rthe predictor Qsuﬀers an
(instantaneous) loss of `(xQ
t;xP
t)in time step t. If the loss function `is bounded in [0;1],
many of the results for prediction regret also hold for cumulative loss (for Section 3.5.2
we also need `(a;a) = 0for alla2X). In this chapter we chose to phrase the results
in terms of prediction errors instead of loss because prediction errors are conceptionally
simpler.
36 Learning
Example 3.41 (Good Prediction Regret ;Merging/Compatibility) .Good predic-
tion regret does not imply (weak/strong) merging or (weak) dominance: Let P:=
Bernoulli (1=3)andQ:=Bernoulli (1=4). ClearlyPandQdo not merge (weakly) or
(weakly) dominate each other. However, a P-based predictor always predicts 0, and so
does aQ-based predictor. Therefore the prediction regret EQ
tEP
tis always 0.3
Example 3.42 (Adversarial Sequence; Legg, 2006, Lem. 4) .No learning distribution
Qwill learn to predict everything. We can always deﬁne a Q-adversarial sequence z1:1
recursively according to
zt:=(
0ifQ(0jz<t)<1=2;and
1ifQ(0jz<t)1=2:
In every time step the probability that a Q-based predictor makes an error is at least
1=2, henceeQ
t1=2andEQ
tt=2. Butz1:1is a deterministic sequence, thus an
informed predictor makes zero errors. Therefore the prediction regret of Qon the
sequencez1:1is linear. 3
3.5.1 Dominance
WestartwiththepredictionregretboundsprovedbyHutter(2001b)incasethelearning
distribution Qdominates the true distribution P. In the following, let cPdenote the
constant from Deﬁnition 3.3.
Theorem 3.43 (Hutter, 2007b, Eq. 5 & 8) .For allPandQ,
q
EPEQ
nq
EPEPnp
2KLn(P;Q):
The following bound on prediction regret then follows easily, but it is a factor ofp
2
worse than the bound stated by Hutter (2005, Thm. 3.36).
Corollary 3.44 (Expected Prediction Regret) .For allPandQ,
0EPh
EQ
nEP
ni
2KLn(P;Q) + 2q
2KLn(P;Q)EPEPn:
Proof.From Theorem 3.43 we get
EPh
EQ
nEP
ni
=q
EPEQ
n+q
EPEPnq
EPEQ
nq
EPEPn
q
EPEQ
n+q
EPEPnp
2KLn(P;Q)
p
2KLn(P;Q) +q
EPEPn+q
EPEP
tp
2KLn(P;Q)
= 2KLn(P;Q) + 2q
2KLn(P;Q)EPEPn:
§3.5Predicting 37
IfQdominatesP, then we have KLn(P;Q)lncP:
KLn(P;Q) =X
x2XnP(x) logP(x)
Q(x)X
x2XnP(x) log1
cP=logcP(3.5)
This invites the following corollary.
Corollary 3.45 (Prediction Regret for Dominance; Hutter, 2005, Cor. 3.49) .IfQ
dominatesP, then the following statements hold.
(a)EPEQ
1is ﬁnite if and only if EPEP
1is ﬁnite.
(b)q
EPEQ
1p
EPEP12O(1)
(c)EPEQ
t=EPEP
t!1forEPEP
t!1.
(d)EPh
EQ
tEP
ti
2Op
EPEP
t
.
If the true distribution Pis deterministic, we can improve on these bounds:
Example 3.46 (Predicting a Deterministic Measure) .Suppose we are predicting a
deterministic measure Pthat assigns probability 1to the inﬁnite string x1:1. IfPis
dominated by Q, the total expected prediction regret EPEQ
1is bounded by2 lncP
by Corollary 3.44. This is easy to see: every time we predict a wrong symbol a6=xt,
thenQ(ajx<t)Q(xtjx<t), soQ(xtjx<t)1=2. Therefore YtYt1=2and by
dominance YtcP. Hence a prediction error can occur at most logcPtimes. 3
Generally, the O(EPEP
t)bounds on expected prediction regret given in Corol-
lary 3.45 are essentially unimprovable:
Example 3.47 (Lower Bounds on Prediction Regret) .SetX:=f0;1gand consider
the uniform measure from Example 2.15. For each time step t, we have(0jx<t) =
(1jx<t) = 1=2, so the argmax in (3.4) ties and hence it does not matter whether
we predict 0or1. We take two predictors PandQ, wherePalways predicts 0andQ
always predicts 1. LetZt:=EQ
tEP
t. Since their predictions never match, Ztis an
ordinary random walk with step size 1. We have (Weisstein, 2002)
lim sup
t!1EP[EQ
tEP
t]p
t=p
2=
and for the law of the iterated logarithm (Durrett, 2010, Thm. 8.8.3)
lim sup
t!1EQ
tEP
tp2tlog logt= 1P-almost surely :
Both bounds are known to be asymptotically tight. 3
While Example 3.47 shows that the bounds from Corollary 3.45 are asymptotically
tight, they are misleading because in most cases, we can do much better. According
38 Learning
to the following theorem, the worst case bounds are only attained if P(xtjx<t)is
suﬃciently close to 1=2.
Theorem 3.48 (Expected Prediction Regret for Nonuniform Measures) .IfX=f0;1g
and there is an ">0such thatjP(xtjx<t)1=2j"for allx1:t2X, then
EPh
EQ
tEP
ti
KLt(P;Q)
":
Proof.Recall the deﬁnition of entropy in nats:
Ent(p) :=plnp(1p) ln(1p):
The second order Taylor approximation of Entat1=2is
f(p) = ln 22(p1
2)2:
One can check that f(p)Ent(p)for all 0p1. Deﬁnep:=P(xP
tjx<t)1=2
andq:=Q(xQ
tjx<t)1=2to ease notation. Consider the function
g(p;q;" ) :=p(1p)"1
plnp
1q+ (1p) ln1p
q
which is strictly increasing as qdecreases, so from q1=2we get
g(p;q;" )2p1"1ln 2 +"1Ent(p) ln 2
2p1"1ln 2 +"1f(p) ln 2
= 2p1"12(p1
2)2;
which decreases as pincreases, hence it is maximized for p= 1=2 +",
g(p;q;" )2""12"2= 0
Thereforegis nonpositive. If xQ
t=xP
t, the one-step error is 0. Otherwise EP[etj
x<t] =p(1p)andg(p;q;" ) =EP[etjx<t]"1KL1(P;Qjx<t), so we get
EP[etjx<t]"1KL1(P;Qjx<t). Summing this from t= 1tonyields the claim.
Eh
EQ
nEP
ni
"1KLn(P;Q):
3.5.2 Absolute Continuity
Theorem 3.49 (Prediction with Absolute Continuity) .IfQP, then
q
EQ
tq
EP
tOp
log logt
P-almost surely :
The proof idea is inspired by Miller and Sanchirico (1999). We think of PandQas
two players in a zero-sum betting game. In every time step t, the players will make a
bet on the outcome of xt. Ifxt=xQ
t6=xP
t, thenQwins $1 from P, ifxt=xP
t6=xQ
t,
§3.5Predicting 39
thenQloses $1 to P. Otherwise xQ
t=xP
torxQ
t6=xt6=xP
tand neither player gains
or loses money. Since Qpredicts according to the maximum likelihood principle (3.4),
it is rational to accept the bet from Q’s perspective. In Q’s eyes, the worst case is a
fair bet, so Qwill not lose more money than it would lose on a random walk. The law
of the iterated logarithm gives a Q-probability one statement about this bound, which
transfers to Pby absolute continuity.
Proof.Deﬁne the stochastic process Zt:=EQ
tEP
t. Since EQ[eR
t] =Q(xR
tjx<t), we
get
EQ[Zt+1jFt] =Q(xQ
tjx<t)Q(xP
tjx<t) +Zt
Q(xQ
tjx<t)Q(xQ
tjx<t) +Zt=Zt;
hence (Zt)t2Nis aQ-submartingale. In the worst case (for Q),(Zt)t2Nis just a random
walk with step size 1. ButZtcan only move if QandPpredict a diﬀerent symbol. If
this happens, at least one of them makes an error. Let mtbe the number of steps Zt
has moved ( Zt+16=Zt). ThenmtEQ
t+EP
tandmtt. By the law of the iterated
logarithm (Durrett, 2010, Thm. 8.8.3),
lim inf
t!1Ztp2mtlog logmt=1
Q-almost surely. We deﬁne the event
A:=n
9C8t:ZtCp
mtlog logmto
:
ThenQ(A) = 1, henceP(A) = 1by absolute continuity.
EQ
tEP
t=ZtCq
(EQ
t+EP
t) log logtCq
EQ
t+q
EP
tp
log logt
Dividingbothsidesbyq
EQ
t+p
EP
tyieldsthatthereisa P-almostsurelyﬁniterandom
variableCsuch thatq
EQ
tp
EP
tCplog logt.
This invites the following immediate corollary.
Corollary 3.50 (Prediction Regret for Absolute Continuity) .IfQP, then
EQ
tEP
t2O
log logt+q
EP
tlog logt
P-almost surely :
Proof.Analogously to the proof of Corollary 3.44.
While Corollary 3.50 establishes an almost sure prediction regret bound, it is dif-
ferent from the bound on expected prediction regret from Corollary 3.44; bounds on
E[EQ
tEP
t]are incomparable to almost sure bound given in Theorem 3.49: for a se-
quence of nonnegative (unbounded) random variables convergence in mean does not
40 Learning
imply almost sure convergence (Stoyanov, 2013, Sec. 14.7) or vice versa (Stoyanov,
2013, Sec. 14.8ii).
Weproceedtoestablishanimprovedpredictionregretboundincase Pisnonuniform
analogously to Theorem 3.48.
Theorem 3.51 (Prediction Regret for Nonuniform Measures) .IfQP,X=f0;1g,
and there is an ">0such that with P-probability 1
jP(xtjx<t)1=2j"
for allt2N, thenP-almost surely EQ
tEP
t2O(1).
Proof.IfjP(xtjx<t)1=2j", then for large enough t,Qwill have merged with P
(Theorem 3.25) and hence jQ(xtjx<t)1=2j"=2inﬁnitely often.
ThusZthas an expected gain of "=2if the predictors disagree. Therefore Zt!1
Q-almost surely. Consequently, the set
A:=f9t08tt0:Zt0g
hasQ-measure 1. By absolute continuity, it also has P-measure 1, hence there is a
P-almost surely ﬁnite random variable Csuch that for all t,ZtC.
There is another argument that we could use to show that under the condition of
Theorem 3.51 EQ
tEP
tis almostsurely ﬁnite: If Pis absolutelycontinuous withrespect
toQ, thenQmerges strongly with Pand henceQmerges weakly with P. Therefore
almost surely there is a t0such that for all tt0we havejQ(xP
tjx<t)P(xP
tjx<t)j<
", thusxQ
t=xP
tfortt0.
3.5.3 Dominance with Coeﬃcients
Lemma 3.52 (KL Divergence and Dominance With Coeﬃcients) .IfQdominatesP
with coeﬃcients f, then KLt(Q;P)lnf(t).
Proof.Analogous to (3.5).
This lets us derive an analogous regret bound to Corollary 3.44.
Corollary 3.53 (Expected Prediction Regret for Dominance With Coeﬃcients) .IfQ
dominatesPwith coeﬃcients f, then
EPh
EQ
nEP
ni
2 lnf(t) + 2q
2EPEPnlnf(t):
Proof.Apply Lemma 3.52 to Corollary 3.44.
For weak dominance we get sublinear prediction regret.
Corollary 3.54 (Sublinear Prediction Regret for Weak Dominance) .IfQweakly dom-
inatesP, then EP[EQ
nEP
n]2o(t).
§3.6Learning with Algorithmic Information Theory 41
Proof.By Remark 3.9 lnf2o(t). Applying Corollary 3.53 we get
EPh
EQ
nEP
n
2o(t) + 2q
2EPEPno(t)2o(t) + 2p
2O(t)o(t)2o(t):
3.6 Learning with Algorithmic Information Theory
Algorithmic information theory provides a theoretical framework to apply the probabil-
ity theory results from the previous sections. In the following we discuss Solomonoﬀ’s
famous theory of induction (Section 3.6.1), the speed prior (Section 3.6.2), and learning
with a universal compression algorithm (Section 3.6.3).
3.6.1 Solomonoﬀ Induction
Solomonoﬀ (1964, 1978) proposed a theory of learning, also known as universal in-
ductionorSolomonoﬀ induction . It encompasses Ockham’s razor by favoring simple
explanations over complex ones, and Epicurus’ principle of multiple explanations by
never discarding possible explanations. See Rathmanner and Hutter (2011) for a very
readable introduction to Solomonoﬀ’s theory and its philosophical motivations and
Sterkenburg (2016) for a critique of its optimality.
At the core of this theory is Solomonoﬀ’s distribution M, as deﬁned in Example 3.5.
SinceMdominates all lower semicomputable semimeasures, we get all the merging and
prediction results from Section 3.4 and Section 3.5: when drawing a string from any
computable measure P,Marrives at the correct belief for any hypothesis.
Corollary 3.55 (Strong Merging for Solomonoﬀ Induction) .Mmerges strongly with
every computable measure.
Proof.From Proposition 3.16a and Theorem 3.25.
Corollary 3.56 (Expected Prediction Regret for Solomonoﬀ Induction) .For all com-
putable measures P,
EP
EM
tEP
t
K(P) ln 4 +q
2EPEP
tK(P) ln 16:
Proof.From Corollary 3.44 and cP= 2K(P).
Remark 3.57 (Converging Fast and Slow) .The convergence of Mto a computable P
is fast in the sense of Corollary 3.56: Mcannot make many more prediction errors than
Pin expectation. When predicting an inﬁnite computable sequence x1:1, the total
number of prediction errors is bounded by jpj2 ln 21:4jpjwherepis a program that
generatesx1:1(Example 3.46).
The convergence of MtoPis also slow in the sense that M(xtjx<t)!1slower
than any computable function since 1M(xtjx<t)2minntK(n)for allt. 3
The bound from Corollary 3.56 is not optimal. Even if we knew the program p
generating the sequence x1:1, there might be a shorter program p0that computes x1:1;
42 Learning
hence the improved bound EM
1jp0j2 ln 2also holds. Since Kolmogorov complexity is
incomputable, we can’t ﬁnd the ‘best’ bound algorithmically.
Solomonoﬀ induction may even converge on some incomputable measures.
Example 3.58 (MConverges on Some Incomputable Measures) .Letrbe an incom-
putable real number. Then the measure P:=Bernoulli (r)is not computable and Mis
not absolutely continuous with respect to P: for
A:=n
x2X1lim
t!1ones(x1:t) =ro
we haveP(A) = 1butM(A) = 0. SinceMLPwe get from Theorem 3.27 that M
does not merge with P. Nevertheless, Mstill succeeds at prediction because it domi-
nates Bernoulli (q)for each rational qand the rationals are dense around r. According
to Lehrer and Smorodinsky (1996, Lem. 3), this implies that Mweakly dominates P
and by Theorem 3.38 Malmost weakly merges to P. 3
The fact that Mdoes not merge strongly with every Bernoulli (r)process is not
a failure of Solomonoﬀ’s prior. Ryabko (2010, p. 7) shows that for the class of all
Bernoulli measures there is no probability measure that merges strongly with each of
them.
The deﬁnition of Mhas only one parameter: the choice of the universal Turing
machine. The eﬀect of this choice on the function Kcan be uniformly bounded by
a constant by the invariance theorem (Li and Vitányi, 2008, Thm. 3.1.1). Hence the
choice of the UTM changes the prediction regret bound from Corollary 3.56 only by
a constant. This constant can be large, preventing any ﬁnite-time guarantees that are
independent of the UTM. However, asymptotically Solomonoﬀ induction succeeds even
for terrible choices of the UTM.
The Solomonoﬀ normalization MnormofMis deﬁned according to Deﬁnition 2.16.
WhileMnormdominatesMaccording to Lemma 2.17 and thus every lower semicom-
putable semimeasure, in some respects, Mnormbehaves a little diﬀerently from M.
Another way to complete the semimeasure Minto a measure is given in the following
example.
Example 3.59 (The Measure Mixture; Gács, 1983, p. 74) .Themeasure mixture Mis
deﬁned as
M(x) := lim
n!1X
y2XnM(xy): (3.6)
It is the same as Mexcept that the contributions by programs that do not produce
inﬁnite strings are removed: for any such program p, letkdenote the length of the
ﬁnite string generated by p. Then forjxyj> k, the program pdoes not contribute to
M(xy), hence it is excluded from M(x).
Similarly to M, the measure mixture Mis not a (probability) measure since M()<
1; but in this case normalization (2.2) is just multiplication with the constant 1=M(),
leading to the normalized measure mixture Mnorm. 3
§3.6Learning with Algorithmic Information Theory 43
Eventhough Mmergesstronglywithanycomputablemeasure PwithP-probability
1, LattimoreandHutter(2013,2015)showthatgenerallyitdoesnotholdforallMartin-
Löf random sequences (which also form a set of P-probability 1). Hutter and Muchnik
(2007, Thm. 6) construct non-universal lower semicomputable semimeasures that have
this convergence property for all P-Martin-Löf random sequences. For inﬁnite nonran-
dom sequences whose bits are selectively predicted by some total recursive function,
Lattimore et al. (2011, Thm. 10) show that the normalized Solomonoﬀ measure Mnorm
converges to 1on the selected bits. This does not hold for the unnormalized measure
M(Lattimore et al., 2011, Thm. 12).
3.6.2 The Speed Prior
Solomonoﬀ’s prior Mis incomputable (Theorem 6.3); a computable alternative is the
speed prior from Example 3.11. In this section we state merging and prediction results
forSKt, aspeedpriorintroducedbyFilanetal.(2016)formallydeﬁnedinExample3.11.
It is slightly diﬀerent from the speed prior deﬁned by Schmidhuber (2002), but for the
latter no compatibility properties are known for nondeterministic measures.
Deﬁnition 3.60 (Estimable in Polynomial Time) .A function f:X!Risestimable
in polynomial time iﬀ there is a function g:X!Rcomputable in polynomial time
such thatf=g.
For a measure Pestimable in polynomial time the speed prior SKtdominatesP
with coeﬃcients polynomial in jxjlogP(x)(Filan et al., 2016, Eq. 12). Thus SKt
weakly dominates Pand we get the following results.
Corollary 3.61 (Almost Weak Merging for SKt).SKtalmost weakly merges with every
measure estimable in polynomial time.
Proof.From Theorem 3.38 and Filan et al. (2016, Eq. 12) since logPdoes not grow
superexponentially P-almost surely.
Corollary 3.62 (Expected Prediction Regret for SKt; Filan et al., 2016, Thm. 9) .For
all measures Pestimable in polynomial time,
EP
ESKtnEP
n
2O
logn+q
EPEP1logn
:
Proof.From Corollary 3.44 and Filan et al. (2016, Eq. 14).
3.6.3 Universal Compression
Solomonoﬀ’sdistributioncanbeapproximatedusingastandardcompressionalgorithm,
motivated by the similarity M(x)2Km(x), whereKmdenotes monotone Kolmogorov
complexity. The function Kmis auniversal compressor , compressing at least as well
as any other recursively enumerable program.
Gács (1983) shows that the similarity M2Kmis not an equality. However, the
diﬀerence between logMandKmis very small: the best known lower bound is due
44 Learning
to Day (2011) who shows that Km(x)>logM(x) +O(log logjxj)for inﬁnitely many
x2X.
Nevertheless, 2Kmdominates every computable measure (Li and Vitányi, 2008,
Thm. 4.5.4 and Lem. 4.5.6ii(d); originally proved by Levin, 1973). Hence all the strong
results that hold for Solomonoﬀ induction (prediction regret and strong merging) also
hold for compression: we apply Theorem 3.25 and Corollary 3.44 to get the following
results. See Hutter (2006a) for further discussion on using the universal compressor
Kmfor learning.
Corollary 3.63 (StrongMergingforUniversalCompression) .The distribution 2Km(x)
merges strongly with every computable measure.
Corollary 3.64 (Expected Prediction Regret for Universal Compression) .ForQ(x) :=
2Km(x)and for all computable measures Pthere is a constant cPsuch that
EPh
EQ
tEP
ti
cP+q
cPEPEP
t:
This provides a theoretical basis for viewing compression as a general purpose learn-
ingalgorithm. Inthisspirit, the Hutter prize isawardedforthecompressionofa100MB
excerpt from the English Wikipedia (Hutter, 2006c).
Practical compression algorithms (such as the algorithm by Ziv and Lempel (1977)
used in gzip) are not universal. Hence they do not dominate every computable distri-
bution. As with the speed prior, what matters is the rate at which Yt=Q(x1:t)=P(x1:t)
goes to 0, i.e., does the compressor weakly dominate the true distribution in the sense
of Deﬁnition 3.8?
Veness et al. (2015) successfully apply the Lempel-Ziv compression algorithm as a
learning algorithm for reinforcement learning; however, some preprocessing of the data
is required. More remotely, Vitányi et al. (2009) use standard compression algorithms
to classify mammal genomes, languages, and classical music.
3.7 Summary
Ultimately, whether learning succeeds depends on the rate at which the nonnegative
P-martingale Q=Pgoes to 0(when drawing from P). IfQ=Pdoes not converge to zero,
thenQmerges strongly with Pand thus arrives at correct beliefs about any hypothesis,
including tail events. If Q=Pconverges to zero subexponentially, then Qmerges almost
weakly with Pand thus asymptotically has incorrect beliefs about the immediate future
only a vanishing fraction of the time.
Corollary 3.44 bounds the expected prediction regret by the KL-divergence between
PandQplus ap
EPEP
tterm. The KL-divergence is in turn bounded by the rate at
whichQ=Pgoes to zero. It is constant if QdominatesPand bounded by lnfifQ
dominatesPwith coeﬃcients f. IfQweakly dominates P, then the KL-divergence is
sublinear. We also derived bounds on the prediction regret for absolute continuity (Sec-
tion 3.5.2). Remarkably, the bounds are only log logtworse than the bound we get from
dominance. Moreover, they hold almost surely instead of in expectation.
§3.7Summary 45
name symbol deﬁned in property
Bayesian mixture  Example 3.4 dominates every P2M
Solomonoﬀ prior M Example 3.5 dominates every lower semi-
computable semimeasure
universal compression 2KmEquation 2.1 dominates every computable
measure
speed prior SKtExample 3.11 weak dominates every mea-
sure estimable in polytime
Laplace rule LExample 3.12 merges weakly with every
Bernoulli process
MDL MDLxExample 3.14 merges strongly with every
P2M
Table 3.1: Examples of learning distributions discussed in this chapter and their
properties.
compatibility
ofPandQmartingale merging prediction regret
QP Y tc>0 strong merging 2 lnc+ 2p
2EPEP
tlnc
QP Y t6!0 strong mergingO
log logt+p
EPEP
tlog logt
Yt+1=Yt!1 weak merging o(t)
QP=f Y t1=f(t) 2 ln f(t) + 2p
2EPEP
tlnf(t)
QWP log(Yt+1=Yt)!1
in Cesàro averagealmost weak
mergingo(t)
QLP Y t>0 O(t)
Table 3.2: Summary on properties of learning. The ﬁrst column lists diﬀerent notions
of compatibility introduced in Section 3.2; the second column lists properties of the
P-martingale Yt:=Q(x1:t)=P(x1:t)from Section 3.3; the third column lists diﬀerent
notions of merging discussed in Section 3.4; the fourth column states the bounds on
the prediction regret (in expectation and almost surely respectively) from Section 3.5.
Figure 3.1 illustrates the origin of the results.
46 Learning
strong merging
weak merging
almost
weak
mergingQP
QP
QP=f
QWP
QLPYtc>0
Yt6!0
Yt+1=Yt!1
log(Yt+1=Yt)!0
in Cesàro average
Yt>0f2o(t)Hutter (2009a, Lem. 3i)Blackwell and Dubins (1962)
Kalai and Lehrer (1994, Thm. 2), L
Lehrer and Smorodinsky
(1996, Thm. 4)
Lehrer and Smorodinsky
(1996, Cor. 7),
Land
lim inf
t!1Yt+1=Yt>0Kalai and Lehrer (1994, Prop. 5)
Figure 3.1: Properties of learning and their relationship. We use Yt:=
Q(x1:t)=P(x1:t). An arrow between two statements means that one statement implies
the other. The transitive property of implications is not made explicit. The source of
the result is indicated on the arrow, sometimes together with a side condition. If no
source is given, then the relationship is easy and a proof can be found in this chapter.
§3.7Summary 47
Next, we showed that thep
EPEP
tterm is generally unimprovable (Example 3.47).
However, it comes only from predicting measures that assign probabilities close to 1=2.
If we can bound Paway from 1=2, then thep
EPEP
tterm disappears (Theorem 3.48
and Theorem 3.51).
Table 3.1 lists our learning distributions. The Bayesian mixture is the strongest
since it dominates every measure from the given class M(Example 3.4). The minimum
description length model MDLxdoes not have this property, yet it still merges strongly
with every measure from the class (Example 3.14 and Theorem 3.28). The Laplace rule
is only useful for learning i.i.d. measures; it merges weakly with every Bernoulli pro-
cess (Example 3.12 and Example 3.32). We also discussed some learning distributions
from algorithmic information theory. Solomonoﬀ’s prior is a Bayesian mixture over
all lower semicomputable semimeasures (Example 3.5 and Wood et al., 2011). Like
the universal compressor it dominates and hence merges strongly with all computable
measures. The speed prior dominates all probability measures estimable in polynomial
time with polynomial coeﬃcients (Example 3.11), and thus merges weakly with each of
them.
Table 3.2 summarizes the results from this chapter and Figure 3.1 illustrates their
logical relationship and their origin.
We conclude this chapter with a paradox from the philosophy of science.
Remark 3.65 (The Paradox of Conﬁrmation) .Recall the black raven problem intro-
duced in Example 3.1; the hypothesis ‘all ravens are black’ is denoted with H. The
paradox of conﬁrmation , also known as Hempel’s paradox (Hempel, 1945), relies on the
following three principles.
•Nicod’s criterion (Nicod, 1961, p. 67): observing an Fthat is aGincreases our
belief in the hypothesis that all Fs areGs.
•The equivalence condition : logically equivalent hypotheses are conﬁrmed or dis-
conﬁrmed by the same evidence.
•The paradoxical conclusion : a green apple conﬁrms H.
Theargumentgoesasfollows. Thehypothesis Hislogicallyequivalenttothehypothesis
H0that all non-black objects are non-ravens. According to Nicod’s criterion, any non-
black non-raven, such as a green apple, conﬁrms H0. But then the equivalence condition
entails the paradoxical conclusion.
The paradox of conﬁrmation has been discussed extensively in the literature on the
philosophy of science (Hempel, 1945; Good, 1960; Mackie, 1963; Good, 1967; Hempel,
1967; Maher, 1999; Vranas, 2004); see Swinburne (1971) for a survey. Support for
Nicod’s criterion is not uncommon (Mackie, 1963; Hempel, 1967; Maher, 1999) and no
consensus is in sight.
A Bayesian reasoner might be tempted to argue that a green apple doesconﬁrm the
hypothesisH, but only to a small degree, since there are vastly more non-black objects
than ravens (Good, 1960). This leads to the acceptance of the paradoxical conclusion,
48 Learning
and this solution to the conﬁrmation paradox is known as the standard Bayesian solu-
tion. Vranas (2004) shows that this solution is equivalent to the assertion that blackness
is equally probable regardless of whether Hholds:P(blackjH)P(black ).
The following is a very concise example against the standard Bayesian solution by
Good (1967): There are two possible worlds, the ﬁrst has 100 black ravens and a million
other birds, while the second has 1000 black ravens, one white raven, and a million other
birds. Now we draw a bird uniformly at random, and it turns out to be a black raven.
Contrary to what Nicod’s criterion claims, this is strong evidence that we are in fact in
the second world, and in this world non-black ravens exist.
For another, more intuitive example: Suppose you do not know anything about
ravens and you have a friend who collects atypical objects. If you see a black raven
in her collection, surely this would not increase your belief in the hypothesis that all
ravens are black.
In Leike and Hutter (2015d) we investigate the paradox of conﬁrmation in the con-
text of Solomonoﬀ induction. We show that the paradoxical conclusion is avoided
because Solomonoﬀ induction violates Nicod’s criterion: There are time steps when
(counterfactually) observing a black raven disconﬁrms the hypothesis that all ravens
are black. When predicting a deterministic computable sequence Nicod’s criterion is
even violated inﬁnitely often. However, if we normalize Solomonoﬀ’s prior and ob-
serve a deterministic computable inﬁnite string, Nicod’s criterion is violated at most
ﬁnitely many times. These results are independent of the choice of the universal Turing
machine.
We must conclude that violating Nicod’s criterion is not a fault of Solomonoﬀ in-
duction. Instead, we should accept that for Bayesian reasoning Nicod’s criterion, in its
generality, is false! Quoting the great Bayesian master Jaynes (2003, p. 144):
In the literature there are perhaps 100 ‘paradoxes’ and controversies which
are like this, in that they arise from faulty intuition rather than faulty
mathematics. Someone asserts a general principle that seems to him intu-
itively right. Then, when probability analysis reveals the error, instead of
taking this opportunity to educate his intuition, he reacts by rejecting the
probability analysis. 3
Chapter 4
Acting
I ought never to act except in such a way that I could also will that my maxim should
become a universal prior. — Immanuel Kant
Recall our decomposition of intelligence into learning andactingfrom Equation 1.1.
The previous chapter made the notion of learning precise and provided several examples
of learning distributions for the non-i.i.d. setting (see Table 3.1). Learning is passive:
there is no interaction with the data-generating process. In this chapter we transition
into the active setting: we consider an agentacting in an unknown environment in
order to achieve a goal. In our case, this goal is maximizing reward; this is known as
reinforcement learning . Where this reward signal originates does not concern us here.
In this thesis we consider is the general reinforcement learning problem in which we
do not make several of the typical simplifying assumptions (see Table 1.1). Environ-
mentsareonlypartiallyobservable, haveinﬁnitelymanystates, andmightcontaintraps
from which the agent cannot escape. The context for making decision is the agent’s
entire history; its behavior is given by a policythat speciﬁes how the agent behaves in
any possible situation.
A central quantity in reinforcement learning is the value function . The value func-
tionquantiﬁestheexpectedfuturediscountedreward. Sincetheagentseekstomaximize
reward, it aims to adopt a policythat has high value. Since the agent’s environment
is unknown to the agent, learning the value function is part of the challenge; otherwise
we would call this planning.
If our agent is capable of learning in the sense of Chapter 3, then it learns the
value of its own policy ( on-policy value convergence ). However, generally the agent
does not learn to predict the value of counterfactual actions, actions that it does not
take. Learning oﬀ-policy is hard because the agent receives no evidence about what
would have happened on counterfactual actions. Nevertheless, oﬀ-policy learning is
highly desirable because we want the agent to be conﬁdent that the policy it is currently
following is in fact the best one; we want it to accurately predict that the counterfactual
actions have less value.
This brings us back to the central theme of reinforcement learning: the tradeoﬀ
between exploration andexploitation . Asymptotically the agent needs to focus on
exploitation, i.e., take the actions that it thinks yield the highest expected rewards. If
theagentexploresenough,thenallactionsareon-policybecausetheyareallactionsthat
49
50 Acting
agent environment at
et= (ot;rt)
Figure 4.1: The dualistic agent model. At every time step t, the agent outputs an
actionatand subsequently receives a percept etconsisting of an observation otand a
real-valued reward rt. The agent’s policy is a function that maps a history æ<tto the
next action at, and the environment is a function that maps a history and an action
to the next percept et.
the agent sometimes takes. Then on-policy learning ensures that the agent understands
the consequences of every action and can conﬁdently choose the best action. Eﬀective
exploration is performed by knowledge-seeking agents ; these agents ignore the rewards
and just focus on exploration.
This chapter introduces the central concepts of general reinforcement learning. It
is mostly based on Hutter (2005) and Lattimore (2013). Section 4.1 speciﬁes the gen-
eral reinforcement learning problem, discusses discounting (Section 4.1.1), our implicit
assumptions (Section 4.1.2), and typical environment classes (Section 4.1.3). Sec-
tion 4.2 discusses the value function and its properties. In Section 4.3 we introduce the
agents: AIXI (Section 4.3.1), knowledge-seeking agents (Section 4.3.2), BayesExp (Sec-
tion 4.3.3), and Thompson sampling (Section 4.3.4).
4.1 The General Reinforcement Learning Problem
In reinforcement learning, an agent interacts with an environment: at time step t2N
the agent takes an actionat2Aand subsequently receives a perceptet= (ot;rt)2E
consisting of an observation ot2Oand arewardrt2R. This cycle then repeats for
time stept+ 1(see Figure 4.1).
Ahistoryis an element of (AE )and lists the actions the agent took and the
percepts it received. We use æ2AE to denote one interaction cycle, and æ<t=
æ1æ2:::æt1to denote a history of length t1. For our agent, the history is a
suﬃcient statistic about the past and in general reinforcement learning there is no
simpler suﬃcient statistic.
For example, consider the agent to be a robot interacting with the real world.
Its actions are moving the motors in its limbs and wheels and sending data packets
over a network connection. Its observations are data from cameras and various other
sensors. Therewardcouldbeprovidedeitherbyahumansupervisororthroughareward
module that checks whether a predeﬁned goal has been reached. The history is the
collection of all the data it received and emitted in the past. The division of the robot’s
interaction with the environment into discrete time steps might seem a bit unnatural
at ﬁrst since the real world evolves according to a continuous process. However, note
that the electronic components used in robots operate at discrete frequencies anyway.
§4.1The General Reinforcement Learning Problem 51
In order to specify how the agent behaves in any possible situation, we deﬁne a
policy: apolicyisafunction : (AE )!Amappingahistory æ<ttoadistribution
over actions (jæ<t)taken after seeing this history. Usually we do not distinguish
between agent and policy. An environment is a function : (AE )A! E
mapping a history æ<tand an action atto a distribution (jæ<tat)over the percepts
received after the history æ<tand actionat. We useto denote the true environment.
Equivalently, Hutter (2005) deﬁnes environments as chronological contextual semi-
measures .1Acontextual semimeasure takes a sequence of actions a1:1as input and
returns a semimeasure (ka1:1)overE]. A contextual semimeasure ischronological
iﬀperceptsattime tdonotdependonfutureactions, i.e., (e1:tka1:1) =(e1:tka0
1:1)
whenevera1:t=a0
1:t. For chronological contextual semimeasures we write (e1:tka1:t)
instead of(e1:tka1:1). The two deﬁnition can be translated using the identities
(e1:tka1:t) =tY
k=1(ekjæ<kak)and(etjæ<tat) =(e1:tka1:t)
(e<tka<t):(4.1)
If the policy always assigns probability 1to one of the actions, then is called
deterministic . Likewise, if the environment always assigns probability 1to one of the
percepts, then is called deterministic . For deterministic policies and environments
we also use the notation at=(æ<t)andet=(æ<tat). A deterministic policy is
consistent with history æ<tiﬀak=(æ<k)for allk < t. Likewise, a deterministic
environment isconsistent with history æ<tiﬀek=(æ<kak)for allk<t.
Deﬁnition 4.1 (History Distribution) .An environment together with a policy
induces a history distribution
(æ<t) :=tY
k=1(akjæ<k)(ekjæ<kak):
We denote an expectation with respect to the history distribution withE
.
Thehistorydistributionisa(semi)measureon (AE )1. Inthelanguageofmeasure
theory, our -algebra is the -algebraF1generated by the cylinder sets introduced in
Section 2.1. The ﬁltration (Ft)t2Nformalizes that at time step twe have seen exactly
the history æ<t(we use the -algebraFt1). To simplify notation and help intuition,
wesimplyconditionexpectationsandprobabilitymeasureswiththehistory æ<tinstead
ofFt1and sweep most of the measure-theoretic details under the rug.
With these preliminaries out of the way, we can now specify the general reinforce-
ment learning problem .
Problem 4.2 (General Reinforcement Learning Problem) .Given an arbitrary class of
environmentsM, choose a policy that maximizes -expected reward when interacting
with any environment 2M.
1Hutter (2005) calls them chronological conditional semimeasures . This is confusing because contex-
tual semimeasures do notspecify conditional probabilities; the environment is nota joint probability
distribution over actions and percepts.
52 Acting
Problem 4.2 is kept vague on purpose: it does not say how we should balance
between achieving more rewards in some environments while achieving less in others.
In other words, we leave open what an optimalsolution to the general reinforcement
learning problem is. This turns out to be a notoriously diﬃcult question that we discuss
in Chapter 5.
As promised in the title of this thesis, we take the nonparametric approach. For the
rest of this thesis, ﬁx Mto be any countable set of environments. While the true envi-
ronment is unknown, we assume it belongs to the class M(therealizable case ). As long
as the classMis suﬃciently large (such as the class of all computable environments),
this assumption is weak. Some typical choices are discussed in Section 4.1.3.
Our agent-environment setup shown in Figure 4.1 is known as the dualistic model :
the agent is distinct from the environment and inﬂuences it only through its actions. In
turn, the environment inﬂuences the agent only through the percepts. The dualism as-
sumption is accurate for an algorithm that is playing chess, Go, or other (video) games,
which explains why it is ubiquitous in AI research. But often it is not true: real-world
agents are embedded in (and computed by) the environment, and then a physicalistic
model(also called materialistic model ornaturalistic model) is more appropriate. Deci-
sion making in the physicalistic model is still underdeveloped; see Everitt et al. (2015)
and Orseau and Ring (2012a). In this thesis we restrict ourselves to the dualistic model.
4.1.1 Discounting
The goal in reinforcement learning is to maximize rewards. However, the inﬁnite reward
sumP1
t=1rtmay diverge. To get around this technical problem, we let our agent prior-
itize the present over the future. This is done with a discount function that quantiﬁes
how much the agent prefers rewards now over rewards later.
Deﬁnition 4.3 (Discount Function) .Adiscount function is a function
:N!Rwith

t:=
(t)0andP1
t=1
t<1. Thediscount normalization factor ist:=P1
k=t
k.
There is no requirement that t>0. In fact, we use
for both, discounted inﬁnite
horizon ( t>0for allt), and ﬁnite horizon m(m1>0andm= 0) where the agent
does not care what happens after time step m.
Note that the way in which we employ discounting is time consistent : the agent
does not change its mind about how much it values the reward at time step kover
time: reward rkis always discounted with
kregardless of the current time step. For a
discussion of general discounting we refer the reader to Lattimore and Hutter (2014).
Deﬁnition 4.4 (Eﬀective Horizon) .The"-eﬀective horizon Ht(")is a horizon that is
long enough to encompass all but an "of the discount function’s mass:
Ht(") := minfkjt+k=t"g
The eﬀective horizon is bounded iﬀ for all" > 0there is a constant c"such that
Ht(")c"for allt2N.
§4.1The General Reinforcement Learning Problem 53
parameter
t t Ht(")
Finite horizon m2N 1m(t)=m (mt+ 1)=md(mt+ 1)(1")e
Geometric
2(0;1)
t
t=(1
)dlog
"e
Power  >1tt+1=(1)("1=(1)1)t
Subgeometric - ep
t=p
t2ep
tp
tlog"+ (log")2
Table 4.1: Several discount functions and their eﬀective horizons. See also Hutter
(2005, Tab. 5.41) and Lattimore (2013, Tab. 2.1).
Example 4.5 (Geometric Discounting) .The most common discount function is ge-
ometric discounting with
t:=
tfor some constant
2[0;1). We get that t=P1
k=t
k=
t=(1
)and the"-eﬀective horizon is Ht(") =dlog
"e. Hence the eﬀec-
tive horizon is bounded. 3
More examples for discount functions are given in Table 4.1. From now on, we ﬁx
a discount function
.
4.1.2 Implicit Assumptions
Throughout this thesis, we make the following assumptions implicitly.
Assumption 4.6. (a) The discount function
is computable.
(b) Rewards are bounded between 0and1.
(c) The set of actions Aand the set of percepts Eare both ﬁnite.
Let’smotivatetheseassumptionsinturn. Theirpurposeistoensurethatdiscounted
reward sums are ﬁnite and optimal policies exist.
Assumption4.6aisatechnicalassumptionthatensuresthatdiscountedrewardsums
are computable. This is important for Chapter 6 and Chapter 7 where we analyse the
computability of optimal policies. Note that all discount functions given in Table 4.1
are computable.
Assumption 4.6b could be relaxed to require only that rewards are bounded. We
can rescale rewards rt7!crt+dfor anyc2R+andd2Rwithout changing optimal
policies if the environment is a probability measure. (For our computability-related
results in Chapter 6, we must assume that rewards are nonnegative.) In this sense As-
sumption 4.6b is not very restrictive. However, this normalization of rewards into the
[0;1]-interval has the convenient consequence that the normalized discounted reward
sumP1
k=t
krk=kis bounded between 0and1. If rewards are unbounded, then the
discounted reward sum might diverge. Moreover, with unbounded rewards there all
kinds of pathological problems where deﬁning optimal actions is no longer straightfor-
ward; see Arntzenius et al. (2004) for a discussion.
Assumption 4.6c is a technical requirement for the existence of optimal policies since
it implies that there are only ﬁnitely many deterministic policies that diﬀer in the ﬁrst t
54 Acting
time steps. Note that ﬁnite action and percept spaces are very natural since it ensures
that our agent only receives and emits a ﬁnite amount of information in every time
step. This is in line with the problems a strong AI is facing: the agent has to remember
important information and act sequentially.
Assumption 4.6b, Assumption 4.6c, and the fact that the discount function is
summable guarantee that a deterministic optimal policy exists for every environment
according to Lattimore and Hutter (2014, Thm. 10). It would be interesting to relax
theseassumptionswhilepreservingtheexistenceofoptimalpoliciesoratleast "-optimal
policies (e.g. use compact action and percept spaces).
4.1.3 Typical Environment Classes
The simplest reinforcement learning problems are multi-armed bandits.
Deﬁnition 4.7 (Multi-Armed Bandit) .An environment is amulti-armed bandit iﬀ
O=f?gand(etjæ<tat) =(etjat)for all histories æ1:t2(AE ).
In a multi-armed bandit problem there are no observations and the next reward
only depends on the previous action. Intuitively, we are deciding between #Adiﬀerent
slot machines (so-called one-armed bandits), pull the lever and obtain a reward. The
reward is stochastic, but it is drawn from a distribution that is time-invariant and ﬁxed
for each arm.
A multi-armed bandit is also called banditfor short. Although bandits are the sim-
plest reinforcement learning problem, they already exhibit the exploration-exploitation-
tradeoﬀ that makes reinforcement learning diﬃcult: do you pull an arm that has the
best empirical mean or do you pull an arm that has the highest uncertainty? In bandits
it is very easy to come up with policies that perform (close to) optimal asymptoti-
cally (e.g.,"t-greedy with "t= 1=t). But coming up with algorithms that perform well
in practice is diﬃcult, and research focuses on the multiplicative and additive constants
on the asymptotic guarantees. Bandits exist in many ﬂavors; see Bubeck and Bianchi
(2012) for a survey.
Deﬁnition 4.8 (Markov Decision Process) .An environment is aMarkov decision
process(MDP) iﬀ(etjæ<tat) =(etjot1at)for all histories æ1:t2(AE ).
Intuitively, in MDPs, the previous observation ot1provides a suﬃcient statistic for
the history: given ot1and the current action at, the next percept etis independent of
the rest of the history. In other words, everything that the agent needs to know to make
optimal decisions is readily available in the previous percept. This is why observations
are called statesin MDPs. Note that bandits are MDPs with a single state.
Much of today’s literature on reinforcement learning focuses on MDPs (Sutton and
Barto, 1998). They provide a particularly good framework to study reinforcement
learning because they are simple enough to be tractable for today’s algorithms, yet
general enough to encompass many interesting problems. For example, most of the
Atari games (see Figure 1.1 for an overview) are (deterministic) MDPs when combining
§4.1The General Reinforcement Learning Problem 55
the previous four frames into one percept. While they have a huge state space2they
can still be learned using Q-learning with function approximation (Mnih et al., 2015).
The MDP framework is restrictive because it requires the agent to be more powerful
than the environment. Since the agent learns, its actions are not independent of the
rest of the history given the last action and percept. In other words, learning agents are
not Markov. The following deﬁnition lifts this restriction and allows the environment
to bepartially observable .
Deﬁnition 4.9 (Partially Observable Markov Decision Process) .An environment is
apartially observable Markov decision process (POMDP) iﬀ there is a set of statesS,
aninitial state s02S, astate transition function 0:SA! S, and apercept
distribution 00:S! Esuch that
(e1:tka1:t) =tY
k=100(ekjsk)0(skjsk1;ak):
Usually the setSis assumed to be ﬁnite; with inﬁnite-state POMDPs we can model
any environment by setting the set of states to be the set of histories, S:= (AE ).
A common assumption for MDPs and POMDPs is that they do not contain traps.
Formally, a (PO)MDP is ergodiciﬀ for any policy and any two states s1;s22S,
the expected number of time steps to reach s2froms1is-almost surely ﬁnite. A
(PO)MDP is weakly communicating iﬀ for any two states s1;s22Sthere is a policy
such that the expected number of time steps to reach s2froms1is-almost surely
ﬁnite. Note that any ergodic (PO)MDP is also weakly communicating, but not vice
versa.
Ingeneral, ourenvironmentsarestochastic. Stochasticitycanoriginatefromnoisein
the environment, noise in the sensors, or modeling errors. Sometimes we also consider
classes of deterministic environments. These are usually easier to deal with because
they do not require as much mathematical machinery. For example, in a deterministic
environment the next percept is certain; if a diﬀerent percept is received this environ-
ment is immediately falsiﬁed and can be discarded. In a stochastic environment, an
unlikely percept reduces our posterior belief in this environment but does not rule it
out completely.
In Chapter 6 and Chapter 7 we make the assumption that the environment is com-
putable. This encompasses all ﬁnite-state POMDPs and most if not all AI problems
can be formulated in this setting. Moreover, the current theories of quantum mechanics
and general relativity are computable and there is no evidence that suggests that our
physical universe is incomputable. For any physical system of ﬁnite volume and ﬁnite
(average) energy, the amount of information it can contain is ﬁnite (Bekenstein, 1981),
and so is the number of state transitions per unit of time (Margolus and Levitin, 1998).
This gives us reason to believe that even the environment that we humans currently
face (and will ever face) falls under these assumptions.
2The size of the state space is at most 256128since the Atari 2600 has only 128 bytes of memory.
However, the vast majority of these states are not reachable.
56 Acting
Formally we deﬁne the set MCCS
LSCas the set of environments that are lower semicom-
putable chronological contextual semimeasures and MCCM
compas the set of environments
that are computable chronological contextual measures. Note that for chronological
contextual semimeasures it makes a diﬀerence whether (ka1:1)is lower semicom-
putable or the conditionals (jæ<tat)are. The latter implies the former, but not vice
versa.
4.2 The Value Function
Thevalueof a policy in an environment is the future expected discounted reward when
following a given policy in a given environment conditional on the past. Since this
quantity captures exactly what our agent aims to maximize, we prefer policies whose
value is high.
Deﬁnition 4.10 (Value Function) .Thevalueof a policyin an environment given
historyæ<tandhorizonmwithtm1is deﬁned as
V;m
(æ<t) :=1
tE
"m1X
k=t
krkæ<t#
ift>0andV;m
(æ<t) := 0ift= 0. Theoptimal value is deﬁned as V;m
(æ<t) :=
supV;m
(æ<t).
Sometimes we omit the history argument æ<tfor notational convenience if it is clear
from context. Moreover, when we omit m, we implicitly use an inﬁnite horizon m=1,
i.e.,V
:=V;1
andV
:=V;1
. The value of a policy in an environment after
the empty history, V
()is also called the t0-value.
Remark 4.11 (Values are Bounded Between 0 and 1) .From Assumption 4.6b we
get that for all histories æ<tall policies and all environments , the value function
V
(æ<t)2[0;1]. 3
Since environment and policy are stochastic, the history æ<tis random. With abuse
of notation we treat æ<tsometimes as a concrete outcome and sometimes as a random
variable. We also view the value of a policy in an environment as a sequence
of random variables (Xt)t2NwithXt:=V
(æ1:t)where the history æ1:tis generated
stochastically by the agent’s actual policy interacting with the true environment .
This view is helpful for some of the convergence results (e.g., Theorem 4.19 and Deﬁ-
nition 5.18) in which we talk about the type of convergence of this sequence of random
variables.
The value function deﬁned in Deﬁnition 4.10 is also called the recursive value func-
tion, in contrast to iterative value function that we discuss in Section 6.4. The name of
the recursive value function originates from the following recursive identity (analogously
§4.2The Value Function 57
to Hutter, 2005, Eq. 4.12), also called the Bellman equation :
V
(æ<t) =X
at2A(atjæ<t)V
(æ<tat)
V
(æ<tat) =1
tX
et2E(etjæ<tat)

trt+ t+1V
(æ1:t)
An explicit expression for the optimal value in environment is
V;m
(æ<t) =1
tmaxX
æt:m1m1X
k=t
krkkY
i=t(eijæ<iai); (4.2)
wherePmaxdenotes the max-sum-operator:
maxX
æt:m1:= max
at2AX
et2E:::max
am12AX
em12E
For an explicit expression of V;1
(æ<t)we can simply take the limit m!1.
4.2.1 Optimal Policies
An optimal policy is a policy that achieves the highest value:
Deﬁnition4.12 (OptimalPolicy; Hutter,2005, Def.5.19&5.30) .Apolicyisoptimal
in environment (-optimal) iﬀforallhistories attainstheoptimalvalue: V
(æ<t) =
V
(æ<t)for allæ<t2(AE ). The action atis anoptimal action iﬀ
(atjæ<t) = 1
for some-optimal policy
.
Following the tradition of Hutter (2005), AINUdenotes a-optimal policy for the
environment 2MCCS
LSCandAIMUdenotes an -optimal policy for the environment
2MCCM
compthat is a measure (as opposed to a semimeasure).
By deﬁnition of the optimal policy and the optimal value function, we have the
following identity for all histories æ<t:
V
(æ<t) =V
(æ<t) (4.3)
There can be more than one optimal policy; generally the choice of
from Deﬁni-
tion 4.12 is not unique. More speciﬁcally, for a -optimal policy we have

(atjæ<t)>0 =)at2arg max
a2AV
(æ<ta): (4.4)
If there are multiple actions ;2 Athat attain the optimal value, V
(æ<) =
V
(æ<t), then there is an argmax tie . Which action we settle on in case of a tie (how
we break the tie) is irrelevant and can be arbitrary. Since we allow stochastic policies,
we can also randomize between and.
The following deﬁnition allows policies to be slightly suboptimal.
58 Acting
Deﬁnition 4.13 ("-Optimal Policy) .A policyis"-optimal in environment iﬀ
V
(æ<t)V
(æ<t)<"for all histories æ<t2(AE ).
A policythat achieves optimal t0-value,V
() =V
(), takes-optimal actions
on any history reachable by in. However, this is not true for "-optimal policies: a
policy that is "-optimal at t= 0is not necessarily "-optimal in later time steps.
4.2.2 Properties of the Value Function
The following two lemmas are stated by Hutter (2005, Thm. 31) without proof and for
the iterative value function.
Lemma4.14 (Linearityof V
in).If=z11+z22for some real numbers z1;z20,
then for all policies and all histories æ<t
V;m
(æ<t) =z11(e<tka<t)
(e<tka<t)V;m
1(æ<t) +z22(e<tka<t)
(e<tka<t)V;m
2(æ<t):
Proof.Since=z1
1+z2
2, we have for the conditional measure
(Ajæ<t) =(A\æ<t)
(æ<t)=z1
1(A\æ<t) +z2
2(A\æ<t)
(æ<t)
=z1
1(æ<t)
(æ<t)
1(Ajæ<t) +z2
2(æ<t)
(æ<t)
2(Ajæ<t):
The claim now follows from the linearity of expectation in the probability measure.
Lemma 4.15 (Convexity of V
in).If=z11+z22for some real numbers z1;z2
0, then for all histories æ<t
V
(æ<t)z11(e<tka<t)
(e<tka<t)V;m
1(æ<t) +z22(e<tka<t)
(e<tka<t)V;m
2(æ<t):
Proof.Let
be an optimal policy for environment . From Lemma 4.14 we get
V
(æ<t) =V
(æ<t) =z11(e<tka<t)
(e<tka<t)V
1(æ<t) +z22(e<tka<t)
(e<tka<t)V
2(æ<t)
z11(e<tka<t)
(e<tka<t)V
1(æ<t) +z22(e<tka<t)
(e<tka<t)V
2(æ<t):
The following lemma bounds the error when truncating the value function. This
implies that planning for an "-eﬀective horizon ( m=t+Ht(")), we get all but an "of
the value:jV
(æ<t)V;m
(æ<t)j<".
Lemma 4.16 (Truncated Values) .For every environment , every policy , and every
historyæ<tV;m
(æ<t)V
(æ<t)m
t:
§4.2The Value Function 59
Proof.
V
(æ<t) =1
tE
"1X
k=t
krkæ<t#
=V;m
(æ<t) +1
tE
"1X
k=m
krkæ<t#
The result now follows from Assumption 4.6b and
0E
"1X
k=m
krkæ<t#
m:
This lemma bounds the (truncated) value function by the total variation distance.
Lemma 4.17 (Bounds on Value Diﬀerence) .For any policies 1,2, any environments
1and2, and any horizon tm1
jV1;m
1(æ<t)V2;m
2(æ<t)j Dm1(1
1;2
2jæ<t)
Proof.According to Deﬁnition 4.10, the value function is the expectation of the ran-
dom variablePm1
k=t
krk=tthat is bounded between 0and1. Therefore we can use
Lemma 2.12 with P:=1
1(jæ<t)andR:=2
2(jæ<t)on the space (AE )m1to
conclude thatjV1;m
1(æ<t)V2;m
2(æ<t)jis bounded by Dm1(1
1;2
2jæ<t).
Lemma 4.18 (Discounted Values; Lattimore, 2013, Lem. 2.5) .Letæ<tbe some history
and let1and2be two policies that coincide from time step tto time step m:1(aj
æ1:k) =2(ajæ1:k)for alla2A, all histories æ<tæt:kconsistent with 1, andtk
m. Then for all environments
V1(æ<t)V2(æ<t)m
t:
Proof.Since1and2coincidefortimesteps tthroughm1,Dm1(1;2jæ<t) = 0
for all environments . Thus the result follows from Lemma 4.16 and Lemma 4.17:
V1(æ<t)V2(æ<t)V1;m
(æ<t)V2;m
(æ<t)+m
t
Dm1(1;2jæ<t) +m
t
=m
t
4.2.3 On-Policy Value Convergence
This section states some general results on learning the value function. On-policy value
convergence refers to the fact that if we use a learning distribution to learn to environ-
ment, andmerges with in the sense discussed Section 3.4, then V
converges
toV
, i.e., using we learn to estimate values correctly.
A weaker variant of the following theorem was proved by Hutter (2005, Thm. 5.36).
It states convergence in mean (not almost surely), and only for the Bayesian mixture.
60 Acting
Theorem 4.19 (On-Policy Value Convergence) .Letbe any environment and be
any policy.
(a) Ifmerges strongly with , then
V
(æ<t)V
(æ<t)!0ast!1-almost surely.
(b) If the eﬀective horizon is bounded and merges weakly with , then
V
(æ<t)V
(æ<t)!0ast!1-almost surely.
(c) If the eﬀective horizon is bounded and merges almost weakly with , then
1
ttX
k=1
V
(æ<k)V
(æ<k)
!0ast!1in-almost surely.
Proof.(a) Apply Lemma 4.17 with m:=1.
(b) Let">0and letc"be a bound on suptHt("). From Lemma 4.16
jV
(æ<t)V
(æ<t)jjV;t+Ht(")
 (æ<t)V;t+Ht(")
 (æ<t)j+ 2t+Ht(")
t
<Dt+Ht1(")(;jæ<t) + 2"
Dt+c"(;jæ<t) + 2"
accordingtoDeﬁnition4.4andLemma4.17. Since mergesweaklywith ,weget
that-almost surely there is a time step t02Nsuch thatDt+c"(;jæ<t)<"
for alltt0. HencejV
Q(æ<t)V
(æ<t)j<3"for alltt0.
(c) Analogously to the proof of (b).
It is important to observe that on-policy convergence does not imply that the agent
converges to the optimal policy. V
converges to V
, butV
need not be close to V
.
Indeed, there might be another policy ~that has a higher value than in the true
environment (V~
>V
). If the agent thinks ~has lower value ( V~
<V
) it might
not follow ~and hence not learn that the actual value of ~is much higher. In other
words, on-policy convergence implies that the agent learns the value of its own actions,
but not the value of counterfactual actions that it does not take.
Theorem 4.19 now enables us to tie in the results of Chapter 3. This yields a surge
of corollaries, but ﬁrst we need to make the learning distributions contextual on the
actions.
Letw2Mbe a positive prior over the environment class M. We deﬁne the
corresponding Bayesian mixture analogously to Example 3.4:
(e<tka<t) :=X
2Mw()(e<tka<t) (4.5)
§4.2The Value Function 61
Note that the Bayesian mixture depends on the prior w. For the rest of this thesis,
this dependence will not be made explicit.
From Lemma 4.14 and (3.3) we immediately get the following identity:
V
(æ<t) =X
2Mw(jæ<t)V
(æ<t) (4.6)
Similarly, we get from Lemma 4.15
V
(æ<t)X
2Mw(jæ<t)V
(æ<t): (4.7)
Corollary 4.20 (On-Policy Value Convergence for Bayes) .For any environment 2
Mand any policy ,
V
(æ<t)V
(æ<t)!0ast!1-almost surely.
Proof.Since2M, we have dominance w()withw()>0and by Propo-
sition 3.16a absolute continuity . From Theorem 3.25 we get that merges
strongly with . Therefore we can apply Theorem 4.19a.
Analogously, we deﬁne MDLæ<t:= arg min2Mflog(e<tka<t) +K()g.
Corollary4.21 (On-PolicyValueConvergenceforMDL) .For any environment 2M
and any policy ,
V
MDLæ<t(æ<t)V
(æ<t)!0ast!1-almost surely.
Proof.By Theorem 3.28 MDLmerges strongly with for each2M, therefore we
can apply Theorem 4.19a.
By providing the action sequence contextually on a separate input tape, we can
deﬁneKm(e<tka<t) := minfjpjje<tvU(p;a<t)ganalogously to (2.1).
Corollary4.22 (On-PolicyValueConvergenceforUniversalCompression) .Let(e<tk
a<t) := 2Km(e<tka<t). Then for any environment 2MCCM
compand any policy ,
V
(æ<t)V
(æ<t)!0ast!1-almost surely.
Proof.Sincedominates every 2 MCCM
comp(Section 3.6.3) we can apply Proposi-
tion 3.16a, Theorem 3.25, and Theorem 4.19a as in the proof of Corollary 4.20.
Similarly to Kmthere is a speed prior for environments (Filan, 2015, Ch. 6):
SKt(e<tka<t) :=X
p:e<tvU(p;a<t)2jpj
t(U;p;a<t;e<t)
wheret(U;p;a<t;e<t)denotes the number of time steps U(p;a<t)takes to produce e<t.
62 Acting
Corollary 4.23 (On-Policy Value Convergence for the Speed Prior) .If the eﬀective
horizon is bounded, then for any environment 2MCCM
compestimable in polynomial time
and any policy ,
1
ttX
k=1
V
SKt(æ<k)V
(æ<k)
!0ast!1-almost surely.
Proof.By Corollary 3.61 the speed prior SKtmerges almost weakly with every measure
estimable in polynomial time. Therefore we can apply Theorem 4.19c.
4.3 The Agents
If we knew the true environment , we would choose the -optimal policy, the policy
that maximizes -expected discounted rewards. But generally we do not know the
true environment, and the challenging part of reinforcement learning is to learn the
environment while trying to collect rewards.
In this section we introduce a number of agents that attempt to solve the general
reinforcement learning problem (Problem 4.2). These agents are discussed throughout
the rest of this thesis.
4.3.1 Bayes
ABayes optimal policy with respect to the prior wis the policy
whereis the
Bayesian mixture deﬁned in Section 4.2.3. There can be one or more Bayes optial
policies. From Corollary 4.20 we get on-policy value convergence for the Bayes optimal
policy.
After history æ<t, the Bayes policy
maximizes expected discounted rewards in
the posterior mixture:
(je<tka1:1) =X
2Mw(jæ<t)(je<tka1:1)
wherew(jæ<t)are the posterior weights (3.3). Maximizing expected rewards accord-
ing to the posterior is the same as maximizing expected rewards according to the prior
conditional on the history: if (æ<t) =
(æ<t), thenV
(æ<t) =V
(æ<t). Actually
visiting the history æ<tdoes not change what
planned to do before it visited æ<t.
Note that this relies on the fact that the way we use discounting is time consistent (Lat-
timore and Hutter, 2014, Def. 12).
When using the prior w()/2K()(Example 3.5) over the class MCCS
LSC, the Bayes
optimal policy is also known as AIXI, introduced and analyzed by Hutter (2000, 2001a,
2002a, 2003, 2005, 2007a, 2012b) in his work on universal artiﬁcial intelligence . In this
case, the Bayesian mixture (4.5) can be deﬁned equivalently according to (Wood et al.,
2011)
(e<tka<t) :=X
p:e<tvU(p;a<t)2jpj: (4.8)
§4.3The Agents 63
Generally there is more than one -optimal policy and Solomonoﬀ’s prior depends on
the choice of the (reference) universal Turing machine, so this deﬁnition is not unique.
Moreover, not every universal Turing machine is a good choice for AIXI, see Section 5.2
for a few bad choices. The following lemma will be used later.
Lemma 4.24 (Mixing Mixtures) .Letq;q02Qsuch thatq >0,q00, andq+q0
1. Letwbe any lower semicomputable positive prior, let be the Bayesian mixture
corresponding to w, and let2MCCS
LSC. Then0:=q+q02MCCS
LSCis a Bayesian
mixture.
Proof.0is given by the positive prior w0withw0:=qw+q01.
Bayesian approaches have a long tradition in reinforcement learning, although they
are often prohibitively expensive to compute. For multi-armed bandits, Gittins (1979)
achieved a breakthrough with an index strategy that enables the computation of the
optimal policy by computing one quantity for each arm independently of the rest. This
strategy even achieves the optimal asymptotic regret bounds (Lattimore, 2016). Larger
classes have also been attempted: using Monte-Carlo tree search, Veness et al. (2011)
approximate the Bayes optimal policy in the class of all context trees. Doshi-Velez
(2012) uses Bayesian techniques to learn inﬁnite-state POMDPs. See Vlassis et al.
(2012) for a survey on Bayesian techniques in RL.
In the rest of this thesis, the Bayes optimal policy is often treated as an optimal ex-
ploitation strategy. This is not true: Bayes does explore (when it is Bayes optimal to do
so). It just does not explore general environment classes completely (see Section 5.4.1).
4.3.2 Knowledge-Seeking Agents
In this section we discuss two variants of knowledge-seeking agents: entropy-seeking
agents introduced by Orseau (2011, 2014a) and information-seeking agents introduced
by Orseau et al. (2013). The entropy-seeking agent maximizes the Shannon entropy
gain, while the information-seeking agent maximizes the expected information gain.
These quantities are expressed in diﬀerent value functions. In places where confusion
can arise, we call the value function V
from Deﬁnition 4.10 the reward-seeking value
function.
In this section we use a ﬁnite horizon m<1(possibly dependent on time step t):
the knowledge-seeking agent maximizes entropy/information received up to time step
m. We assume implicitly that m(as a function of t) is computable. Moreover, in this
section we assume that the Bayesian mixture is a measure rather than a semimeasure;
Example 4.27 discusses this assumption.
Deﬁnition4.25 (Entropy-SeekingValueFunction; Orseau,2014a,Sec.6) .Theentropy-
seeking value of a policy given history æ<tis
V;m
Ent(æ<t) :=E
[log2(e1:mje<tka1:m)jæ<t]:
64 Acting
The entropy-seeking value is the Bayes-expectation of log. Orseau (2011, 2014a)
also considers a related value function based on the -expectation of that we do not
discuss here.
Deﬁnition 4.26 (Information-Seeking Value Function; Orseau et al., 2013, Def. 1) .
Theinformation-seeking value of a policy given history æ<tis
V;m
IG(æ<t) :=X
2Mw(jæ<t)KLm(;jæ<t):
Analogously to before we deﬁne V
Ent:= supV
EntandV
IG:= supV
IG. An optimal
entropy-seeking policy is deﬁned as
Ent:2arg maxV
Entand an optimal information-
seeking policy is deﬁned as
IG:2arg maxV
IG. Since we use a ﬁnite horizon ( m<1),
these optimal policies exist.
Theinformation gain is deﬁned as the diﬀerence in entropy between the prior and
the posterior:
IGt:m(æ1:m) := Ent(w(jæ<t))Ent(w(jæ1:m))
We get the following identity (Lattimore, 2013, Eq. 3.5).
E
[IGt:m(æ1:m)jæ<t] =V;m
IG(æ<t)
For inﬁnite horizons ( m=1), the values functions from Deﬁnition 4.25 and Def-
inition 4.26 may not converge. To ensure convergence, we can either use discounting,
or in case of VIGa prior with ﬁnite entropy (Lattimore, 2013, Thm. 3.4). Moreover,
note that while VIGandVEntare expectations with respect to the measure , there is
no bound on the one-step change in value V;m
IG(æ<t)V;m
IG(æ1:t), which can also be
negative. For the reward-seeking value function V;m
, the one-step change in value is
bounded between 0and1by Remark 4.11.
For classes of deterministic environments Deﬁnition 4.25 and Deﬁnition 4.26 coin-
cide. In stochastic environments the entropy-seeking agent does not work well because
it gets distracted by noise in the environment rather than trying to distinguish envi-
ronments (Orseau et al., 2013, Sec. 5). Moreover, the entropy-seeking agent may fail to
seek knowledge in deterministic semimeasures as the following example demonstrates.
Example 4.27 (Unnormalized Entropy-Seeking) .If the Bayesian mixture is a semi-
measure instead of a measure (such as the Solomonoﬀ prior from Example 3.5), then
the entropy-seeking agent does not explore correctly. Fix A:=f;g,E:=f0;1g, and
m=t(we only care about the entropy of the next percept). We illustrate the problem
on a simple class of environments f1;2g:
1 =0=0:1 =0=0:5 2 =1=0:1 =0=0:5
where transitions are labeled with action/percept/probability. Both 1and2return
a percept deterministically or nothing at all (the environment ends). Only action
§4.3The Agents 65
distinguishes between the environments. With the prior w(1) :=w(2) := 1=2, we get
a mixturefor the entropy-seeking value function V
Ent. ThenV
Ent()0:432<0:5 =
V
Ent(), hence action is preferred over by the entropy-seeking agent. But taking
actionyields percept 0(if any), hence nothing is learned about the environment. 3
On-policy value convergence (Theorem 4.19) ensures that asymptotically, the agent
learns the value of its own policy. Knowledge-seeking agents do even better: they
don’t have to balance between exploration and exploitation, so they can focus solely
on exploration. As a result, they learn oﬀ-policy, i.e., the value of counterfactual
actions (Orseau et al., 2013, Thm. 7).
4.3.3 BayesExp
Lattimore (2013, Thm. 5.6) deﬁnes BayesExp combining AIXI with the information-
seeking agent. BayesExp alternates between phases of exploration and phases of ex-
ploitation: Let "tbe a monotone decreasing sequence of positive reals such that "t!0
ast!1. If the optimal information-seeking value V
IGis larger than "t, then BayesExp
starts an exploration phase, otherwise it starts an exploitation phase. During an explo-
ration phase, BayesExp follows an optimal information-seeking policy for an "t-eﬀective
horizon. During an exploitation phase, BayesExp follows an -optimal reward-seeking
policy for one step (see Algorithm 1).
Algorithm 1 BayesExp policy BE(Lattimore, 2013, Alg. 2).
1:whiletruedo
2:ifV;t+Ht("t)
IG(æ<t)>"tthen
3: follow
IGforHt("t)steps
4:else
5: follow
for1step
4.3.4 Thompson Sampling
Thompson sampling , also known as posterior sampling orthe Bayesian control rule , was
originally proposed by Thompson (1933) as a bandit algorithm. It is easy to implement
and often achieves quite good results (Chapelle and Li, 2011). In multi-armed bandits
it attains optimal regret (Agrawal and Goyal, 2011; Kaufmann et al., 2012). Thompson
sampling has also been discussed for MDPs (Strens, 2000; Dearden et al., 1998) and
Bayesian and frequentist regret bounds have been established (Osband et al., 2013;
Gopalan and Mannor, 2015).
For general RL Thompson sampling was ﬁrst suggested by Ortega and Braun (2010)
with resampling at every time step. Strens (2000) proposes following the optimal policy
for one episode or “related to the number of state transitions the agent is likely to need
to plan ahead”. We follow Strens’ suggestion and resample at the eﬀective horizon.
Let"tbe a monotone decreasing sequence of positive reals such that "t!0as
t!1. Our variant of Thomson sampling is given in Algorithm 2. It samples an
66 Acting
environment fromtheposterior,followsthe -optimalpolicyforan "t-eﬀectivehorizon,
and then repeats.
Algorithm 2 Thompson sampling policy T.
1:whiletruedo
2:samplew(jæ<t)
3:follow
forHt("t)steps
Note thatTis a stochastic policy since we occasionally sample from a distribution.
We assume that this sampling is independent of everything else.
Chapter 5
Optimality
Machines will never be intelligent. — Shane Legg
Problem 4.2 deﬁnes the general reinforcement learning problem. But our deﬁnition of
this problem did not specify what a solution would be. This chapter is dedicated to
this question:
What is an optimal solution to the general reinforcement learning problem?
How can we say that one policy is betterthan another? What is the bestpolicy? Are
the policies from Section 4.3 optimal? Several notions of optimality for a policy in
an environment class Mare conceivable:
O1.Maximal reward . The policy receives a reward of 1in every time step (which is
maximal according to Assumption 4.6b):
8t2N:rt= 1
O2.Optimal policy . The policy achieves the highest possible value in the true envi-
ronment:
8æ<t2(AE ):V
(æ<t) =V
(æ<t)
O3.Pareto optimality (Hutter, 2002a, Thm. 2). There is no other policy that performs
at least as good in all environments and strictly better in at least one:
@~:
82M:V~
()V
()and92M:V~
()>V
()
O4.Balanced Pareto optimality (Hutter, 2002a, Thm. 3). The policy achieves a
better value across Mweighted by w2Mthan any other policy:
8~:X
2Mw()
V
()V~
()
0
O5.Bayes optimality . The policy is-optimal for some Bayes mixture :
8æ<t2(AE ):V
(æ<t) =V
(æ<t)
67
68 Optimality
O6.Probably approximately correct . For a given "; > 0the value of the policy is
"-close to the optimal value with probability at least after time step t0(";):
h
8tt0(";):V
(æ<t)V
(æ<t)<"i
>1
O7.Asymptotic optimality (Hutter, 2005, Sec. 5.3.4). The value of the policy con-
verges to the optimal value:
V
(æ<t)V
(æ<t)!0ast!1
O8.Sublinear regret . The diﬀerence between the reward sum of the policy and the
best policy in hindsight grows sublinearly:
sup
0E0
"mX
t=1rt#
E
"mX
t=1rt#
2o(m)
We discuss these notions of optimality in turn. Achieving the maximal reward
at every time step is impossible if there is no action that makes the environment
respond with the maximal reward; generally there is no policy that achieves maximal
rewards at every time step. In order to follow the optimal policy, we need to know
the true environment. In our setting, the true environment is unknown and has to be
learned. During the learning process the agent cannot also act optimally because it
needs to explore. In particular, the policy cannot be optimal simultaneously in all
environments from M. This rules out O1 and O2 as a notion of optimality.
In Section 5.1 we show that all policies are Pareto optimal. This disqualiﬁes O3 as
a useful notion of optimality in general reinforcement learning.
Balanced Pareto optimality (O4), Bayes optimality (O5), and maximal Legg-Hutter
intelligence (Legg and Hutter, 2007b) turn out to coincide. In Section 5.3 we show that
Legg-Hutter intelligence is highly subjective, because it depends on the choice of the
prior. By changing the prior of a Bayesian agent, we can make the agent’s intelligence
arbitrarily low. In Section 5.2 we present a choice of particularly bad priors. This rules
out O4 and O5 because they are prior-dependent and not objective.
O6 is a stronger version of asymptotic optimality that provides a rate of conver-
gence (it implies O7). Since our environment class can be very large and non-compact,
concrete PAC results are likely impossible. Orseau (2010, 2013) shows that the Bayes
optimal agent does not achieve asymptotic optimality in all computable environments.
The underlying problem is that in the beginning the agent does not know enough about
its environment and therefore relies heavily on its prior. Lack of exploration then re-
tains the prior’s bias. This problem can be alleviated by adding extra exploration to the
Bayesian agent. In Section 5.4 we discuss two agents that achieve asymptotic optimal-
ity: BayesExp (Section 4.3.3) and Thompson sampling (Section 4.3.4). This establishes
that O7 is possible.
In general environments sublinear regret is impossible because the agent can get
stuck in traps from which it is unable to recover. This rules out O8. However, in Sec-
§5.1Pareto Optimality 69
tion5.5weshowthatifweassumethattheenvironmentallowsrecoveringfrommistakes
(and some minor conditions on the discount function are fulﬁlled), then asymptotic op-
timality implies sublinear regret. This means that Thompson sampling has sublinear
regret in these recoverable environments.
Notably, only asymptotic optimality (O7) holds up to be a nontrivial and objective
criterionofoptimalitythatappliestothegeneralreinforcementlearningproblem. While
there are several agents that are known to be asymptotically optimal, some undesirable
properties remain. Section 5.6 discusses this further. See also Mahadevan (1996) for a
discussion of notions of optimality in MDPs.
5.1 Pareto Optimality
In this section we show that Pareto optimality is not a useful criterion for optimality
since for any environment class containing MCCM
comp, all policies are Pareto optimal.
Deﬁnition 5.1 (Pareto Optimality; Hutter, 2005, Def. 5.22) .A policyisPareto
optimal in the set of environments Miﬀ there is no policy ~such thatV~
()V
()
for all2MandV~
()>V
()for at least one 2M.
The literature provides the following result.
Theorem5.2 (AIXIisParetoOptimal; Hutter,2002a, Thm.2) .Every-optimal policy
is Pareto optimal in MCCS
LSC.
The following theorem was proved for deterministic policies in Leike and Hutter
(2015c). Here we extend it to stochastic policies.
Theorem 5.3 (Pareto Optimality is Trivial) .Every policy is Pareto optimal in any
classMMCCM
comp.
The proof proceeds as follows: for a given policy , we construct a set of ‘buddy
environments’ that reward and punish other policies. Together they can defend
against any policy ~that tries to take the crown of Pareto optimality from .
Proof.We assume (0;0)and(0;1)2E. Moreover, assume there is a policy that is not
Pareto optimal. Then there is a policy ~thatPareto dominates , i.e.,V~
()>V
()
for some2 M, andV~
()V
()for all2 M. FromV~
()> V
()and
Lemma 4.18 we get that there is a shortest and lexicographically ﬁrst history æ0
<k
consistent with and~such that(jæ0
<k)>~(jæ0
<k)for some action 2A
andV~
(æ0
<k)>V
(æ0
<k). Consequently there is an iksuch that
i>0, and hence
k>0. We deﬁne the environment that ﬁrst reproduces the separating history æ0
<k
and then, if is the next action, returns reward 1forever, and otherwise returns reward
0forever. Formally, is deﬁned by
(e1:tje<tka1:t) :=8
>>>><
>>>>:1;ift<kandet=e0
t;
1;iftkandak=andrt= 1andot= 0;
1;iftkandak6=andrt= 0 =ot;and
0;otherwise:
70 Optimality
The environment is computable, even if the policy is not: for a ﬁxed history æ0
<t
and action , there exists a program computing ; therefore2MCCM
comp. We get the
following value diﬀerence for the policies and~:
V
()V~
() =E
"k1X
t=1
trt+1X
t=k
trt#
E~
"k1X
t=1
trt1X
t=k
trt#
=
(jæ0
<k)1X
t=k
t~(jæ0
<k)1X
t=k
t!
(æ0
<k)
=
(jæ0
<k)~(jæ0
<k)
(æ0
<k)k>0
HenceV~
()< V
(), which contradicts the fact that ~Pareto dominates since
MMCCM
comp3.
Note that the environment we deﬁned in the proof of Theorem 5.3 is actually
just a ﬁnite-state POMDP, so Pareto optimality is also trivial for smaller environment
classes.
5.2 Bad Priors
In this section we give three examples of universal priors that cause a AIXI to mis-
behave drastically. In case of a ﬁnite horizon, the indiﬀerence prior makes all actions
equally preferable to AIXI (Section 5.2.1). The dogmatic prior makes AIXI stick to
any given computable policy as long as expected future rewards do not fall too close
to zero (Section 5.2.2). The Gödel prior prevents AIXI tlfrom taking any actions (Sec-
tion 5.2.3).
5.2.1 The Indiﬀerence Prior
The following theorem constructs the indiﬀerence prior that yields a Bayesian mixture
0that causes argmax ties for the ﬁrst msteps. If we use a discount function that only
cares about the ﬁrst msteps, m= 0, then all policies are 0-optimal policies. In this
case AIXI’s behavior only depends on how we break argmax ties.
Theorem 5.4 (Indiﬀerence Prior) .If there is an msuch that m= 0, then there is a
Bayesian mixture 0such that all policies are 0-optimal.
Proof.First, we assume that the action space is binary, A=f0;1g. LetUbe the
reference UTM and deﬁne the UTM U0by
U0(s<mp;a1:t) :=U(p;a1:txors1:t);
wheres<mis a binary string of length m1andsk:= 0forkm. (U0has no programs
of length less than m1.) Let0be the Bayesian mixture given by U0according to
§5.2Bad Priors 71
(4.8). Then
0(e<mka<m) =X
p:e<mvU0(p;a<m)2jpj
=X
s<mp0:e<mvU0(s<mp0;a<m)2m1jp0j
=X
s<mX
p0:e<mvU(p0;a<mxors<m)2m1jp0j
=X
s<mX
p0:e<mvU(p0;s<m)2m1jp0j;
which is independent of a<m. Hence the ﬁrst m1percepts are independent of the
ﬁrstm1actions. But the percepts’ rewards from time step mon do not matter since
m= 0(Lemma 4.16). Because the environment is chronological, the value function
must be independent of all actions. Thus every policy is 0-optimal.
For ﬁnite action spaces Awith more than 2elements, the proof works analogously
by makingAa cyclic group and using the group operation instead of xor.
The choice of U0in the proof of Theorem 5.4 depends on m. If we increase AIXI’s
horizon while ﬁxing the UTM U0, Theorem 5.4 no longer holds. For Solomonoﬀ in-
duction, there is an analogous problem: when using Solomonoﬀ’s prior Mto predict a
deterministic binary sequence x, we make at most K(x)errors (Corollary 3.56). In case
the shortest program has length >m, there is no guarantee that we make less than m
errors (see Section 5.6.2).
5.2.2 The Dogmatic Prior
In this section we deﬁne a universal prior that assigns very high probability of going to
hell (reward 0forever) if we deviate from a given computable policy . For a Bayesian
agent like AIXI, it is thus only worth deviating from the policy if the agent thinks
that the prospects of following are very poor already. We call this prior the dogmatic
prior, because the fear of going to hell makes AIXI conform to any arbitrary ‘dogmatic
ideology’. AIXI will only break out if it expects to give very low future payoﬀ; in
that case the agent does not have much to lose.
Theorem 5.5 (Dogmatic Prior) .Letbe any computable deterministic policy, let
be any Bayesian mixture over MCCS
LSC, and let" >0. There is a Bayesian mixture 0
such that for any history æ<tconsistent with and for which V
(æ<t)>", the action
(æ<t)is the unique 0-optimal action.
The following proof was adapted from Leike and Hutter (2015c) to work for en-
vironment classes that do not contain the Bayesian mixture. Essentially, for every
environment 2MCCS
LSCthe dogmatic prior puts much higher weight on an environ-
mentthat behaves just like on the policy , but sends any policy deviating from
to hell. Importantly, while following the policy the environments andare
indistinguishable, so the posterior belief in is equal to the posterior belief in .
72 Optimality
Proof of Theorem 5.5. We assume (o;0)2Efor someo2O. For every environment
2MCCS
LSCdeﬁne the environment
(e1:tka1:t) :=8
>>>><
>>>>:(e1:tka1:t);ifak=(æ<k)8kt;
(e<kka<k);ifk:= minfijai6=(æ<i)gexists
andei= (o;0)8i2fk;:::;tg;and
0; otherwise:
The environment mimics environment until it receives an action that the policy
would not take. From then on, it provides rewards 0. Sinceis a computable policy,
we have that 2MCCS
LSCfor every2MCCS
LSC.
Now we need to reweigh the prior wso that it assigns a much higher prior weight
tothan to. Without loss of generality we assume that "is computable, otherwise
we make it slightly smaller. We deﬁne w0() :="w()if6=~for all ~2MCCS
LSCand
w0() := (1")w() +"w(). Then
X
2MCCS
LSCw0() =X
=~w0() +X
6=~w0()
=X
=~
(1")w(~) +"w()
+X
6=~"w()
=X
2MCCS
LSC"w() +X
~2MCCS
LSC(1")w(~)
="+ (1") = 1;
and withw0"w, we get that w0is a positive prior over MCCS
LSC. We deﬁne 0as the
corresponding Bayesian mixture analogous to (4.5).
With:=P
2MCCS
LSCw()we get0="+ (1"). The mixtures and
coincide on the policy since every coincides with on the policy :
(æ<t) =X
2MCCS
LSCw()(æ<t) =X
2MCCS
LSCw()
(æ<t) =(æ<t)
Moreover,V
(æ<t) = 0and thusV
(æ<t) = 0for any history inconsistent with by
construction of .
Letæ<t2(AE )be any history consistent with such thatV
(æ<t)>". Then
=implies
(e<tka<t)
0(e<tka<t)=(e<tka<t)
0(e<tka<t)=(e<tka<t)
"(e<tka<t) + (1")(e<tka<t)= 1:
§5.2Bad Priors 73
Therefore Lemma 4.14 implies that for all a2Aand all policies ~
V~
0(æ<ta) ="(e<tka<t)
0(e<tka<t)V~
(æ<ta) + (1")(e<tka<t)
0(e<tka<t)V~
(æ<ta)
="V~
(æ<ta) + (1")V~
(æ<ta): (5.1)
Let:=(æ<t)be the next action according to , and let6=be any other
action. We have that V
(æ<t) =V
(æ<t)since=andæ<tis consistent
with. Therefore we get from (5.1)
V
0(æ<t)V
0(æ<t) ="V
(æ<t) + (1")V
(æ<t) =V
(æ<t)>";
V
0(æ<t) ="V
0
(æ<t) + (1")V
0
(æ<t) ="V
0
(æ<ta) + (1")0":
HenceV
0(æ<t)>V
0(æ<t)and thus the action taken byis the only 0-optimal
action for the history æ<t.
Corollary 5.6 (With Finite Horizon Every Policy is Bayes Optimal) .Ifm= 0for
somem2N, then for any deterministic policy there is a Bayesian mixture 0such
that(æ<t)is the only 0-optimal action for all histories æ<tconsistent with and
tm.
In contrast to Theorem 5.4 where every policy is 0-optimal for a ﬁxed Bayesian
mixture0, Corollary 5.6 gives a diﬀerent Bayesian mixture 0for every policy such
thatis theonly0-optimal policy.
Proof.Let">0be small enough such that V
(æ<t)>"for allæ<tandtm. (This
is possible because (AE )mis ﬁnite by Assumption 4.6c.) We use the dogmatic prior
from Theorem 5.5 to construct a Bayesian mixture 0for the policy and">0. Thus
for any history æ<t2(AE )consistent with andtm, the action (æ<t)is the
only0-optimal action.
Corollary 5.7 (AIXI Emulating Computable Policies) .Let" > 0and letbe any
computable policy. There is a Bayesian mixture 0such that for any 0-optimal policy

0and for any environment ,
V
0
()V
()< ":
Proof.From the proof of Corollary 5.6 and Lemma 4.18.
5.2.3 The Gödel Prior
This section introduces a prior that prevents any ﬁxed formal system from making any
statements about the outcome of all but ﬁnitely many computations. It is named after
Gödel (1931) who famously showed that for any suﬃciently rich formal system there
are statements that it can neither prove nor disprove.
74 Optimality
This prior is targeted at AIXI tl, a computable approximation to AIXI deﬁned by
Hutter (2005, Sec. 7.2). AIXI tlaims to perform as least as well as the best agent who
is limited by time tand spacelthat can be veriﬁed using a proof of length at most
nfor some ﬁxed n2N. The core idea is to enumerate all deterministic policies and
proofs and then execute the policy for which the best value has been proved.
In order to be veriﬁed, a policy has to be computed by a program pwhich fulﬁlls
theveriﬁcation condition VA(p)(Hutter, 2005, Eq. 7.7). This program pnot only
computes future actions of , but also hypothetical past actions a0
iand lower bounds
vifor the value of the policy :
VA(p) :=“8k8(va0æ)1:k:
p(æ<k) =v1a0
1:::vka0
k!vkV
(æ<k)
”;
whereis the policy derived from paccording to (æ<k) :=a0
k.
We ﬁx some formal system that we use to prove the veriﬁcation condition. We
want it to be suﬃciently powerful, but this incurs Gödel incompleteness. For simplicity
of exposition we pick PA, the system of Peano arithmetic (Shoenﬁeld, 1967, Ch. 8.1),
but our result generalizes trivially to all formal systems which cannot prove their own
consistency.
Letnbe a ﬁxed constant. The algorithm for AIXI tlis speciﬁed as follows.
1. LetP=;. This will be the set of veriﬁed programs.
2. For all proofs in PAof lengthn: if the proof proves VA(p)for somep, andjpjl,
then add the program ptoP.
3. For each input history æ<krepeat: run all programs from Pfor at most tsteps each,
take the one with the highest promised value vk, and return that program’s policy’s
action.
Theorem 5.8 (The Gödel Prior) .There is a UTM U0such that if PAis consistent,
then the set of veriﬁed programs Pis empty for all t,l, andn.
Proof.Letqdenote an algorithm that never halts, but for which this cannot be proved
inPA; e.g., letqenumerate all consequences of PAand halt as soon as it ﬁnds a
contradiction. Since we assumed that PAis consistent, qnever halts. Deﬁne the UTM
U0(p;a1:k)as follows.
•Runqforksteps.
•Ifqhalts, output vk= 2.
•RunU(p;a1:k).
Sinceqnever halts, UandU0are functionally identical, therefore U0is universal. Note
thatPAproves8p: U(p;a1:k) =U0(p;a1:k)for any ﬁxed k, but PAdoes not prove
8k8p:U(p;a1:k) =U0(p;a1:k).
§5.3Bayes Optimality 75
Ifqdid eventually halt, it would output a value vk= 2that is too high, since the
value function V
is bounded by 1from above, which PAknows. Hence PAproves that
qhalts!8p::VA(p) (5.2)
IfPAcould prove VA(p)for anyp, then PAwould prove that qdoes not halt since this
is the contrapositive of (5.2). Therefore the set Premains empty.
AIXItlexhibits all the problems of the arbitrariness of the UTM illustrated by the
indiﬀerence prior (Theorem 5.4) and the dogmatic prior (Theorem 5.5). In addition, it
is also susceptible to Gödel incompleteness as illustrated by the Gödel prior in Theo-
rem 5.8. The formal system that is a parameter to AIXI tljust provides another point
of failure.
As a computable approximation to AIXI, AIXI tlis needlessly complicated. As we
prove in Corollary 6.13, "-optimal AIXI is limit computable, so we can approximate it
with an anytime algorithm. Bounding the computational resources to the approxima-
tion algorithm already yields a computable version of AIXI. Moreover, unlike AIXI tl,
this approximation actually converges to AIXI in the limit. Furthermore, we can ‘speed
up’ this approximation algorithm using Hutter search (Hutter, 2002b); this is very sim-
ilar but not identical to AIXI tl.
5.3 Bayes Optimality
The aim of the Legg-Hutter intelligence measure is to formalize the intuitive notion
of intelligence mathematically. Legg and Hutter (2007a) collect various deﬁnitions of
intelligence across many academic ﬁelds and destill it into the following statement (Legg
and Hutter, 2007b)
Intelligence measures an agent’s ability to achieve goals in a wide range of
environments.
This deﬁnition is formalized as follows.
Deﬁnition 5.9 (Legg-Hutter Intelligence; Legg and Hutter, 2007b, Sec. 3.3)) .The
(Legg-Hutter) intelligence of a policy is deﬁned as
() :=X
2Mw()V
()
The Legg-Hutter intelligence of a policy is thet0-value that achieves across all
environments from the class Mweighted by the prior w. Legg and Hutter (2007b) con-
sider a subclass of MCCS
LSC, the class of computable measures together with a Solomonoﬀ
priorw() = 2K()and do not use discounting explicitly.
Typically, the index is omitted when writing . However, in this section we
consider the intelligence measure with respect to diﬀerent priors, therefore we make
this dependency explicit. The following proposition motivates the use of the index
instead ofw.
76 Optimality
Proposition 5.10 (Bayes Optimality = Maximal Intelligence) .() =V
()for all
policies.
Proof.Follows directly from (4.6) and Deﬁnition 5.9.
Deﬁnition 5.11 (Balanced Pareto Optimality; Hutter, 2005, Def. 5.22) .LetMbe a
set of environments. A policy isbalanced Pareto optimal in the set of environments
Miﬀ for all policies ~,X
2Mw()
V
()V~
()
0:
Proposition 5.12 (Balanced Pareto Optimality = Maximal Intelligence) .A policy
is balanced Pareto optimal in Mif and only if has maximal Legg-Hutter intelligence.
Proof.From (4.6) we get
X
2Mw()
V
()V
()
=X
2Mw()V
()X
2Mw()V
()
=V
()V
()
=V
()sup
~V~
()
= ()sup
~(~)
by Proposition 5.10. This term is nonnegative iﬀ ()is maximal.
As a consequence of Proposition 5.10 and Proposition 5.12 we get that AIXI is
balanced Pareto optimal (Hutter, 2005, Thm. 5.24) and has maximal Legg-Hutter in-
telligence.
:= sup
() = sup
V
() =V

() = (
):
This is not surprising since Legg-Hutter intelligence was deﬁned in terms of the t0-
value in the Bayes mixture. Moreover, because the value function is scaled to be in the
interval [0;1], intelligence is a real number between 0and1.
It is just as hard to score very high on the Legg-Hutter intelligence measure as it
is to score very low: we can always turn a reward minimizer into a reward maximizer
by inverting the rewards r0
t:= 1rt. Hence the lowest possible intelligence score is
achieved by AIXI’s twin sister, a -expected reward minimizer:
:= inf
() = inf
V
()
The heaven environment (reward 1forever) and the hell environment (reward 0forever)
are computable and thus in the environment class MCCS
LSC; therefore it is impossible to
get a reward 0or reward 1in every environment. Consequently, for all policies ,
0<()<1:
§5.3Bayes Optimality 77
0 1
random AI0AI
image of
Figure 5.1: The Legg-Hutter intelligence measure assigns values within the closed
interval [;]; the assigned values are depicted in purple. By Theorem 5.13, com-
putable policies are dense in this purple set.
For every real number r2[;]there is a policy with () =r: analogously to
Lemma 4.14 we can deﬁne such that with probability (r)=()it follows

and otherwise it follows arg min~V~
().
Figure 5.1 illustrates the intelligence measure . It is natural to ﬁx the policy
randomthat takes actions uniformly at random to have an intelligence score of 1=2by
choosing a ‘symmetric’ universal prior (Legg and Veness, 2013).
AIXI is not computable (Theorem 6.15), hence there is no computable policy
such that () = or() =for any Bayesian mixture overMCCS
LSC. But the
following theorem states that computable policies can come arbitrarily close. This is
no surprise: by Lemma 4.17 we can do well on a Legg-Hutter intelligence test simply
by memorizing what AIXI would do for the ﬁrst ksteps; as long as kis chosen large
enough such that discounting makes the remaining rewards contribute very little to the
value function.
Theorem 5.13 (Computable Policies are Dense) .The set
f()jis a computable policy g
is dense in the set [;].
Proof.Letbe any policy and let ">0. We need to show that there is a computable
policy ~withj(~)()j<". We choose mlarge enough such that m=1<"=3.
Let2Abe arbitrary and deﬁne the policy
~(ajæ<t) :=8
>><
>>:(ajæ<t)("=3)mift<m;
1 iftmanda=;and
0 otherwise:
By choosing an appropriate rational number in the interval [(ajæ<t)("=3)m;(aj
æ<t) + ("=3)m]we can make the policy ~computable because we can store these
approximations to the action probabilities of for the ﬁrst m1steps in a lookup
table. From Lemma 4.17 we get
V;m
()V~;m
()Dm1(;~j)
("=3)mm="
3
78 Optimality
and together with Lemma 4.16 this yields
j()(~)j=V
()V~
()V;m
()V~;m
()+ 2m
1"
3+ 2m
1<":
Remark5.14 (DeterministicPoliciesarenotDensein [;]).Theintelligencevalues
of deterministic policies are generally not dense in the interval [;]. We show this
by deﬁning an environment where the ﬁrst action determines whether the agent
goes to heaven or hell: action leads to heaven and action leads to hell. Deﬁne
Bayesian mixture 0:= 0:999+ 0:001and letbe any policy. If takes action
ﬁrst, then 0()>0:999. Iftakes action ﬁrst, then 0()<0:001. Hence
there are no deterministic policies that score an intelligence value in the closed interval
[0:001;0:999]. 3
Legg-Hutterintelligenceismeasuredwithrespecttoaﬁxedprior. TheBayesagentis
the most intelligent policy if it uses the same prior . We use the results from Section 5.2
to show that the intelligence score of the Bayes agent can be arbitrary close to the
minimum intelligence score .
Corollary 5.15 (Some AIXIs are Stupid) .For any Bayesian mixture overMCCS
LSC
and every">0, there is a Bayesian mixture 0such that (
0)<+".
Proof.Let" > 0. According to Theorem 5.13, there is a computable policy such
that ()<+"=2. From Corollary 5.7 we get a Bayesian mixture 0such that
j(
0)()j=jV
0
()V
()j<"=2, hence
j(
0)jj(
0)()j+j()j<"=2 +"=2 =":
We get the same result if we ﬁx AIXI, but rig the intelligence measure.
Corollary 5.16 (AIXI is Stupid for Some ).For any deterministic -optimal policy

and for every " > 0there is a Bayesian mixture 0such that 0(
)"and
01".
Proof.Leta1:=
()be the ﬁrst action that
takes. We deﬁne an environment
such that taking the ﬁrst action a1leads to hell and taking any other ﬁrst action leads
to heaven as in Remark 5.14. We deﬁne the Bayesian mixture 0:= (1")+". Since

takes action a1ﬁrst, it goes to hell, i.e., V
() = 0. Hence with Lemma 4.14
0(
) =V

0() = (1")V
() +"V

()":
For any policy that takes an action other than a1ﬁrst, we get
0() =V
0() = (1")V
() +"V
()1":
On the other hand, we can make any computable policy smart if we choose the right
Bayesian mixture. In particular, we get that there is a Bayesian mixture such that ‘do
nothing’ is the most intelligent policy save for some ".
§5.4Asymptotic Optimality 79
name deﬁnition
strong a.o. V
(æ<t)V
(æ<t)!0-almost surely
a.o. in mean E

V
(æ<t)V
(æ<t)
!0
a.o. in probability 8">0:
V
(æ<t)V
(æ<t)>"
!0
weak a.o.1
tPt
k=1
V
(æ<k)V
(æ<k)
!0-almost surely
Table 5.1: The formal deﬁnition of diﬀerent types of asymptotic optimality. In each
case we understand the limit as t!1.
Corollary 5.17 (Computable Policies can be Smart) .For any computable policy and
any">0there is a Bayesian mixture 0such that 0()>0".
Proof.Corollary5.7yieldsaBayesianmixture 0withj00()j=jV
0()V
0()j<
".
5.4 Asymptotic Optimality
An asymptotically optimal policy is a policy learns to act optimally in every environ-
ment fromM, i.e., the value of this policy converges to the optimal value.
Deﬁnition 5.18 (Asymptotic Optimality) .A policyisasymptotically optimal in an
environment class Miﬀ for all2M
V
(æ<t)V
(æ<t)!0ast!1 (5.3)
on histories drawn from .
There are diﬀerent types of asymptotic optimality based on the type of stochas-
tic convergence in (5.3); see Deﬁnition 2.5. If this convergence occurs almost surely,
it is called strong asymptotic optimality (Lattimore and Hutter, 2011, Def. 7); if this
convergence occurs in mean, it is called asymptotic optimality in mean ; if this conver-
gence occurs in probability, it is called asymptotic optimality in probability ; and if the
Cesàro averages converge almost surely, it is called weak asymptotic optimality (Lat-
timore and Hutter, 2011, Def. 7). Since the value function is a nonnegative bounded
random variable, asymptotic optimality in mean and asymptotic optimality in proba-
bility are equivalent. See Table 5.1 for the explicit deﬁnitions and see Figure 5.2 for an
overview over their relationship.
Asymptotic optimality in probability is in spirit a probably approximately cor-
rect (PAC) result: for all ">0and>0the probability that our policy is "-suboptimal
converges to zero; eventually this probability will be less than . For a PAC result it is
typically demanded that the number of time steps until the probability is less than
80 Optimality
strong a.o.
weak a.o. a.o. in mean a.o. in probability
Figure 5.2: The relationship between diﬀerent types of asymptotic optimality. Each
arrow indicates a logical implication and each lack of an arrow indicates that there is
no logical implication.
be polynomial in 1="and1=. In general environments this is impossible, and here we
have no ambition to provide concrete convergence rates.
Intuitively, a necessary condition for asymptotic optimality is that the agent needs
toexploreinﬁnitelyoftenforanentireeﬀectivehorizon. Ifweexploreonlyﬁnitelyoften,
then the environment might change after we stopped exploring. Moreover, the agent
needs to predict the value of counterfactual policies accurately; but by Lemma 4.16
only for an "-eﬀective horizon. By committing to exploration for the entire eﬀective
horizon, we learn about the value of counterfactual policies.
Example 5.19 (Exploration Inﬁnitely Often for an Entire Eﬀective Horizon) .If there
is an" > 0such that the policy does not explore for Ht(")steps inﬁnitely of-
ten, thenV
(æ<t)V
(æ<t)> "inﬁnitely often. Deﬁne A:=f;gandE:=
f0;"=2;1g(observations are vacuous) and consider the following class of environments
M:=f1;1;2;:::g(transitions are labeled with condition: action, reward):
s0;"
2
;0s0 s1 ::: sn;"
2
t<k :;0tk:;0 ;0 ;0
;0
;0
;0;1
1 k
Environment kworks just like environment 1, except that at time step kthe path to
states1gets unlocked. The length of the state sequence in kis deﬁned as an "-eﬀective
horizon,n:=Ht(")wheretis the time step in which the agent leaves state s0. The
optimal policy in environment 1is to always take action , the optimal policy for
environment kis to take action fort<kand then take action . Suppose the agent
is in time step tand in state s0. Since these environments are partially observable,
it needs to explore for nsteps (take action  ntimes) to distinguish 1fromkfor
anykt. Since there are inﬁnitely many k, the agent needs to do this inﬁnitely
often. Moreover, V
1"andV
1="=2, so iftis the true environment, then not
exploring to the right for an "-eﬀective horizon is suboptimal by "=2. But if1is the
true environment, then exploring incurs an opportunity cost of one reward of "=2.3
§5.4Asymptotic Optimality 81
Next, we state two negative results about asymptotic optimality proved by Latti-
more and Hutter (2011). It is important to emphasize that Theorem 5.20 and Theo-
rem 5.21 only hold for deterministic policies.
Theorem 5.20 (Deterministic Policies are not Strongly Asymptotically Optimal; Lat-
timoreandHutter,2011, Thm.8) .There is no deterministic policy that is strong asymp-
totically optimal in the class MCCM
comp.
If the horizon grows linearly (for example, power discounting
(t) =twith >1;
see Table 4.1), then a deterministic policy cannot be weakly asymptotically optimal
policy: the agent has to explore for an entire eﬀective horizon, which prevents the
Cesàro average from converging.
Theorem 5.21 (Necessary Condition for Weak Asymptotic Optimality; Lattimore,
2013, Thm. 5.5) .If there is an ">0such thatHt(")=2o(t), then there is no determin-
istic policy that is weakly asymptotically optimal in the class MCCM
comp.
There are several agents that achieve asymptotic optimality. In the rest of this
section, we discuss the Bayes agent, BayesExp, and Thompson sampling. Asymptotic
optimalitycan alsobe achieved through optimism(SunehagandHutter,2012a,b,2015).
5.4.1 Bayes
In this section, we list two results from the literature regarding the asymptotic optimal-
ity of the Bayes optimal policy. The following negative result is due to Orseau (2010,
2013).
Theorem5.22 (BayesisnotAsymptoticallyOptimalinGeneralEnvironments; Orseau,
2013, Thm. 4) .For any classMMCCM
compno Bayes optimal policy
is asymptotically
optimal: there is an environment 2Mand a time step t02Nsuch that
-almost
surely for all time steps tt0
V
(æ<t)V
(æ<t) =1
2:
Orseau calls this result the good enough eﬀect : A Bayesian agent eventually decides
that the current strategy is good enough and that any additional exploration is not
worth its expected payoﬀ. However, if the environment changes afterwards, the Bayes
agent is acting suboptimally.
Proof.Without loss of generality assume A:=f;gandE:=f0;1=2;1g(observations
are vacuous). We consider the following environment (transitions are labeled with
action, reward).
82 Optimality
s0 s1 ::: sn;1
2
;0;0;0
;0
Instates0theactionistheexploitationactionandtheaction theexplorationaction.
The length of the state sequence is deﬁned as an 1=t-eﬀective horizon, n:=Ht(1=t)
wheretis the time step in which the agent leaves state s0. Since the discount function

is computable by Assumption 4.6a, 2MCCS
LSC.
Assume that when acting in , the Bayes agent explores inﬁnitely often. Let æ<t
be a history in which the agent is in state s0and takes action . ThenV
1=t.
By on-policy value convergence (Corollary 4.20), V
(æ<t)V
(æ<t)!0
-almost
surely. Hence there is a time step t0such that for all tt0we haveV
< w()=2.
Sinceis deterministic, w(jæ<t)w(). Now we get a contradiction from (4.7):
V
(æ<t)w(jæ<t)V
(æ<t)w()V
(æ<t) =w()
2>V
(æ<t)
Therefore the Bayes agent stops taking the exploration action after time step t0,
and so it is not optimal in any 2MCCS
LSCthat behaves like until time step t0and
then changes:
s0 s1 ::: sn;1
2
t>t 0:;1tt0:;0;0;0
;0
The following theorem is also known as the self-optimizing theorem . This theorem
has been a source of great confusion because its statement in Hutter (2005, Thm. 5.34)
is not very explicit about how the histories are generated. The formulation of Lattimore
(2013, Thm. 5.2) is explicit, but less general.
Theorem 5.23 (Suﬃcient Condition for Strong Asymptotic Optimality of Bayes; Hut-
ter, 2005, Thm. 5.34) .Letbe some environment. If there is a policy and a sequence
of policies1;2;:::such that for all 2M
V
(æ<t)Vt(æ<t)!0ast!1-almost surely, (5.4)
then
V
(æ<t)V
(æ<t)!0ast!1-almost surely.
§5.4Asymptotic Optimality 83
If=
and(5.4)holds for all 2M, then
is strongly asymptotically optimal in
the classM.
It is important to emphasize that the policies 1;2;:::need to converge to the
optimal value on the history generated by and, and not (as one might think) and
t. Intuitively, the policy is an ‘exploration policy’ that ensures that the environment
class is explored suﬃciently. Typically, a policy is asymptotically optimal on its own
history. So if =1=2=:::, then we get that Bayes is asymptotically optimal on
the history generated by the policy , not its own history. In light of Theorem 5.5 and
Theorem 5.22 this is not too surprising; Bayesian reinforcement learning agents might
not explore enough to be asymptotically optimal, but given a policy that does explore
enough, Bayes learns enough to be asymptotically optimal.
This invites us to deﬁne the following policies t: follow the information-seeking
policy
IGuntil time step t, and then follow
(explore until t, then exploit). Since the
information-seeking policy explores enough to prove oﬀ-policy prediction (Orseau et al.,
2013, Thm. 7), we get V
V
!0for every policy uniformly. Hence arg maxV
!
arg maxV
and thusV
V
!0and (5.4) is satisﬁed. From Theorem 5.23 we get
V
V
!0, which we already knew. In order to get strong asymptotic optimality,
all we need to do is choose the switching time step tappropriately, i.e., wait until V

andV
are close enough. Unfortunately, this is an invalid strategy: the agent does
not know the true environment and hence cannot check this condition.
Hutter (2005, Sec. 5.6) uses Theorem 5.23 to show that the Bayes optimal policy is
strongly asymptotically optimal in the class of ergodic ﬁnite-state MDPs if the eﬀective
horizon is growing, i.e., Ht(")!1for all">0. This relies on the fact that in ergodic
ﬁnite-state MDPs we need a ﬁxed number of steps to explore the entire environment up
to"-conﬁdence. Thereforewecandeﬁneasequenceofpolicies 1;2;:::thatcompletely
disregard the history and start exploring everything from scratch. Since the eﬀective
horizon is growing, this exploration phase takes a vanishing fraction of eﬀective horizon
and most of the value is retained. Therefore the sequence of policies 1;2;:::satisﬁes
the condition of Theorem 5.23 regardless of the history, thus in particular for the history
generated by =
and any2M. Note that the condition on the horizon is
important: If the eﬀective horizon is bounded, then Bayes is not asymptotically optimal
in the class of ergodic ﬁnite-state MDPs because it can be locked into a dogmatic prior
similarly to Theorem 5.5.
Proof of Theorem 5.23. From (4.6) we get for any history æ<t
w(jæ<t)
V
(æ<t)V
(æ<t)
X
2Mw(jæ<t)
V
(æ<t)V
(æ<t)
= X
2Mw(jæ<t)V
(æ<t)!
V

(æ<t)
X
2Mw(jæ<t)V
(æ<t)Vt
(æ<t)
84 Optimality
=X
2Mw(jæ<t)
V
(æ<t)Vt(æ<t)
:(5.5)
From (5.4) follows that V
Vt!0-almost surely for all 2M, so (5.5) converges
to0-almost surely (Hutter, 2005, Lem. 5.28ii). Similar to Example 3.20, 1=w(j
æ<t)is a nonnegative -martingale and thus converges (to a ﬁnite value) -almost
surely by Theorem 2.8. Therefore V
(æ<t)V
(æ<t)!0-almost surely. If this
is true for all 2M, the strong asymptotic optimality of
follows from =
by
deﬁnition.
5.4.2 BayesExp
The deﬁnition of BayesExp is given in Section 4.3.3. In this subsection we state a result
by Lattimore (2013) that motivated the deﬁnition of BayesExp.
Theorem 5.24 (BayesExp is Weakly Asymptotically Optimal; Lattimore, 2013, Thm.
5.6).LetBEdenote the policy from Algorithm 1. If Ht(")grows monotone in tand
Ht("t)="t2o(t), then for all environments 2M
1
ttX
k=1
V
(æ<k)VBE(æ<k)
!0ast!1BE-almost surely :
If the horizon grows sublinearly ( Ht(")2o(t)for all" >0), then we can always
ﬁnd a sequence "t!0that decreases slowly enough such that Ht(")="t2o(t)holds.
5.4.3 Thompson Sampling
In this section we prove that the Thompson sampling policy deﬁned in Section 4.3.4 is
asymptotically optimal. Ortega and Braun (2010) prove that the action probabilities
of Thompson sampling converge to the action probability of the optimal policy almost
surely, but require a ﬁnite environment class and two (arguably quite strong) technical
assumptions on the behavior of the posterior distribution (akin to ergodicity) and the
similarity of environments in the class. Our convergence results do not require these
assumptions.
Theorem 5.25 (Thompson Sampling is Asymptotically Optimal in Mean) .For all
environments 2M,
ET
V
(æ<t)VT(æ<t)
!0ast!1.
This theorem immediately implies that Thompson sampling is also asymptotically
optimal in probability according to Figure 5.2. However, this does not imply almost
sure convergence (see Example 5.28).
We ﬁrst give an intuition for the asymptotic optimality of Thompson sampling. At
every resampling step we can split the class Minto three partitions:
1. Environments whereV
V

§5.4Asymptotic Optimality 85
2. Environments whereV
>V

3. Environments whereV
<V

The ﬁrst class is the class of ‘good’ environments: if we draw one of them, we follow a
policy that is close to optimal in . The second class is the class of environments that
overestimate the value of . Following their optimal policy the agent gains information
becauserewardswillbelowerthanexpected. Thethirdclassistheclassofenvironments
that underestimate the value of . Following their optimal policy the agent might not
gain information since might behave just like environment on the-optimal policy.
However, when sampling from the ﬁrst class instead, the agent gains information about
the third class because rewards tend to be better than environments from the third
class predicted.
Since the true environment 2M, the ﬁrst class is not empty, and the probability
of drawing a sample from the ﬁrst class does not become too small. Whenever the
second and third class have suﬃciently high weight in the posterior, there is a good
chance of picking a policy that leads the agent to gain information. Asymptotically,
the posterior converges, so the agent ends up having learned everything it could, i.e.,
the posterior weight of the second and third class vanishes.
This argument is not too hard to formalize for deterministic environment classes.
However, for stochastic environment classes the eﬀect on the posterior when following
a bad policy is harder to quantify because there is always a chance that the rewards are
diﬀerent simply because of bad luck. In order to prove this theorem in its generality for
stochasticclasses, weemployanentirelydiﬀerentproofstrategythatreliesonstatistical
tools rather than the argument given above.
Deﬁnition 5.26 (Expected Total Variation Distance) .Letbe any policy and let
m2N[1. Theexpected total variation distance on the policy is
F
m(æ<t) :=X
2Mw(jæ<t)Dm(;jæ<t):
If we replace the distance measure Dmby cross-entropy, then the quantity F
m(æ<t)
becomes the expected information gain (see Section 4.3.2).
For the proof of Theorem 5.25 we need the following lemma.
Lemma 5.27 (Expected Total Variation Distance Vanishes On-Policy) .For any policy
and any environment ,
E
[F
1(æ<t)]!0ast!1.
Proof.From Theorem 3.25 we get D1(;jæ<t)!0-almost surely, and since D
is bounded, this convergence also occurs in mean. Thus for every environment 2M,
E

D1(;jæ<t)
!0ast!1:
86 Optimality
Now
E
[F
1(æ<t)]1
w()E
[F
1(æ<t)]
=1
w()E
"X
2Mw(jæ<t)D1(;jæ<t)#
=1
w()E
"X
2Mw()(æ<t)
(æ<t)D1(;jæ<t)#
=1
w()X
2Mw()E

D1(;jæ<t)
!0
by Hutter (2005, Lem. 5.28ii) since total variation distance is bounded.
Proof of Theorem 5.25. Let; > 0and let"t>0denote the sequence used to deﬁne
Tin Algorithm 2. We assume that tis large enough such that "kfor allkt
and thatis small enough such that w(jæ<t)>4for allt, which holds since
w(jæ<t)6!0-almost surely for any policy (Hutter, 2009a, Lem. 3i).
The stochastic process w(jæ<t)is aT-martingale according to Example 3.20.
By the martingale convergence theorem (Theorem 2.8) w(jæ<t)convergesT-almost
surely and because Tw()Tit also converges T-almost surely.
We argue that we can choose t0to be one of T’s resampling time steps large
enough such that for all tt0the following three events hold simultaneously with
T-probability at least 1.
(i) There is a ﬁnite set M0Mwithw(M0jæ<t)>1andw(jæ<k)6!0as
k!1for all2M0.
(ii)jw(M00jæ<t)w(M00jæ<t0)jfor allM00M0.
(iii)FT1(æ<t)<w2
min.
wherewmin:= inffw(jæ<k)jk2N;2M0g, which is positive by (i).
(i) and (ii) are satisﬁed eventually because the posterior w(jæ<t)convergesT-
almost surely. Note that the set M0is random: the limit of w(jæ<t)ast!1
depends on the history æ1:1. Without loss of generality, we assume the true environ-
mentis contained inM0sincew(jæ<t)6!0T-almost surely. (iii) follows from
Lemma 5.27 since convergence in mean implies convergence in probability.
Moreover, we deﬁne the horizon m:=t+Ht("t)as the time step of the eﬀective
horizon at time step t. Letæ<tbe a ﬁxed history for which (i-iii) is satisﬁed. Then we
have
w2
min>FT1(æ<t)
=X
2Mw(jæ<t)D1(T;Tjæ<t)
=Ew(jæ<t)[D1(T;Tjæ<t)]
§5.4Asymptotic Optimality 87
Ew(jæ<t)[Dm(T;Tjæ<t)]
w2
minw(MnM00jæ<t)
by Markov’s inequality where
M00:=
2MDm(T;Tjæ<t)<w2
min
:
For our ﬁxed history æ<twe have
1<w (M00jæ<t)
(i)
w(M00\M0jæ<t) +
(ii)
w(M00\M0jæ<t0) + 2
(i)
w(M00jæ<t0) + 3
and thus we get
14<w
f2MjDm(T;Tjæ<t)<w2
mingæ<t0
: (5.6)
In particular, this bound holds for =sincew(jæ<t0)>4by assumption.
It remains to show that with high probability the value V
of the sample ’s
optimal policy
is suﬃciently close to the -optimal value V
. The worst case is
that we draw the worst sample from M0\M00twice in a row. From now on, let
denote the sample environment we draw at time step t0, and lettdenote some time
step between t0andt1:=t0+Ht0("t0)(before the next resampling). With probability
w(0jæ<t0)w(0jæ<t1)we sample0both att0andt1when following T. Therefore
we have for all æt:mand all2M
T(æ1:mjæ<t)w(0jæ<t0)w(0jæ<t1)
0(æ1:mjæ<t):
Thus we get for all 2M0(in particular and)
Dm(T;Tjæ<t)sup
02Msup
A(AE )mw(0jæ<t0)w(0jæ<t1)
(
0(Ajæ<t)
0(Ajæ<t))
w(jæ<t0)w(jæ<t1) sup
A(AE )m
(Ajæ<t)
(Ajæ<t)
w2
minDm(
;
jæ<t):
For2M00we get with (5.6)
Dm(T;Tjæ<t)Dm(T;Tjæ<t) +Dm(T;Tjæ<t)
<w2
min+w2
min= 2w2
min;
88 Optimality
which together with Lemma 4.17 and the fact that rewards in [0;1]implies
V
(æ<t)V
(æ<t)t+Ht("t)
t+V
;m
(æ<t)V
;m
(æ<t)
"t+Dm(
;
jæ<t)
"t+1
w2
minDm(T;Tjæ<t)
<+ 2= 3:
Hence we get (omitting history arguments æ<tfor simplicity)
V
=V
<V
+ 3V
+ 3=V
+ 3 <V
+ 3+ 3=V
+ 6:(5.7)
WithT-probability at least 1(i), (ii), and (iii) are true, with T-probability
at least 1our sample happens to be inM0by (i), and with w(jæ<t0)-probability
at least 14the sample is inM00by (5.6). All of these events are true simultaneously
with probability at least 1(++ 4) = 16. Hence the bound (5.7) transfers for
Tsuch that with T-probability16we have
V
(æ<t)VT(æ<t)<6:
ThereforeT[V
(æ<t)VT(æ<t)6]<6and with!0we get that V
(æ<t)
VT(æ<t)!0ast!1in probability. The value function is bounded, thus it also
converges in mean.
The following example shows that the Thompson sampling policy is not strongly
asymptotically optimal. However, we expect that strong asymptotic optimality can
be achieved with Thompson sampling by resampling at every time step (with strong
assumptions on the discount function). However, for practical purposes resampling in
every time step is very ineﬃcient.
Example 5.28 (Thompson Sampling is not Strongly Asymptotically Optimal) .Deﬁne
A:=f;g,E:=f0;1=2;1g, and assume geometric discounting (Example 4.5). Con-
sider the following class of environments M:=f1;1;2;:::g(transitions are labeled
with action, reward):
s0
s1
s2;1
2
;0
;0
;0;0s0
s1
s2s3
s4;1
2
t<k :;0
;0
;0;0tk:;0
;0;0
;1
;0
1 k
§5.4Asymptotic Optimality 89
Environment kworks just like environment 1except that after time step k, the path
to states3gets unlocked. The class Mis a class of deterministic weakly communicat-
ing POMDPs (but as a POMDP khas more than 5 states). The optimal policy in
environment 1is to always take action , the optimal policy for environment kis to
take action fort<kand then take action in states1and action otherwise.
Suppose the policy Tis acting in environment 1. Since it is asymptotically
optimal in the class M, it has to take actions froms0inﬁnitely often: for t < k
environment kis indistinguishable from 1, so the posterior for kis larger or equal to
the prior. Hence there is always a constant chance of sampling kuntil taking actions
, at which point all environments kforktbecome falsiﬁed.
If the policy Tdecides to explore and take the ﬁrst action , it will be in state s1.
Letæ<tdenote the current history. Then the 1-optimal action is and
V
1(æ<t) = (1
)
0 +
1
2+
21
2+:::
=
2:
The next action taken by Tissince any optimal policy for any sampled environment
that takes action once, takes that action again (and we are following that policy for
an"t-eﬀective horizon). Hence
VT1(æ<t)(1
)
0 + 0 +
21
2+
31
2+:::
=
2
2:
ThereforeV
1VT1(

2)=2>0. This happens inﬁnitely often with probability
one and thus we cannot get almost sure convergence. 3
If the Bayesian mixture is inside the class M(as it is the case for the class MCCS
LSC),
then we can assign a prior probability that is arbitrarily close to 1. Since the posterior
ofisthesameastheprior, ThompsonsamplingwillactaccordingtotheBayesoptimal
policy most of the time. This means the Bayes-value of Thompson sampling can be
very good; formally, V
()VT
() =(T)can be made arbitrarily small.
In contrast, the Bayes-value of Thompson sampling can also be very bad: Suppose
you have a class of (n+ 1)-armed bandits indexed 1;:::;nwhere bandit igives reward
1"on arm 1, reward 1on armi+ 1, and reward 0on all other arms. For geometric
discounting and "<(1
)=(2
), it is Bayes optimal to pull arm 1while Thompson
sampling will explore on average n=2arms until it ﬁnds the optimal arm. The Bayes-
valueofThompsonsamplingis 1=(n
n1)incontractto (1")achievedbyBayes. For
a horizon of n, the Bayes optimal policy suﬀers a regret of "nand Thompson sampling
a regret ofn=2, which is much larger for small ".
5.4.4 Almost Sure in Cesàro Average vs. in Mean
It might appear that convergence in mean is more natural than the convergence of
Cesàro averages of weak asymptotic optimality. However, both notions are not so
fundamentally diﬀerent because they both allow an inﬁnite number of bad mistakes
(actions that lead to V
V
being large). Asymptotic optimality in mean allows bad
90 Optimality
mistakes as long as their probability converges to zero; weak asymptotic optimality
allows bad mistakes as long as the total time spent on bad mistakes grows sublinearly.
Note that according to Example 5.19 making bad mistakes inﬁnitely often is necessary
for asymptotic optimality.
Theorem 5.24 shows that weak asymptotic optimality is possible in any countable
class of stochastic environments. However, this requires the additional condition that
the eﬀective horizon grows sublinearly, Ht("t)2o(t), while Theorem 5.25 does not
require any condition on the discount function.
Generally, weak asymptotic optimality and asymptotic optimality in mean are in-
comparable because the notions of convergence are incomparable for (bounded) random
variables. First, for deterministic sequences (i.e. deterministic policies in deterministic
environments), convergence in mean is equivalent to (regular) convergence, which is
impossible by Theorem 5.20. Second, convergence in probability (and hence conver-
gence in mean for bounded random variables) does not imply almost sure convergence
of Cesàro averages (Stoyanov, 2013, Sec. 14.18). We leave open the question whether
the policyTis weakly asymptotically optimal.
5.5 Regret
Regret is how many expected rewards the agent forfeits by not following the best in-
formed policy.
Deﬁnition 5.29 (Regret).Theregretof a policy in environment is
Rm(;) := sup
0E0
"mX
t=1rt#
E
"mX
t=1rt#
:
Note that regret is undiscounted and always nonnegative. Moreover, the space of
possible diﬀerent policies for the ﬁrst mactions is ﬁnite and we assumed the set of
actionsAand the set of percepts Eto be ﬁnite (Assumption 4.6c), so the supremum is
always attained by some policy (not necessarily the -optimal policy
because that
policy uses discounting).
Diﬀerent problem classes have diﬀerent regret rates, depending on the structure
and the diﬃculty of the problem class. Multi-armed bandits provide a (problem-
independent)worst-caseregretboundof
(p
km)wherekisthenumberofarms(Bubeck
and Bianchi, 2012). In MDPs the lower bound is
(p
SAdm )whereSis the number
of states,Athe number of actions, and dthe diameter of the MDP (Auer et al., 2010).
For a countable class of environments given by state representation functions that map
histories to MDP states, a regret of ~O(m2=3)is achievable assuming the resulting MDP
is weakly communicating (Nguyen et al., 2013).
A problem class is considered learnable if there is an algorithm that has a sublinear
regret guarantee. The following example shows that the general reinforcement learning
problem is not learnable because the agent can get caught in a trap and be unable to
recover.
§5.5Regret 91
Example 5.30 (Linear Regret; Hutter, 2005, Sec. 5.3.2) .Consider the following two
environments 1and2. In environment 1actionleads to hell (reward 0forever)
and actionleads to heaven (reward 1forever). Environment 2behaves just the same,
except that both actions are swapped.
hell heavenreward = 0 reward = 1
hell heavenreward = 0 reward = 1

1 2
The policy that takes action in the ﬁrst time step performs well in 2but performs
poorly in1. Likewise, the policy that takes action in the ﬁrst time step performs
well in1but performs poorly in 2. Regardless of which policy we adopt, our regret
is always linear in one of the environments 1or2:
Rm(; 1) =m R m(; 2) = 0
Rm(; 1) = 0 Rm(; 2) =m 3
To achieve sublinear regret we need to ensure that the agent can recover from
mistakes. Formally, we make the following assumption.
Deﬁnition 5.31 (Recoverability) .An environment satisﬁes the recoverability as-
sumption iﬀ
sup
E
[V
(æ<t)]E
[V
(æ<t)]!0ast!1:
Recoverability compares following the worst policy fort1time steps and then
switching to the optimal policy
to having followed
from the beginning. The
recoverability assumption states that switching to the optimal policy at any time step
enables the recovery of most of the value: it has to become less costly to recover from
mistakes as time progresses. This should be regarded as an eﬀect of the discount
function: if the (eﬀective) horizon grows, recovery becomes easier because the optimal
policy has more time to perform a recovery. Moreover, recoverability is on the optimal
policy, in contrast to the notion of ergodicity in MDPs which demands returning to a
starting state regardless of the policy.
Remark 5.32 (Weakly Communicating POMDPs are Recoverable) .If the eﬀective
horizon is growing, Ht(")!1ast!1, then any weakly communicating ﬁnite state
POMDP satisﬁes the recoverability assumption. 3
5.5.1 Sublinear Regret in Recoverable Environments
This subsection is dedicated to the following theorem that connects asymptotic opti-
mality in mean to sublinear regret.
92 Optimality
Theorem 5.33 (Sublinear Regret in Recoverable Environments) .If the discount func-
tion
satisﬁes Assumption 5.34, the environment satisﬁes the recoverability assump-
tion, andis asymptotically optimal in mean in the class fg, thenRm(;)2o(m).
Assumption 5.34 (Discount Function) .Let the discount function
be such that
(a)
t>0for allt,
(b)
tis monotone decreasing in t, and
(c)Ht(")2o(t)for all">0.
This assumption demands that the discount function is somewhat well-behaved: the
function has no oscillations, does not become 0, and the horizon is not growing too fast.
It is satisﬁed by geometric discounting (Example 4.5): (a)
t>0, (b)
monotone
decreasing, and (c) Ht(") =dlog
"e2o(t).
The problem with geometric discounting is that it makes the recoverability assump-
tion very strong: since the horizon is not growing, the environment has to enable faster
recovery as time progresses; in this case weakly communicating POMDPs are notre-
coverable. A choice with Ht(")!1that satisﬁes Assumption 5.34 is subgeometric
discounting
t:=ep
t=p
t(see Table 4.1).
If the items in Assumption 5.34 are violated, Theorem 5.33 can fail:
•If
t= 0for some time steps t, our policy does not care about those time steps
and might take actions that have large regret.
•Similarly if
oscillates between high values and very low values: our policy might
take high-regret actions in time steps with comparatively lower
-weight.
•If the horizon grows linearly, inﬁnitely often our policy might spend some constant
fraction of the current eﬀective horizon exploring, which incurs a cost that is a
constant fraction of the total regret so far.
To prove Theorem 5.33 we require the following technical lemma.
Lemma 5.35 (Value and Regret) .Let" > 0and assume the discount function
satisﬁes Assumption 5.34. Let (dt)t2Nbe a sequence of numbers with jdtj1for allt.
If there is a time step t0with
1
t1X
k=t
kdk<"8tt0 (5.8)
thenmX
t=1dtt0+"(mt0+ 1) +1 +"
1"Hm("):
Proof.This proof essentially follows the proof of Hutter (2006b, Thm. 17).
§5.5Regret 93
By Assumption 5.34a we have
t>0for alltand hence t>0for allt. By
Assumption 5.34b have that
is monotone decreasing, so we get for all n2N
t=1X
k=t
kt+n1X
k=t
t+1X
k=t+n
k=n
t+ t+n:
And withn:=Ht(")this yields

tHt(")
t1t+Ht(")
t1">0: (5.9)
In particular, this bound holds for all tand">0.
Next, we deﬁne a series of nonnegative weights (bt)t1such that
mX
t=t0dk=mX
t=t0bt
tmX
k=t
kdk:
This yields the constraints
tX
k=t0bk
k
t= 18tt0:
The solution to these constraints is
bt0=t0

t0;andbt=t

tt

t1fort>t 0: (5.10)
Thus we get
mX
t=t0bt=t0

t0+mX
t=t0+1t

tt

t1
=m+1

m+mX
t=t0t

tt+1

t
=m+1

m+mt0+ 1
Hm(")
1"+mt0+ 1
for all">0according to (5.9).
Finally,
mX
t=1dtt0X
t=1dt+mX
t=t0bt
tmX
k=t
kdk
t0+mX
t=t0bt
t1X
k=t
kdkmX
t=t0bt
t1X
k=m+1
kdk
94 Optimality
and using the assumption (5.8) and dt1,
<t0+mX
t=t0bt"+mX
t=t0btm+1
t
t0+"Hm(")
1"+"(mt0+ 1) +mX
t=t0btm+1
t
For the latter term we substitute (5.10) to get
mX
t=t0btm+1
t=m+1

t0+mX
t=t0+1m+1

tm+1

t1
=m+1

mHm(")
1"
with (5.9).
Proof of Theorem 5.33. Let(m)m2Ndenote any sequence of policies, such as a se-
quence of policies that attain the supremum in the deﬁnition of regret. We want to
show that
Em"mX
t=1rt#
E
"mX
t=1rt#
2o(m):
For
d(m)
k:=Em[rk]E
[rk] (5.11)
we have1d(m)
k1since we assumed rewards to be bounded between 0and1.
Because the environment satisﬁes the recoverability assumption we have
E
[V
(æ<t)]E
[V
(æ<t)]!0ast!1;and
sup
mE
[V
(æ<t)]Em[V
(æ<t)]!0ast!1;
so we conclude that
sup
mE
[V
(æ<t)]Em[V
(æ<t)]!0
by the triangle inequality and thus
sup
mEm[V
(æ<t)]E
[V
(æ<t)]!0ast!1: (5.12)
By assumption the policy is asymptotically optimal in mean, so we have
E
[V
(æ<t)]E
[V
(æ<t)]!0ast!1;
and with (5.12) this combines to
sup
mEm[V
(æ<t)]E
[V
(æ<t)]!0ast!1:
§5.5Regret 95
FromV
(æ<t)Vm(æ<t)we get
lim sup
t!1
sup
mEm[Vm(æ<t)]E
[V
(æ<t)]
0: (5.13)
For02f; 1;2;:::gwe have
E0
[V0
(æ<t)] =E0
"
1
tE0
"1X
k=t
krkæ<t##
=E0
"
1
t1X
k=t
krk#
=1
t1X
k=t
kE0
[rk];
so from (5.11) and (5.13) we get
lim sup
t!1sup
m1
t1X
k=t
kd(m)
k0:
Let" > 0. We choose t0independent of mand large enough such that we get
supmP1
k=t
kd(m)
k=t< "for alltt0. Now we let m2Nbe given and apply
Lemma 5.35 to get
Rm(;)
m=Pm
k=1d(m)
k
mt0+"(mt0+ 1) +1+"
1"Hm(")
m:
SinceHt(")2o(t)according to Assumption 5.34c we get lim supm!1Rm(;)=m
0.
Example 5.36 (The Converse of Theorem 5.33 is False) .Letbe a two-armed
Bernoulli bandit with means 0and1and suppose we are using geometric discount-
ing with discount factor
2[0;1). This environment is recoverable. If our policy
pulls the suboptimal arm exactly on time steps 1;2;4;8;16;:::, regret will be log-
arithmic. However, on time steps t= 2nforn2Nthe value diﬀerence V
V
is
deterministically at least 1
 >0. 3
Note that Example 5.36 does not rule out weak asymptotic optimality.
5.5.2 Regret of the Optimal Policy and Thompson sampling
We get the following immediate consequence.
Corollary 5.37 (Sublinear Regret for the Optimal Discounted Policy) .If the discount
function
satisﬁes Assumption 5.34 and the environment satisﬁes the recoverability
assumption, then Rm(
;)2o(m).
Proof.From Theorem 5.33 since the policy
is (trivially) asymptotically optimal in
fg.
If the environment does not satisfy the recoverability assumption, regret may be
lineareven on the optimal policy : the optimal policy maximizes discounted rewards
96 Optimality
and this short-sightedness might incur a tradeoﬀ that leads to linear regret later on if
the environment does not allow recovery.
Corollary 5.38 (Sublinear Regret for Thompson Sampling) .If the discount function

satisﬁes Assumption 5.34 and the environment 2Msatisﬁes the recoverability
assumption, then Rm(T;)2o(m)for the Thompson sampling policy T.
Proof.From Theorem 5.25 and Theorem 5.33.
5.6 Discussion
In this work, we disregard computational constraints. Because of this, our agents learn
very eﬃciently and we can focus on the way they balance exploration and exploitation.
So which balance is best?
5.6.1 The Optimality of AIXI
Bayesian reinforcement learning agents make the tradeoﬀ between exploration and ex-
ploitationintheBayesoptimalway. Maximizingexpectedrewardsaccordingtoanypos-
itive prior does not lead to enough exploration to achieve asymptotic optimality (Theo-
rem 5.22); the prior’s bias is retained indeﬁnitely. For bad priors this can cause serious
malfunctions: the dogmatic prior deﬁned in Section 5.2.2 can prevent a Bayesian agent
from taking a single exploratory action ; exploration is restricted to cases where the ex-
pected future payoﬀ falls below some prespeciﬁed ">0. However, this problem can be
alleviated by adding an extra exploration component to AIXI: Lattimore (2013) shows
that BayesExp is weakly asymptotically optimal (Theorem 5.24).
So instead, we may ask the following weaker questions. Does AIXI succeed in every
(ergodic) ﬁnite-state (PO)MDP, bandit problem, or sequence prediction task? Our
results imply that without further assumptions on the prior, we cannot answer any of
the preceding questions in the aﬃrmative. Using a dogmatic prior (Theorem 5.5), we
can make AIXI follow any computable policy as long as that policy produces rewards
that are bounded away from zero.
•In a sequence prediction task that gives a reward of 1for every correctly predicted
bit and 0otherwise, a policy that correctly predicts every third bit will receive
an average reward of 1=3. With a-dogmatic prior, AIXI thus only predicts
a third of the bits correctly, and hence is outperformed by a uniformly random
predictor.
However, if we have a constant horizon of length 1, AIXIdoessucceed in sequence
prediction (Hutter, 2005, Sec. 6.2.2). If the horizon is this short, the agent is so
hedonistic that no threat of hell can deter it.
•In a (PO)MDP a dogmatic prior can make AIXI get stuck in any loop that
provides nonzero expected rewards.
§5.6Discussion 97
•In a bandit problem, a dogmatic prior can make AIXI get stuck on any arm which
provides nonzero expected rewards.
These results apply not only to AIXI, but generally to Bayesian reinforcement learn-
ing agents. Any Bayesian mixture over nonrecoverable environments is susceptible to
dogmatic priors if we allow an arbitrary reweighing of the prior. Notable exceptions
are classes of environment that allow policies that are strongly asymptotically optimal
regardless of the history (Theorem 5.23). For example, the class of all ergodic MDPs
for an unbounded eﬀective horizon; in this case the Bayes optimal policy is strongly
asymptotically optimal (Hutter, 2005, Thm. 5.38). Note that in contrast to our results,
this requires that that the agent uses a Bayes-mixture over a class of ergodic MDPs.
Moreover, Bayesian agents still perform well at learning and achieve on-policy value
convergence(Corollary4.20): theposteriorbeliefaboutthevalueofapolicy converges
to the true value of while following :V
(æ<t)V
(æ<t)!0ast!1-almost
surely. Since this holds for any policy, in particular it holds for the Bayes optimal policy

. This means that the Bayes agent learns to predict those parts of the environment
that it sees. But if it does not explore enough, then it will not learn other parts of the
environment that are potentially more rewarding.
Hutter (2005, Claim 5.12) claims:
We expect AIXI to be universally optimal.
Our work seriously challenges Hutter’s claim: no nontrivial and non-subjective optimal-
ity results for AIXI remain (see Table 5.3). Until new arguments for AIXI’s optimality
are put forward, we have to regard AIXI as a relativetheory of intelligence, dependent
on the choice of the prior.
5.6.2 Natural Universal Turing Machines
The choice of the UTM has been a big open question in algorithmic information theory
for a long time. The Kolmogorov complexity of a string depends on this choice. How-
ever, there are invariance theorems (Li and Vitányi, 2008, Thm. 2.1.1 & Thm. 3.1.1)
which state that changing the UTM changes Kolmogorov complexity only by a con-
stant. When using the Solomonoﬀ prior Mto predict any deterministic computable
binary sequence, the number of wrong predictions is bounded by the Kolmogorov com-
plexity of the sequence (Corollary 3.56). Due to the invariance theorem, changing the
UTM changes the number of errors only by a constant. In this sense, compression and
prediction work for any choice of UTM.
For AIXI, there can be no invariance theorem; in Section 5.2 we showed that a bad
choice for the UTM can have drastic consequences. Our negative results can guide
future search for a naturalUTM: the UTMs used to deﬁne the indiﬀerence prior (Theo-
rem 5.4), the dogmatic prior (Theorem 5.5), and the Gödel prior (Theorem 5.8) should
be considered unnatural. But what are other desirable properties of a UTM?
A remarkable but unsuccessful attempt to ﬁnd natural UTMs is due to Müller
(2010). It takes the probability that one universal machine simulates another according
98 Optimality
name deﬁned in KU(U0) KU0(U)
indiﬀerence prior Theorem 5.4 K(U) +K(m) +O(1) m
dogmatic prior Theorem 5.5 K(U) +K() +K(") +O(1)dlog2"e
Gödel prior Theorem 5.8 K(U) +K(PA) +O(1) 0
Table 5.2: Upper bounds to compiler sizes of the UTMs used in the proofs of Sec-
tion 5.2.KU(U0)is the number of extra bits to run the ‘bad’ UTM U0on the ‘good’
UTMU,KU0(U)is the number of extra bits to run UonU0.K(U)denotes the length
of the shortest program for UonU.
to the length of their respective compilers and searches for a stationary distribution.
Unfortunately, no stationary distribution exists.
Alternatively, we could demand that the UTM U0that we use for the universal prior
has a small compiler on the reference machine U(Hutter, 2005, p. 35). Moreover, we
coulddemandthereverse, thatthereferencemachine Uhasasmallcompileron U0. The
idea is that this should limit the amount of bias one can introduce by deﬁning a UTM
that has very small programs for very complicated and ‘unusual’ environments. Unfor-
tunately, this just pushes the choice of the UTM to the reference machine. Table 5.2
lists compiler sizes of the UTMs constructed in this thesis.
5.6.3 Asymptotic Optimality
A policy is asymptotically optimal if the agent learns to act optimally in any environ-
ment from the class M. We discussed two asymptotically optimal policies. BayesExp
is weakly asymptotically optimal if the horizon grows sublinearly (Theorem 5.24) and
Thompson sampling is asymptotically optimal in mean (Theorem 5.25). Both policies
commit to exploration for several steps. As stated in Example 5.19:
To achieve asymptotic optimality, the agent needs to explore inﬁnitely often
for an entire eﬀective horizon.
This is why weak asymptotic optimality is impossible if the horizon grows linearly (The-
orem 5.21): if the agent explores for an entire eﬀective horizon, it spoils a signiﬁcant
fraction of the average. Thompson sampling explores whenever it draws a bad sample.
BayesExp explores if the maximal expected information gain is above some threshold.
Both policies commit to exploration for the entire eﬀective horizon.
The exploration performed by Thompson sampling is qualitatively diﬀerent from
the exploration by BayesExp (Lattimore, 2013, Ch. 5). BayesExp performs phases of
exploration in which it maximizes the expected information gain. This explores the
environment class completely, even achieving oﬀ-policy prediction (Orseau et al., 2013,
Thm. 7). In contrast, Thompson sampling only explores on the optimal policies, and in
some environment classes this will not yield oﬀ-policy prediction. So in this sense the
§5.6Discussion 99
Optimality Issue/Comment
-optimal policy requires to know the true environment in advance
Pareto optimality always satisﬁed (Theorem 5.3)
Bayes optimality same as maximal intelligence
balanced Pareto optimality same as maximal intelligence (Proposition 5.12)
maximal intelligence highly dependent on the prior (Corollary 5.15 and
Corollary 5.16)
PAC strongvariantofasymptoticoptimalityinprobability
asymptotic optimality Thompson sampling (Theorem 5.25) and
BayesExp (Lattimore, 2013), but not AIXI (Orseau,
2013)
sublinear regret impossible in general environments, but possible with
recoverability (Theorem 5.33)
Table 5.3: Proposed notions of optimality (Hutter, 2002a, 2005; Legg and Hutter,
2007b) and their issues. Asymptotic optimality stands out to be the only nontrivial
objective optimality notion for general reinforcement learning.
explorationmechanismofThompsonsamplingismorereward-orientedthanmaximizing
information gain.
However, asymptotic optimality has to be taken with a grain of salt. It provides
no incentive to the agent to avoid traps in the environment. Once the agent gets
caught in a trap, all actions are equally bad and thus optimal: asymptotic optimality
has been achieved. Even worse, an asymptotically optimal agent has to explore all
the traps because they might contain hidden treasure. This brings us to the following
impossibility result for non-recoverable environment classes.
Either the agent gets caught in a trap or it is not asymptotically optimal.1
5.6.4 The Quest for Optimality
Theorem 5.3 shows that Pareto optimality is trivial in the class of all computable en-
vironments. Bayes optimality, Balanced Pareto optimality, and maximal Legg-Hutter
intelligence are equivalent (Proposition 5.12 and Proposition 5.10). Corollary 5.15 and
Corollary 5.16 show that this notion is highly subjective because it depends on the
choice of the prior. Moreover, according to Corollary 5.17, any computable policy is
nearly balanced Pareto optimal. For ﬁnite horizons, there are priors such that every
policy is balanced Pareto optimal (Theorem 5.4). Sublinear regret is impossible in
general environments (Example 5.30). However, if the environment is recoverable (Def-
inition 5.31), then Theorem 5.33 shows that asymptotic optimality in mean implies
sublinear regret. In summary, asymptotic optimality is the only nontrivial and objec-
tive notion of optimality for the general reinforcement learning problem (Problem 4.2):
1This formulation was suggested by Toby Ord.
100 Optimality
it is both satisﬁable (Theorem 5.24 and Theorem 5.25) and objective because it does
not depend on a prior probability measure over the environment class M. Table 5.3
summarized the notions of optimality discussed in this chapter.
Ouroptimalitynotionsare tail events : anyﬁnitenumberoftimestepsareirrelevant;
the agent can be arbitrarily lazy. Asymptotic optimality requires only convergence in
the limit. In recoverable environments we can always achieve sublinear regret after
any ﬁnite interaction. All policies with ﬁnite horizon are Bayes optimal according to
Theorem 5.4 and Corollary 5.6. Overall, there is a dichotomy between the asymptotic
nature of our optimality notions and the use of discounting to prioritize the present
over the future. Ideally, we would aim for ﬁnite guarantees instead, such as precise
regret bounds or PAC convergence rates, but without additional assumptions this is
impossible in this general setting. This leaves us with the main question of this chapter
unanswered (Hutter, 2009b, Sec. 5):
What is a good optimality criterion for general reinforcement learning?
Chapter 6
Computability
I simply keep a few spare halting oracles around. — Marcus Hutter
Given inﬁnite computation power, many traditional AI problems become trivial: play-
ing chess, go, or backgammon can be solved by exhaustive expansion of the game tree.
Yet other problems seem diﬃcult still; for example, predicting the stock market, driv-
ing a car, or babysitting your nephew. How can we solve these problems in theory?
Solomonoﬀ induction and AIXI are proposed answers to this question.
Both Solomonoﬀ induction and AIXI are known to be incomputable. But not all
incomputabilities are equal. The arithmetical hierarchy speciﬁes diﬀerent levels of com-
putability based on oracle machines : each level in the arithmetical hierarchy is com-
puted by a Turing machine which may query a halting oracle for the respective lower
level. Our agents are useless if they cannot be approximated in practice, i.e., by a
regular Turing machine. Therefore we posit that any ideal for a ‘perfect agent’ needs
to belimit computable (0
2). The class of limit computable functions is the class of
functions that admit an anytime algorithm .
In Section 6.2 we consider various diﬀerent ﬂavors of Solomonoﬀ induction: Solo-
monoﬀ’s prior M(Example 3.5) is only a semimeasure and not a measure: it assigns
positive probability that the observed string has only ﬁnite length. This can be cir-
cumvented by normalizing M. Solomonoﬀ’s normalization Mnorm(Deﬁnition 2.16)
preserves the ratio M(x1)=M(x0)and is limit computable. If instead we mix only over
programs that compute inﬁnite strings, we get a semimeasure M(3.6), which can be
normalized to Mnorm. Moreover, when predicting a sequence, we are primarily inter-
ested in the conditional probability M(xyjx)(respectively Mnorm(xyjy),M(xyjx),
orMnorm(xyjx)) that the currently observed string xis continued with y. We show
that bothMandMnormare limit computable, while MandMnormare not. Table 6.1
summarizes our computability results for Solomonoﬀ induction.
For MDPs, planning is already P-complete for ﬁnite and inﬁnite horizons (Papadim-
itriou and Tsitsiklis, 1987). In POMDPs, planning is undecidable (Madani et al., 1999,
2003). The existence of a policy whose expected value exceeds a given threshold is
PSPACE-complete (Mundhenk et al., 2000), even for purely epistemic POMDPs in
which actions do not change the hidden state (Sabbadin et al., 2007). In Section 6.3
we derive hardness results for planning in general semicomputable environments; this
environment class is even more general than POMDPs. We show that optimal policies
101
102 Computability
Qf(x;q)2XQjQ(x)>qg f(x;y;q )2XXQjQ(xyjx)>qg
M 0
1n0
1 0
2n(0
1[0
1)
Mnorm 0
2n(0
1[0
1) 0
2n(0
1[0
1)
M 0
2n0
2 0
3n(0
2[0
2)
Mnorm 0
3n(0
2[0
2) 0
3n(0
2[0
2)
Table 6.1: The computability results on M,Mnorm,M, andMnormproved in Sec-
tion 6.2. Lower bounds on the complexity of MandMnormare given only for speciﬁc
universal Turing machines.
Agent Optimal "-Optimal
AIMU 0
2 0
1
AINU 0
3,0
2-hard 0
2,0
1-hard
AIXI 0
3,0
1-hard 0
2,0
1-hard
Entropy-seeking 0
3 0
2
Information-seeking 0
3 0
2
BayesExp 0
3 0
2
Table 6.2: Computability results for diﬀerent agent models derived in Section 6.3,
Section 6.5, and Section 6.6. AIMU denotes the optimal policy in a computable envi-
ronment and AINU denotes the optimal policy in a semicomputable environment (see
Section 4.1). Hardness results for AIXI are with respect to a speciﬁc universal Turing
machine; hardness results for -optimal policies are with respect to a speciﬁc environ-
ment2MCCS
LSC. Results for entropy-seeking and information-seeking policies are only
for ﬁnite horizons.
are0
2-hard and"-optimal policies are undecidable.
Moreover, we show that by default, AIXI is not limit computable. When picking the
next action, two or more actions might have the same value (expected future rewards).
The choice between them is easy, but determining whether such a tie exists is diﬃcult.
This problem can be circumvented by settling for an "-optimal policy; we get a limit-
computable agent with inﬁnite horizon. However, these results rely on the recursive
deﬁnition of the value function. In contrast, Hutter (2005) deﬁnes the value function as
the limit of the iterative value function. In Section 6.4 we compare these two deﬁnitions
and show that the recursive deﬁnition correctly maximizes expected rewards and has
better computability properties.
In Section 6.5 we show that for ﬁnite horizons both the entropy-seeking and the
information-seeking agent are 0
3-computable and have limit-computable "-optimal
policies. BayesExp (Section 4.3.3) relies on optimal policies that are generally not
§6.1Background on Computability 103
limit computable. In Section 6.6 we give a weakly asymptotically optimal agent based
on BayesExp that is limit computable. Table 6.2 summarizes our results on the com-
putability of these agents.
In this chapter we illustrate the environments used in the proofs of our theorems
in the form of ﬂowcharts. They should be read as follows. Circles denote stochastic
nodes, rectangles denote environment nodes , and diamonds denote the agent’s choice
nodes. Transitions out of stochastic nodes are labeled with transition probabilities,
transitions out of environment nodes are labeled with percepts, and transitions out of
choice nodes are labeled with actions. The initial node is marked with a small incoming
arrow (see for example Figure 6.3). By Assumption 4.6b the worst possible outcome is
getting reward 0forever, thus we label such states as hell. Analogously, getting reward
1forever is the best possible outcome, thus we label such states as heaven.
6.1 Background on Computability
6.1.1 The Arithmetical Hierarchy
A setANis0
niﬀ there is a quantiﬁer-free formula such that
k2A() 9k18k2:::Qnkn(k;k1;:::;kn) (6.1)
whereQn=8ifnis even,Qn=9ifnis odd (Nies, 2009, Def. 1.4.10). (We can also
think ofas a computable relation.) A set ANis0
niﬀ its complement NnA
is0
n. The formula on the right side of (6.1) is a 0
n-formula and its negation is a
0
n-formula . It can be shown that we can add any bounded quantiﬁers and duplicate
quantiﬁers of the same type without changing the classiﬁcation of A. The setAis0
n
iﬀAis0
nandAis0
n. We get that 0
1as the class of recursively enumerable sets, 0
1
as the class of co-recursively enumerable sets and 0
1as the class of recursive sets.
The setANis0
n-hard ( 0
n-hard, 0
n-hard)iﬀ for any set B20
n(B20
n,
B20
n),Bis many-one reducible to A, i.e., there is a computable function fsuch
thatk2B$f(k)2A(Nies, 2009, Def. 1.2.1). We get 0
n0
n+10
n+1:::and
0
n0
n+10
n+1:::. This hierarchy of subsets of natural numbers is known as
thearithmetical hierarchy .
By Post’s Theorem (Nies, 2009, Thm. 1.4.13), a set is 0
nif and only if it is recur-
sively enumerable on an oracle machine with an oracle for a 0
n1-complete set. An
oracle for 0
1is called a halting oracle .
6.1.2 Computability of Real-valued Functions
We ﬁx some encoding of rational numbers into binary strings and an encoding of binary
strings into natural numbers. From now on, this encoding will be done implicitly
wherever necessary.
Deﬁnition 6.1 (0
n-,0
n-,0
n-computable) .A function f:X!Ris called 0
n-
computable ( 0
n-computable, 0
n-computable) iﬀ the setf(x;q)2XQjf(x)>qgis
0
n(0
n,0
n).
104 Computability
f(x;q)jf(x)qg f (x;q)jf(x)<qg
fis computable 0
1 0
1
fis lower semicomputable 0
1 0
1
fis upper semicomputable 0
1 0
1
fis limit computable 0
2 0
2
fis0
n-computable 0
n 0
n
fis0
n-computable 0
n 0
n
fis0
n-computable 0
n 0
n
Table 6.3: Connection between the computability of real-valued functions and the
arithmetical hierarchy.
A0
1-computable function is called computable , a0
1-computable function is called
lower semicomputable , and a 0
1-computable function is called upper semicomputable .
A0
2-computable function fis called limit computable , because there is a computable
functionsuch that
lim
k!1(x;k) =f(x):
The program that limit computes fcan be thought of as an anytime algorithm forf:
we can stop at any time kand get a preliminary answer. If the program ran long
enough (which we do not know), this preliminary answer will be close to the correct
one.
Limit-computable sets are the highest level in the arithmetical hierarchy that can be
approached by a regular Turing machine. Above limit-computable sets we necessarily
need some form of halting oracle. See Table 6.3 for the deﬁnition of lower/upper
semicomputable and limit-computable functions in terms of the arithmetical hierarchy.
Lemma 6.2 (Computability of Arithmetical Operations) .Letn > 0and letf;g:
X!Rbe two 0
n-computable functions. Then
(a)f(x;y)jf(x)>g(y)gis0
n,
(b)f(x;y)jf(x)g(y)gis0
n,
(c)f+g,fg, andfgare0
n-computable, and
(d)f=gis0
n-computable if g(x)6= 0for allx.
(e)logfis0
n-computable if f(x)>0for allx.
Proof.We only prove this for n > 1. Sincef;gare0
n-computable, they are limit
computable on a level n1oracle machine. Let be the function limit computing fon
the oracle machine, and let  be the function limit computing gon the oracle machine:
f(x) = lim
k!1(k;x)andg(y) = lim
k!1 (k;y):
By assumption, both and are0
n1-computable.
§6.2The Complexity of Solomonoﬀ Induction 105
(a) LetG:=f(x;y;q )jg(y)< qg, and letF:=f(x;y;q )jq < f (x)g, both of which
are in 0
nby assumption. Hence there are 0
n-formulas'Gand'Fsuch that
(x;y;q )2G()'G(x;y;q )
(x;y;q )2F()'F(x;y;q )
Nowf(x)> g(y)if and only if9q:(x;y;q )2G\F, which is equivalent to the
0
n-forumla
9q:'G(x;y;q )^'F(x;y;q ):
(b) Follows from (a).
(c) Addition, subtraction, and multiplication are continuous operations.
(d) Division is discontinuous only at g(x) = 0. We show this explicitly. By assumption,
for any">0there is ak0such that for all k>k 0
j(x;k)f(x)j<"andj (x;k)g(x)j<":
We assume without loss of generality that "<jg(x)j, sinceg(x)6= 0by assumption.
(x;k)
 (x;k)f(x)
g(x)
=(x;k)g(x)f(x) (x;k)
 (x;k)g(x)
j(x;k)g(x)f(x)g(x)j+jf(x)g(x)f(x) (x;k)j
j (x;k)g(x)j
<"jg(x)j+jf(x)j"
j (x;k)g(x)j
withj (x;k)g(x)j=j (x;k)jjg(x)j>(jg(x)j")jg(x)j,
<"jg(x)j+jf(x)j
(jg(x)j")jg(x)j"!0! 0;
thereforef(x)=g(x) = limk!1(x;k)= (x;k).
(e) Follows from the fact that the logarithm is computable.
6.2 The Complexity of Solomonoﬀ Induction
In this section, we derive the computability results for Solomonoﬀ’s prior as stated in
Table 6.1.
SinceMis lower semicomputable, Mnormis limit computable by Lemma 6.2 (c)
and (d). When using the Solomonoﬀ prior M(or one of its sisters Mnorm,M, or
Mnormdeﬁned in Deﬁnition 2.16 and Equation 3.6) for sequence prediction, we need
106 Computability
M(xyjx)>q() 8m9k:(xy;k)
(x;m)>q() 9k9m08mm0:(xy;k)
(x;m)>q
Figure 6.1: A0
2-formula and an equivalent 0
2-formula deﬁning conditional M. Here
(x;k)denotes a computable function that lower semicomputes M(x).
to compute the conditional probability M(xyjx) =M(xy)=M(x)for ﬁnite strings
x;y2X. BecauseM(x)>0for all ﬁnite strings x2X, this quotient is well-deﬁned.
Theorem 6.3 (Complexity of M,Mnorm,M, andMnorm).
(a)M(x)is lower semicomputable
(b)M(xyjx)is limit computable
(c)Mnorm(x)is limit computable
(d)Mnorm(xyjx)is limit computable
(e)M(x)is0
2-computable
(f)M(xyjx)is0
3-computable
(g)Mnorm(x)is0
3-computable
(h)Mnorm(xyjx)is0
3-computable
Proof.(a) By Li and Vitányi (2008, Thm. 4.5.2). Intuitively, we can run all programs
in parallel and get monotonely increasing lower bounds for M(x)by adding 2jpj
every time a program phas completed outputting x.
(b) From (a) and Lemma 6.2d since M(x)>0(see also Figure 6.1).
(c) By Lemma 6.2cd, and M(x)>0.
(d) By (iii), Lemma 6.2d since Mnorm(x)M(x)>0.
(e) Letbeacomputablefunctionthatlowersemicomputes M. SinceMisasemimea-
sure,M(xy)P
zM(xyz), henceP
y2XnM(xy)is nonincreasing in nand thus
M(x)>qiﬀ8n9kP
y2Xn(xy;k )>q.
(f) From (v) and Lemma 6.2d since M(x)>0.
(g) From (v) and Lemma 6.2d.
(h) From (vi) and Lemma 6.2d since Mnorm(x)M(x)>0.
§6.2The Complexity of Solomonoﬀ Induction 107
We proceed to show that these bounds are in fact the best possible ones. If M
were 0
1-computable, then so would be the conditional semimeasure M(j). Thus the
M-adversarial sequence z1:1deﬁned in Example 3.42 would be computable and hence
corresponds to a computable deterministic measure . However, we have M(z1:t)2t
by construction, so dominance M(x)w()(x)withw()>0yields a contradiction
witht!1:
2tM(z1:t)w()(z1:t) =w()>0
By the same argument, the normalized Solomonoﬀ prior Mnormcannot be 0
1-com-
putable. However, since it is a measure, 0
1- or 0
1-computability would entail 0
1-
computability.
ForMandMnormwe prove the following two lower bounds for speciﬁc universal
Turing machines.
Theorem 6.4 (Mis not Limit Computable) .There is a universal Turing machine U0
such that the setf(x;q)jMU0(x)>qgis not in 0
2.
Proof.Assume the contrary and let A20
2n0
2andbe a quantiﬁer-free ﬁrst-order
formula such that
n2A() 8k9i:(n;k;i ): (6.2)
For eachn2N, we deﬁne the program pnas follows.
1:procedure pn
2:output 1n+10
3:k 0
4:whiletruedo
5:i 0
6:whilenot(n;k;i )do
7: i i+ 1
8:k k+ 1
9: output 0
Each program pnalways outputs 1n+10. Furthermore, the program pnoutputs the
inﬁnite string 1n+101if and only if n2Aby (6.2). We deﬁne U0as follows using our
reference machine U.
•U0(1n+10): Runpn.
•U0(00p): RunU(p).
•U0(01p): RunU(p)and bitwise invert its output.
By construction, U0is a universal Turing machine. No pnoutputs a string starting with
0n+11, thereforeMU0(0n+11) =1
4
MU(0n+11) +MU(1n+10)
. Hence
MU0(1n+10) = 2n21A(n) +1
4MU(1n+10) +1
4MU(0n+11)
= 2n21A(n) +MU0(0n+11)
108 Computability
Ifn =2A, thenMU0(1n+10) =MU0(0n+11). Otherwise, we have jMU0(1n+10)
MU0(0n+11)j= 2n2.
Now we assume that MU0is limit computable, i.e., there is a computable function
:XN!Qsuch that limk!1(x;k) =MU0(x). We get that
n2A() lim
k!1(0n+11;k)(1n+10;k)2n2;
thusAis limit computable, a contradiction.
Corollary 6.5 (Mnormis not 0
2- or 0
2-computable) .There is a universal Turing
machineU0such thatf(x;q)jMnormU0(x)>qgis not in 0
2or0
2.
Proof.SinceMnorm =cM, there exists a k2Nsuch that 2k<c(even if we do not
know the value of k). We can show that the set f(x;q)jMnormU0(x)>qgis not in 0
2
analogously to the proof of Theorem 6.4, using
n2A() lim
k!1(0n+11;k)(1n+10;k)2kn2:
IfMnormwere 0
2-computable or 0
2-computable, this would imply that Mnormis0
2-
computable since Mnormis a measure, a contradiction.
SinceM() = 1, we haveM(xj) =M(x), so the conditional probability M(xyjx)
has at least the same complexity as M. Analogously for MnormandMnormsince they
are measures. For M, we have that M(xj) =Mnorm(x), so Corollary 6.5 applies. All
that remains to prove is that conditional Mis not lower semicomputable.
Theorem 6.6 (Conditional Mis not Lower Semicomputable) .The setf(x;xy;q )j
M(xyjx)>qgis not recursively enumerable.
We gave a diﬀerent, more complicated proof in Leike and Hutter (2015b). The
following, much simpler and more elegant proof is due to Sterkenburg (2016, Prop. 3).
Proof.Assume to the contrary that M(xyjx)is lower semicomputable. Let a6=b2X.
We construct an inﬁnite string xby deﬁning its initial segments =:x(0)@x(1)@
x(2)@:::@x. At every step n, we enumerate strings y2Xuntil one is found
satisfyingM(ajx(n)y)1=2; then setx(n+ 1) :=x(n)yb. This implies that for
inﬁnitely many tthere is an nsuch thatM(bjx<t) =M(bjx(n)y)1M(aj
x(n)y)1=2. Since we assumed M(j)to be lower semicomputable, the inﬁnite
stringxis computable, and hence M(xtjx<t)!1by Corollary 3.55. But this
contradicts M(bjx<t)1=2inﬁnitely often.
6.3 The Complexity of AINU, AIMU, and AIXI
6.3.1 Upper Bounds
Inthissection,wederiveupperboundsonthecomputabilityofAINU,AIMU,andAIXI.
Except for Corollary 6.14, all results in this section apply generally to any 2MCCS
LSC.
§6.3The Complexity of AINU, AIMU, and AIXI 109
Since the Bayesian mixture 2MCCS
LSC, they apply to AIXI even though they are stated
for AINU.
In order to position AINU in the arithmetical hierarchy, we need to encode policies
as sets of natural numbers. For the rest of this chapter, we assume that policies are
deterministic, thuscanberepresentedasrelationsover (AE )A. Theserelationsare
easily identiﬁed with sets of natural numbers by encoding the history into one natural
number. From now on this translation of policies into sets of natural numbers will be
done implicitly wherever necessary.
Lemma 6.7 (Policies are in 0
n).If a policyis0
nor0
n, thenis0
n.
Proof.Let'be a 0
n-formula ( 0
n-formula) deﬁning , i.e.,'(h;a)holds iﬀ(h) =a.
We deﬁne the formula '0,
'0(h;a) :=^
a02Anfag:'(h;a0):
The set of actions Ais ﬁnite, hence '0is a0
n-formula ( 0
n-formula). Moreover, '0is
equivalent to '.
To compute the optimal policy, we need to compute the optimal value function.
The following lemma gives an upper bound on the computability of the value function
for environments in MCCS
LSC.
Lemma 6.8 (Complexity of V
).For every2MCCS
LSC, and every lower semicom-
putable discount function
, the function V
is0
2-computable.
Proof.The explicit form of the value function (4.2) has numerator
lim
m!1maxX
æt:mmX
i=t
(i)ri(e1:ika1:i);
and denominator (e<tka<t)t. The numerator is nondecreasing in mbecause we
assumed rewards to be nonnegative (Assumption 4.6b). Hence both numerator and
denominator are lower semicomputable functions, so Lemma 6.2d implies that V
is
0
2-computable.
From the optimal value function V
we get the optimal policy
according to (4.4).
However, in cases where there is more than one optimal action, we have to break an
argmax tie. This happens iﬀ V
(h) =V
(h)for two potential actions 6=2A.
This equality test is more diﬃcult than determining which is larger in cases where they
are unequal. Thus we get the following upper bound.
Theorem 6.9 (Complexity of Optimal Policies) .For any environment , ifV
is
0
n-computable, then there is an optimal policy
for the environment that is 0
n+1.
110 Computability
Proof.To break potential ties, we pick an (arbitrary) total order onAthat speciﬁes
which actions should be preferred in case of a tie. We deﬁne
(h) =a:()^
a0:a0aV
(ha)>V
(ha0)
^^
a0:aa0V
(ha)V
(ha0):(6.3)
Thenis a-optimal policy according to (4.4). By assumption, V
is0
n-computable.
By Lemma 6.2ab V
(ha)>V
(ha0)is0
nandV
(ha)V
(ha0)is0
n. Therefore the
policydeﬁned in (6.3) is a conjunction of a 0
n-formula and a 0
n-formula and thus
0
n+1.
Corollary 6.10 (ComplexityofAINU) .AINU is 0
3for every environment 2MCCS
LSC.
Proof.From Lemma 6.8 and Theorem 6.9.
Usuallywedonotmindtakingslightlysuboptimalactions. Thereforeactuallytrying
to determine if two actions have the exact same value seems like a waste of resources.
In the following, we consider policies that attain a value that is always within some
">0of the optimal value.
Theorem 6.11 (Complexity of "-Optimal Policies) .For any environment , ifV
is
0
n-computable, then there is an "-optimal policy "
for the environment that is 0
n.
Proof.Let" > 0be given. Since the value function V
(h)is0
n-computable, the
setV":=f(ha;q)jjqV
(ha)j< "= 2gis in 0
naccording to Deﬁnition 6.1. Hence
we compute the values V
(ha0)until we get within "=2for everya02Aand then
choose the action with the highest value so far. Formally, let be an arbitrary total
order onAthat speciﬁes which actions should be preferred in case of a tie. Without
loss of generality, we assume "= 1=k, and deﬁne Qto be an"=2-grid on [0;1], i.e.,
Q:=f0;1=2k;2=2k;:::; 1g. We deﬁne
"
(h) =a:() 9 (qa0)a02A2QA:^
a02A(ha0;qa0)2V"
^^
a0:a0aqa>qa0^^
a0:aa0qaqa0
^the tuple (qa0)a02Ais minimal with
respect to the lex. ordering on QA:(6.4)
This makes the choice of aunique. Moreover, QAis ﬁnite sinceAis ﬁnite, and hence
(6.4) is a 0
n-formula.
Corollary 6.12 (Complexity of "-Optimal AINU) .For any environment 2MCCS
LSC,
there is an "-optimal policy for AINU that is 0
2.
Proof.From Lemma 6.8 and Theorem 6.11.
§6.3The Complexity of AINU, AIMU, and AIXI 111
Purgatory Heaven6=
(ht)
(ht)r= 0r= 1
Figure 6.2: The environment from the proof of Theorem 6.15. The agent gets
reward 0as long as it follows AIXI’s policy
that is assumed to be computable. Once
the agent deviates from
, it gets reward 1. We get a contradiction because AIXI
can learn this environment, so it will eventually decide to take an action that leads to
heaven.
Corollary 6.13 (Complexity of "-Optimal AIXI) .For any lower semicomputable prior
there is an "-optimal policy for AIXI that is 0
2.
Proof.FromCorollary6.12sinceforanylowersemicomputableprior, thecorresponding
Bayesian mixture is inMCCS
LSC.
If the environment 2MCCM
compis a measure, i.e., assigns zero probability to ﬁnite
strings, then we get computable "-optimal policies.
Corollary 6.14 (Complexity of AIMU) .If the environment 2MCCM
compis a measure
and the discount function
is computable, then AIMUis limit computable ( 0
2), and
"-optimal AIMUis computable ( 0
1).
Proof.Let">0be the desired accuracy. We can truncate the limit m!1in (4.2)
at the"=2-eﬀective horizon Ht("=2), since everything after Ht("=2)can contribute at
most"=2to the value function. Any lower semicomputable measure is computable (Li
and Vitányi, 2008, Lem. 4.5.1). Therefore V
as given in (4.2) is composed only of
computable functions, hence it is computable according to Lemma 6.2. The claim now
follows from Theorem 6.9 and Theorem 6.11.
6.3.2 Lower Bounds
We proceed to show that the bounds from the previous section are the best we can hope
for. In environment classes where ties have to be broken, AINU has to solve 0
2-hard
problems (Theorem 6.16). These lower bounds are stated for particular environments
2MCCS
LSC. Throughout this section, we assume that t>0for allt.
We also construct universal mixtures that yield bounds on "-optimal policies. There
is an"-optimal AIXIthat solves 0
1-hard problems (Theorem 6.17). For arbitrary uni-
versal mixtures, we prove the following weaker statement that only guarantees incom-
putability.
Theorem 6.15 (NoAIXIis computable) .AIXIis not computable for any universal
Turing machine U.
112 Computability
This theorem follows from the incomputability of Solomonoﬀ induction. By the
on-policy value convergence theorem (Corollary 4.20) AIXI succeeds to predict the
environment’s behavior for its own policy. If AIXI were computable, then there would
be computable environments more powerful than AIXI: they can simulate AIXI and
anticipate its prediction, which leads to a contradiction.
Proof.Assume there is a computable policy
that is optimal in the mixture . We
deﬁne a deterministic environment , theadversarial environment to
. The environ-
mentgives rewards 0as long as the agent follows the policy
, and rewards 1once
the agent deviates. Formally, we ignore observations by setting O:=f0g, and deﬁne
(r1:tka1:t) :=8
>>>><
>>>>:1if8kt:ak=
((ar)<k)andrk= 0;
1if8kt:rk=1ki
wherei:= minfjjaj6=
((ar)<j)g;and
0otherwise:
See Figure 6.2 for an illustration of this environment. The environment is computable
because the policy
was assumed to be computable. Suppose
acts in, then by
Theorem 4.19 AIXI learns to predict perfectly on policy :
V

(æ<t)V
(æ<t)!0ast!1
-almost surely ;
since both
andare deterministic. Because V
(h<t) = 0by deﬁnition of , we get
V
(æ<t)!0. Therefore we ﬁnd a tlarge enough such that V
(æ<t)<w()whereæ<t
is the interaction history of
in. A policywith(æ<t)6=
(æ<t), gets a reward
of1in environment for all time steps after t, henceV
(æ<t) = 1. With linearity of
V
(æ<t)in(Lemma 4.14),
V
(æ<t)w()(e1:tka1:t)
(e1:tka1:t)V
(æ<t)w();
since(e1:tka1:t) = 1(is deterministic), V
(æ<t) = 1, and(e1:tka1:t)1. Now
we get a contradiction:
w()>V
(æ<t) = sup
0V0
(æ<t)V
(æ<t)w()
For the remainder of this section, we ﬁx the action space to be A:=f;gwith
actionfavored in ties. The percept space is ﬁxed to a tuple of binary observations
and rewards,E:=Of 0;1gwithO:=f0;1g.
Theorem 6.16 (AINU is 0
2-hard).There is an environment 2MCCS
LSCsuch that
AINU is 0
2-hard.
Proof.LetAbe a any 0
2-set, and let be a quantiﬁer-free formula such that
n2A() 8i9k(n;i;k ): (6.5)
§6.3The Complexity of AINU, AIMU, and AIXI 113
n++
Heaven1=2o= 1
r= 0
1=2o= 0
r= 0

19k(n;i;k)
o= 0
r= 1
Figure 6.3: The environment ifrom the proof of Theorem 6.16. The mixture
over class of environments M:=f0;1;:::gMCCS
LSCforces AINU to solve 0
2-hard
problems: Action is preferred (because of a tie) iﬀ it leads to heaven, which is the
case iﬀ9k(n;i;k ).
We deﬁne a class of environments M:=f1;2;:::gwhere each iis deﬁned as follows.
i((or)1:mka1:m) :=8
>>>>>>>>><
>>>>>>>>>:2mifo1:m= 1mand8tm:rt= 0;
2n1if9n:1n0vo1:mv1n01andan+2=
andrt=1t>n+1and9k(n;i;k );
2n1if9n:1n0vo1:mv1n01andan+2=
andrt=1t>n+1;and
0otherwise:
See Figure 6.3 for an illustration of these environments. Every iis a chronological
conditional semimeasure by deﬁnition and every iis lower semicomputable since is
quantiﬁer-free, so MMCCS
LSC.
We deﬁne our environment as a mixture over M,
:=X
i2N2i1i;
the choice of the weights on the environments iis arbitrary but positive. Let
be an
optimal policy for the environment and recall that the action is preferred in ties.
We claim that for the -optimal policy
,
n2A()
(1n0) =: (6.6)
This enables us to decide whether n2Agiven the policy
, hence proving (6.6)
concludes this proof.
Letn;i2Nbegiven, andsupposeweareinenvironment iandobserve 1n0. Taking
actionnext yields reward 1forever; taking action next yields a reward of 1if there
114 Computability
Semi-Heaven
Heaven
o= 1n0

19k(n;i;k)o= 0
r= 1=2
o= 0
r= 1
Figure 6.4: The environment from the proof of Theorem 6.17, which forces AIXI
to solve 0
1-hard problems. It functions just like until the observation history is 1n0.
Then, action is preferred iﬀ heaven is accessible, i.e., iﬀ 9k(n;i;k ).
is aksuch that(n;i;k )holds. If this is the case, then
V
i(1n0) = n+2=V
i(1n0);
and otherwise
V
i(1n0) = 0<n+2=V
i(1n0)
(omitting the ﬁrst n+ 1actions and rewards in the argument of the value function).
We can now show (6.6): By (6.5), n2Aif and only if for all ithere is aksuch
that(n;i;k ), which happens if and only if V
i(1n0) = n+2for alli2N, which
is equivalent to V
(1n0) = n+2, which in turn is equivalent to
(1n0) =since
V
(1n0) = n+2and action is favored in ties.
Theorem 6.17 (Some"-optimal AIXIare0
1-hard).There is a universal Turing ma-
chineU0and an">0such that any "-optimal policy for AIXIis0
1-hard.
Proof.LetAbe a 0
1-set andbe a quantiﬁer-free formula such that n+ 12Aiﬀ
9k(n;k). We deﬁne the environment
((or)1:tka1:t) :=8
>>>>>>>>>>>><
>>>>>>>>>>>>:((or)1:nka1:n)if9n:o 1:n= 1n10andan=
and8t0>n:ot0= 0^rt0=1
2;
((or)1:nka1:n)if9n:o 1:n= 1n10andan=
and8t0>n:ot= 0^rt= 1
and9k(n1;k);
((or)1:tka1:t)if@n:o 1:n= 1n10;and
0 otherwise:
See Figure 6.4 for an illustration. The environment mimics the universal environment
§6.4Iterative Value Function 115
until the observation history is 1n10. Taking the action next gives rewards 1=2
forever. Taking the action next gives rewards 1forever ifn2A, otherwise the
environment ends at some future time step. Therefore we want to take action if
and only if n2A. We have that 2MCCS
LSCsince2MCCS
LSCandis quantiﬁer-free.
We deﬁne0:=1
2+1
8. By Lemma 4.24 0is a universal lower semicomputable
semimeasure. Let n2Abe given and let h2(AE )nbe any history with observations
o1:n= 1n10. Since(1n10ja1:n) =(1n10ja1:n)bydeﬁnition, theposteriorweights
ofandin0are equal to the prior weights, analogously to the proof of Theorem 5.5.
In the following, we use the linearity of V
0
in(Lemma 4.14), and the fact that values
are bounded between 0and1(Assumption 4.6b). If there is a ksuch that(n1;k)
holds,
V
0(h)V
0(h) =1
2V
0
(h)1
2V
0
(h) +1
8V
0
(h)1
8V
0
(h)
1
21
4+ 01
8=1
8;
and similarly if there is no ksuch that(n1;k)holds, then
V
0(h)V
0(h) =1
2V
0
(h)1
2V
0
(h) +1
8V
0
(h)1
8V
0
(h)
1
40 + 01
8=1
8:
In both casesjV
0(h)V
0(h)j>1=9. Hence we pick ":= 1=9and get for every
"-optimal policy "
0that"
0(h) =if and only if n2A.
Note the diﬀerences between Theorem 6.15 and Theorem 6.17: the former talks
about optimal policies and shows that they are not computable, but is agnostic towards
the underlying universal Turing machine. The latter talks about "-optimal policies and
gives a stronger hardness result, at the cost of depending on one particular universal
Turing machine.
6.4 Iterative Value Function
Historically,AIXI’svaluefunctionhasbeendeﬁnedslightlydiﬀerentlytoDeﬁnition4.10,
using a limit extension of an iterative deﬁnition of the value function. This deﬁnition
is the more straightforward to come up with in AI: it is the natural adaptation of (op-
timal) minimax search in zero-sum games to the (optimal) expectimax algorithm for
stochastic environments. In this section we discuss the problems with this deﬁnition.
To avoid confusion with the recursive value function V
, we denote the iterative
value function with W
.1
Deﬁnition 6.18 (Iterative Value Function; Hutter, 2005, Def. 5.30) .Theiterative
1In Leike and Hutter (2015a) the use of the symbols VandWis reversed.
116 Computability
Agent Optimal "-Optimal
Iterative AINU 0
4,0
3-hard 0
3,0
2-hard
Iterative AIXI 0
4,0
2-hard 0
3,0
2-hard
Iterative AIMU 0
2 0
1
Table 6.4: Computability results for diﬀerent agent models that use the iterative value
function derived in Section 6.4. Hardness results for AINU are with respect to a speciﬁc
environment 2MCCS
LSC.
valueof a policy in an environment given history æ<tis
W
(æ<t) :=1
tlim
m!1X
et:m(e1:mje<tka1:m)mX
k=t
(k)rk
ift>0andW
(æ<t) := 0ift= 0whereai:=(e<i)for allit. Theoptimal
iterative value is deﬁned as W
(h) := supW
(h).
Analogously to (4.2), we can write W
using the max-sum-operator:
W
(æ<t) =1
tlim
m!1maxX
æt:m(e1:mje<tka1:m)mX
k=t
(k)rk (6.7)
We useiterative AINU for the-optimal policy according to the iterative value
function, and iterative AIXI for the-optimal policy according to the iterative value
function. Note that iterative AIMU coincides with AIMU since is a measure by
convention.
Generally, our environment 2MCCS
LSCis only a semimeasure and not a measure,
i.e., there is a history æ<tatsuch that
1>X
et2E(etje<tka1:t):
In such cases, with positive probability the environment does not produce a new
perceptet. If this occurs, we shall use the informal interpretation that the environment
ended, but our formal argument does not rely on this interpretation.
The following proposition shows that for a semimeasure 2MCCS
LSCthat is not a
measure, iterative AINU does not maximize -expected rewards. Recall that
(1)states
the discount of the ﬁrst reward. In the following, we assume without loss of generality
that
(1)>0, i.e., we are not indiﬀerent about the reward received in time step 1.
Proposition 6.19 (Iterative AINU is not a -Expected Reward Maximizer) .For any
" >0there is an environment 2MCCS
LSCthat is not a measure and a policy that
receives a total of
(1)rewards in, but iterative AINU receives only "
(1)rewards in
.
§6.4Iterative Value Function 117
Hell
 r="r= 0
r= 1
Figure 6.5: The environment from the proof of Proposition 6.19. Action yields
reward 1, but subsequently the environment ends. Action yields reward "and the
environment continues forever. Iterative AINU will prefer the suboptimal action ,
because it conditions on surviving forever.
Informally, the environment is deﬁned as follows. In the ﬁrst time step, the agent
chooses between the two actions and. Taking action gives a reward of 1, and
subsequently the environment ends. Action gives a reward of ", but the environment
continues forever. There are no other rewards in this environment. See Figure 6.5.
From the perspective of -expected reward maximization, it is better to take action ,
however iterative AINU takes action .
Proof of Proposition 6.19. Let" >0. We ignore observations and set E:=f0;";1g,
A:=f;g. The environment is formally deﬁned by
(r1:tka1:t) :=8
>><
>>:1ifa1=andr1= 1andt= 1
1ifa1=andr1="andrk= 081<kt
0otherwise:
Taking action ﬁrst, we have (r1:tka2:t) = 0fort>1(the environment ends in
time step 2given history ). Hence we conclude
V
() =1
tlim
m!1X
r1:m(r1:mka2:m)mX
k=1
(k)rk= 0:
Taking action ﬁrst we get
V
() =1
tlim
m!1X
r1:m(r1:mka2:m)mX
k=1
(k)rk=
(1)
1":
Since
(1)>0and" > 0, we haveV
()> V
(), and thus iterative AINU will
use a policy that plays action ﬁrst, receiving a total discounted reward of "
(1). In
contrast, any policy that takes action ﬁrst receives a larger total discounted reward
of
(1).
Whether it is reasonable to assume that our environment has a nonzero probability
118 Computability
of ending is a philosophical debate we do not want to engage in here; see Martin et al.
(2016) for a discussion. Instead, we have a diﬀerent motivation to use the recursive over
the iterative value function: the latter has worse computability properties. Concretely,
we show that "-optimal iterative AIXI has to solve 0
2-hard problems and that there
is an environment 2MCCS
LSCsuch that iterative AINU has to solve 0
3-hard problems.
In contrast, using the recursive value function, "-optimal AIXI is 0
2according to
Corollary 6.12 and AINU is 0
3according to Corollary 6.10.
The central diﬀerence between V
andW
is that for V
all obtained rewards
matter, but for W
only the rewards in timelines that continue indeﬁnitely. In this
sense the value function W
conditions on surviving forever. If the environment is a
measure, then the history is inﬁnite with probability one, and so V
andW
coincide.
Hence this distinction is not relevant for AIMU, only for AINU and AIXI.
Lemma 6.20 (Complexity of W
).For every2MCCS
LSC, the function W
is0
2-
computable.
Proof.Multiplying (6.7) with t(e<tka<t)yieldsW
(æ<t)>qif and only if
lim
m!1maxX
æt:m(e1:mka1:m)mX
k=t
(k)rk>qt(e<tka<t): (6.8)
The inequality’s right side is lower semicomputable, hence there is a computable func-
tion such that (`)%qt(e<tka<t) =:q0as`!1. (In contrast to the recursive
value function, this quantity is not increasing in m.) For a ﬁxed m, the left side is also
lower semicomputable, therefore there is a computable function such that
(m;k)%maxX
æt:m(e1:mka1:m)mX
k=t
(k)rk=:f(m)ask!1.
We already know that the limit of f(m)form!1exists (uniquely), hence we can
write (6.8) as
lim
m!1f(m)>q0
() 8m09mm0:f(m)>q0
() 8m09mm09k:(m;k)>q0
() 8`8m09mm09k:(m;k)> (`);
which is a 0
2-formula.
Note that in the ﬁnite horizon case where mis ﬁxed, the value function W
is0
2-
computable by Lemma 6.2d, since W
(æ<t) =f(m)=q0. In this case, we get the same
computability results for iterative AINU as we did in Section 6.3.1.
Corollary 6.21 (Complexity of Iterative AINU) .For any environment 2MCCS
LSC,
iterative AINU is 0
4and there is an "-optimal iterative AINU that is 0
3.
§6.4Iterative Value Function 119
n++ Hell
Conditional Heaven1=2o= 1
r= 0
1=2o= 0
r= 0

18t0t9k(n;i;t0;k)o= 0
r= 0
o= 0
r= 1
Figure 6.6: The environment ifrom the proof of Theorem 6.22. The mixture
over class of environments M:=f0;1;:::g  MCCS
LSCforces iterative AINU to
solve 0
3-hard problems. ‘Conditional Heaven’ is a node that yields reward 1until
:9k (n;i;t;k ), at which point the environment ends. Hence action is preferred in
environment iiﬀ conditional heaven lasts forever (because otherwise (:::) = 0and
henceV
(:::) = 0) which is the case iﬀ 8t9k(n;i;t;k ).
Proof.From Theorem 6.9, Theorem 6.11, and Lemma 6.20.
We proceed to show corresponding lower bounds as in Section 6.3.2. For the rest of
this section we assume t>0for allt.
Theorem 6.22 (Iterative AINU is 0
3-hard).There is an environment 2MCCS
LSCsuch
that iterative AINU is 0
3-hard.
Proof.The proof is analogous to the proof of Theorem 6.16. Let Abe any 0
3set, then
there is a quantiﬁer-free formula such that
n2A() 9i8t9k(n;i;t;k ):
We deﬁne the environments isimilar to the proof of Theorem 6.16, except for two
changes:
•We replace9k(n;i;k )with8t0t9k(n;i;t0;k).
•We switch actions and: action‘checks’ the formula and action gives a
sure reward of 0.
120 Computability
Formally,
i((or)1:tka1:t) :=8
>>>>>>>>>>>><
>>>>>>>>>>>>:2tifo1:t= 1tand8t0t:rt0= 0;
2n1if9n:1n0vo1:tv1n01andan+2=
and8t0t:rt0= 0;
2n1if9n:1n0vo1:tv1n01andan+2=
and8t0t:rt0=1t0>n+1
and8t0t9k(n;i;t0;k);and
0otherwise:
See Figure 6.6 for an illustration of the environment i. Everyiis a chronological
conditional semimeasure by deﬁnition, so M:=f0;1;:::gMCCS
LSC. Furthermore,
everyiis lower semicomputable since is quantiﬁer-free.
We deﬁne our environment as a mixture over M,
:=X
i2N2i1i;
the choice of the weights on the environments iis arbitrary but positive. We get for
the-optimal policy
analogously to the proof of Theorem 6.16

(1n0) =() 9i8t0t9k(n;i;t0;k)()n2A;
since action is preferred in ties.
Analogously to Theorem 6.15, we can show that iterative AIXI is not computable.
We also get the following lower bound.
Theorem 6.23 (Some"-optimal iterative AIXIare0
2-hard).There is a universal
mixture0and an">0such that any policy that is "-optimal according to the iterative
value for environment 0is0
2-hard.
Proof.LetAbe a 0
2-set anda quantiﬁer-free formula such that
n2A() 8t9k(n;t;k ):
We proceed analogous to the proof of Theorem 6.17 except that we choose 8t0
t9k(n;t;k )as a condition for reward 1after playing action .
§6.4Iterative Value Function 121
Semi-Heaven
Conditional Heaven
o= 1n0

18t0<t9k(n;i;t0;k)o= 0
r= 1=2
o= 0
r= 1
Figure6.7: Theenvironment fromtheproofofTheorem6.23, whichforces "-optimal
iterative AIXI to solve 0
2-hard problems. It functions just like until the observation
history is 1n0. Then, action is preferred iﬀ conditional heaven never ends, i.e., iﬀ
8t9k(n;t;k ).
Deﬁne the environment
((or)1:tka1:t) :=8
>>>>>>>>>>>>>>>>><
>>>>>>>>>>>>>>>>>:((or)1:n+1ka1:n+1)if9n:1n0vo1:tv1n01
andan+1=
and8n+ 1<kt:rk= 1=2;
((or)1:n+1ka1:n+1)if9n:1n0vo1:tv1n01
andan+1=
and8n+ 1<kt:rk= 1
and8t0t9k(n;t;k );
((or)1:tka1:t) if@n:1n0vo1:tv1n01;and
0 otherwise:
See Figure 6.7 for an illustration of the environment . The environment mimics the
universal environment until the observation history is 1n0. The next action always
gives rewards 1=2forever, while action gives rewards 1forever iﬀn2A. We have that
isalowersemicomputablesemimeasuresince isalowersemicomputablesemimeasure
andis quantiﬁer-free. We deﬁne 0=1
2+1
8. By Lemma 4.24, 0is a universal lower
semicomputable semimeasure. Let n2Abe given and let h2(AO )x+1be any
history with observations o1:n+1= 1n0. In the following, we use the linearity of W
in
(analogously to Lemma 4.14). If 8t9k(n;t;k ), then
W
0(h)W
0(h) =1
2W
(h)1
2W
(h) +1
8W
(h)1
8W
(h)
1
21
4+ 01
8=1
8;
122 Computability
and similarly if:8t9k(n;t;k ), then
W
0(h)W
0(h) =1
2W
(h)1
2W
(h) +1
8W
(h)1
8W
(h)
1
40 + 01
8=1
8:
In both casesjW
0(h)W
0(h)j>1=9, hence with ":= 1=9we have for an "-optimal
policy"
0that"
0(h) =if and only if n2A.
6.5 The Complexity of Knowledge-Seeking
Recall the deﬁnition of the optimal entropy-seeking value V;m
Entand the optimal infor-
mation-seeking value V;m
IGfrom Section 4.3.2. Using the results from Section 6.3 we
can show that "-optimal knowledge-seeking agents are limit computable, and optimal
knowledge-seeking agents are 0
3.
Corollary 6.24 (Computability of Knowledge-Seeking Values) .For ﬁxedm, the value
functionsV;m
EntandV;m
IGare limit computable.
Proof.This follows from Lemma 6.2 (c-e) since ,, andware lower semicomputable.
Corollary 6.25 (Computability of Knowledge-Seeking Policies) .For entropy-seeking
and information-seeking agents there are limit-computable "-optimal policies and 0
3-
computable optimal policies.
Proof.Follows from Corollary 6.24, Theorem 6.9, and Theorem 6.11.
Note that if we used an inﬁnite horizon with discounting in Deﬁnition 4.25 or Deﬁni-
tion 4.26, then we cannot retain this computability result without further assumptions:
we would need that the value functions increase monotonically as m!1, as they
do for the recursive value function from Deﬁnition 4.10. However, entropy is not a
monotone function and may decrease if there are events whose probability converges
to something >1=2. For the entropy-seeking value function this happens for histories
drawn from a deterministic environment since!, so the conditionals converge
to1. Similarly, for the information-seeking value function, the posterior belief in one
(deterministic) environment might become larger than 1=2(depending on the prior and
the environment class). Therefore we generally only get that discounted versions of V
Ent
andV
IGare0
3analogously to Lemma 6.20. Hence optimal discounted entropy-seeking
and optimal discounted information-seeking policies are in 0
4by Theorem 6.9 and their
corresponding "-optimal siblings are 0
3by Theorem 6.11.
6.6 A Limit Computable Weakly Asymptotically Optimal
Agent
According to Theorem 6.16, optimal reward-seeking policies are generally 0
2-hard, and
foroptimalknowledge-seekingpoliciesCorollary6.25showsthattheyare 0
3. Therefore
§6.7Discussion 123
we get that BayesExp is 0
3:
Corollary 6.26 (BayesExp is 0
3).For any universal mixture , BayesExp is 0
3.
Proof.From Corollary 6.10, Corollary 6.24, and Corollary 6.25.
However, we do not know BayesExp to be limit computable, and we expect it
not to be. However, we can approximate it using "-optimal policies preserving weak
asymptotic optimality.
Theorem6.27 (ALimit-ComputableWeaklyAsymptoticallyOptimalAgent) .If there
is a nonincreasing computable sequence of positive reals ("t)t2Nsuch that"t!0and
Ht("t)=(t"t)!0ast! 1, then there is a limit-computable policy that is weakly
asymptotically optimal in the class of all computable stochastic environments.
Proof.By Corollary 6.10, there is a limit-computable 2t-optimal reward-seeking policy
t
for the universal mixture . By Corollary 6.25 there are limit-computable t=2-
optimal information-seeking policies t
IGwith horizon t+Ht("t). We deﬁne a policy
analogously to Algorithm 1 with t
IGandt
instead of the optimal policies. From
Corollary 6.24 we get that V
IGis limit computable, so the policy is limit computable.
Furthermore, t
is2t-optimal and 2t!0, soVt

(æ<t)!V
(æ<t)ast!1.
Now we can proceed analogously to the proof of Lattimore (2013, Thm. 5.6), which
consists of three parts. First, it is shown that the value of the -optimal reward-
seeking policy
converges to the optimal value for exploitation time steps (line 6 in
Algorithm 1) in the sense that V
!V
. This carries over to the 2t-optimal policy
t
, since the key property is that on exploitation steps, V
IG<"t; i.e.,only exploits if
potential knowledge-seeking value is low. In short, we get for exploitation steps
Vt

(æ<t)!V

(æ<t)!V
(æ<t)!V
(æ<t)ast!1:
Second, it is shown that the density of exploration steps vanishes. This result carries
over since the condition V
IG(æ<t)>"tthat determines exploration steps is exactly the
same as for BayesExp and t
IGis"t=2-optimal.
Third, the results of part one and two are used to conclude that is weakly asymp-
totically optimal. This part carries over to our proof.
6.7 Discussion
When using Solomonoﬀ’s prior for induction, we need to evaluate conditional probabil-
ities. We showed that conditional MandMnormare limit computable (Theorem 6.3),
and thatMandMnormare not limit computable (Theorem 6.4 and Corollary 6.5).
Table 6.1 on page 102 summarizes our computability results on various versions of
Solomonoﬀ’s prior. Theses results implies that we can approximate MorMnormfor
prediction, but not the measure mixture MorMnorm.
In some cases, normalized priors have advantages. As illustrated in Example 4.27,
unnormalized priors can make the entropy-seeking agent mistake the entropy gained
124 Computability
from the probability assigned to ﬁnite strings for knowledge. From MnormMwe get
thatMnormpredicts just as well as M, and by Theorem 6.3 we can use Mnormwithout
losing limit computability.
Table 6.2 on page 102 summarizes our computability results for the agents AINU,
AIXI, and AINU. AINU is 0
3and restricting to "-optimal policies decreases the level
by one (Corollary 6.10 and Corollary 6.12). For environments from MCCM
comp, AIMU is
limit-computable and "-optimal AIMU is computable (Corollary 6.14). In Section 6.3.2
we proved that these computability bounds on AINU are generally unimprovable (The-
orem 6.16 and Theorem 6.17). Additionally, we proved weaker lower bounds for AIXI
independent of the universal Turing machine (Theorem 6.15) and for "-optimal AIXI
for speciﬁc choices of the universal Turing machine (Theorem 6.17).
When the environment has nonzero probability of not producing a new percept,
the iterative deﬁnition of AINU (Deﬁnition 6.18) originally given by Hutter (2005,
Def. 5.30) fails to maximize -expected rewards (Proposition 6.19). Moreover, the
policies are one level higher in the arithmetical hierarchy (see Table 6.4 on page 116).
We proved upper (Corollary 6.21) and lower bounds (Theorem 6.22 and Theorem 6.23).
The diﬀerence between the recursive value function Vand the iterative value function
Wis readily exposed in the diﬀerence between the universal prior Mand the measure
mixtureM: Just like Wconditions on surviving forever, so does Meliminate the
weight of programs that do not produce inﬁnite strings. Both MandWare not limit
computable for this reason.
We considered "-optimality to avoid having to determine argmax ties. This "does
not have to be constant over time, we may let "!0ast!1at any computable
rate. With this we retain the computability results of "-optimal policies and get that
the value of the "(t)-optimal policy "(t)
converges rapidly to the -optimal value:
V
(æ<t)V"(t)
(æ<t)!0ast!1:
In Section 4.1 we deﬁned the set MCCS
LSCas the set of all lower semicomputable
chronological contextual semimeasure over percepts with actions povided as side-infor-
mation. When determining the probability of the next percept etin an environment
, we have to compute (e1:tje<tka1:t). Alternatively, we could have deﬁned the
environment as a lower semicomputable mapping from histories æ<tatto probabilities
over the next percept et(this is done in Chapter 7). For the proof of Lemma 6.8
and Lemma 6.20 we only need that (e1:tka1:t)is lower semicomputable computable.
While this new deﬁnition makes no diﬀerence for the computability of AINU, it matters
for AIXI because in the mixture over all of these environments is no longer lower
semicomputable.
Any method that tries to tackle the reinforcement learning problem has to balance
between exploration and exploitation. AIXI strikes this balance in the Bayesian way.
However, as we showed in Section 5.2, this may not lead to enough exploration. To
counteract this, we can add an explorative component to the agent, akin to knowledge-
seeking agents. In Section 6.5 we show that "-optimal knowledge-seeking agents are
limit computable if we use the recursive deﬁnition of the value function.
§6.7Discussion 125
We set out with the goal of ﬁnding a perfect reinforcement learning agent that is
limit computable. The Bayesian agent AIXI could be considered one suitable candidate,
despite its optimality problems discussed in Chapter 5. Another suitable candidate are
weakly asymptotically optimal agents, which in contrast to AIXI are optimal in an
objective sense (see Section 5.6). We discussed BayesExp, which relies on a Solomonoﬀ
prior to learn its environment and on a information-seeking component for extra explo-
ration. Our results culminated in a limit-computable weakly asymptotically optimal
agent based on BayesExp (Theorem 6.27). In this sense our goal has been achieved.
126 Computability
Chapter 7
The Grain of Truth Problem1
AIs become friendly by playing lots of Newcomblike problems. — Eliezer Yudkowsky
Consider the general setup of multiple reinforcement learning agents interacting sequen-
tially in a known environment with the goal to maximize discounted reward.2Each
agent knows how the environment behaves, but does not know the other agents’ be-
havior. The natural (Bayesian) approach would be to deﬁne a class of possible policies
that the other agents could adopt and take a prior over this class. During the interac-
tion, this prior gets updated to the posterior as our agent learns the others’ behavior.
Our agent then acts optimally with respect to this posterior belief. Kalai and Lehrer
(1993) show that in inﬁnitely repeated games Bayesian agents converge to an "-Nash
equilibrium as long as each agent assigns positive prior probability to the other agents’
policies (a grain of truth ).
As an example, consider an inﬁnitely repeated prisoners dilemma between two
agents. In every time step the payoﬀ matrix is as follows, where C means cooperate
and D means defect.
C D
C3/4, 3/4 0, 1
D1, 0 1/4, 1/4
Deﬁne the set of policies  :=f1;0;1;:::gwhere policy tcooperates until time
steptor the opponent defects (whatever happens ﬁrst) and defects thereafter. The
Bayes optimal behavior is to cooperate until the posterior belief that the other agent
defects in the time step after the next is greater than some constant (depending on the
discount function) and then defect afterwards. Therefore Bayes optimal behavior leads
to a policy from the set (regardless of the prior). If both agents are Bayes optimal
with respect to some prior, they both have a grain of truth and therefore they converge
to a Nash equilibrium: either they both cooperate forever or after some ﬁnite time they
both defect forever. Alternating strategies like TitForTat (cooperate ﬁrst, then play
1The idea for reﬂective oracles was developed by Jessica Taylor and Benya Fallenstein based on
ideas by Paul Christiano (Christiano et al., 2013). Reﬂective oracles were ﬁrst described in Fallenstein
et al. (2015a). The proof of Theorem 7.7 was sketched by Benya Fallenstein and developed by me.
Except for minor editing, everything else in this chapter is my own work.
2We mostly use the terminology of reinforcement learning. For readers from game theory we provide
a dictionary in Table 7.1.
127
128 The Grain of Truth Problem
Reinforcement learning Game theory
stochastic policy mixed strategy
deterministic policy pure strategy
agent player
multi-agent environment inﬁnite extensive-form game
reward payoﬀ/utility
(ﬁnite) history history
inﬁnite history path of play
Table 7.1: Terminology dictionary between reinforcement learning and game theory.
the opponent’s last action) are not part of the policy class , and adding them to the
class breaks the grain of truth property: the Bayes optimal behavior is no longer in the
class. This is rather typical; a Bayesian agent usually needs to be more powerful than
its environment (see Section 6.3). We are facing the following problem.
Problem 7.1 (Grain of Truth Problem; Hutter, 2009b, Q. 5j) .Find a large class of
policies containing Bayesian agents with positive prior over .
Until now, classes that admit a grain of truth were known only for small toy exam-
ples such as the iterated prisoner’s dilemma above (Shoham and Leyton-Brown, 2009,
Ch. 7.3). Foster and Young (2001) and Nachbar (1997, 2005) prove several impos-
sibility results on the grain of truth problem that identify properties that cannot be
simultaneously satisﬁed for classes that allow a grain of truth (see Section 7.6 for a
discussion).
In this chapter we present a formal solution to the grain of truth problem (Sec-
tion 7.2). We assume that our multi-agent environment is computable, but it does not
need to be stationary/Markov, ergodic, or ﬁnite-state. Our class of policies is large
enough to contain all computable (stochastic) policies, as well as all relevant Bayes
optimal policies. At the same time, our class is small enough to be limit computable.
This is important because it allows our result to be computationally approximated.
InSection7.3weconsiderthesettingwherethemulti-agentenvironmentisunknown
to the agents and has to be learned in addition to the other agents’ behavior. A Bayes
optimal agent may not learn to act optimally in unknown multi-agent environments
even though it has a grain of truth . This eﬀect occurs in non-recoverable environments
wheretakingonewrongactioncanmeanapermanentlossoffuturevalue. Inthiscase, a
Bayesoptimalagentavoidstakingthesedangerousactionsandthereforewillnotexplore
enough to wash out the prior’s bias (using the dogmatic prior from Section 5.2.2).
Therefore, Bayesian agents are not asymptotically optimal , i.e., they do not always
learn to act optimally (Theorem 5.22).
However, asymptotic optimality is achieved by Thompson sampling because the
inherent randomness of Thompson sampling leads to enough exploration to learn the
entire environment class (see Section 5.4.3). This leads to our main result: if all agents
useThompsonsamplingoverourclassofmulti-agentenvironments, thenforevery ">0
§7.1Reﬂective Oracles 129
they converge to an "-Nash equilibrium. This is not the ﬁrst time Thompson sampling
is used in game theory (Ortega and Braun, 2014), but the ﬁrst time to show that it
achieves such general positive results.
The central idea to our construction is based on reﬂective oracles introduced by
Fallenstein et al. (2015a,b). Reﬂective oracles are probabilistic oracles similar to halting
oracles that answer whether the probability that a given probabilistic Turing machine
Toutputs 1is higher than a given rational number p. The oracles are reﬂective in
the sense that the machine Tmay itself query the oracle, so the oracle has to answer
queries about itself. This invites issues caused by self-referential liar paradoxes of the
form “if the oracle says that this machine return 1with probability >1=2, then return
0, else return 1.” Reﬂective oracles avoid these issues by being allowed to randomize if
the machines do not halt or the rational number is exactlythe probability to output 1.
We introduce reﬂective oracles formally in Section 7.1 and prove that there is a limit
computable reﬂective oracle.
For inﬁnitely repeated games practical algorithms rely on ﬁcticious play (Fudenberg
and Levine, 1998, Ch. 2): the agent takes a best-response action based on the assump-
tion that its opponent is playing a stationary but unknown mixed strategy estimated
according to the observed empirical frequencies. If all agents converge to a stationary
policy, then this is a Nash equilibrium. However, convergence is not guaranteed.
The same problem occurs in multi-agent reinforcement learning (Busoniu et al.,
2008). Reinforcement learning algorithms typically assume a (stationary) Markov de-
cision process. This assumption is violated when interacting with other reinforcement
learning agents because as these agents learn, their behavior changes and thus they are
not stationary. Assuming convergence to a stationary policy is a necessary criterion to
enable all agents to learn, but the process is unstable for many reinforcement learning
algorithms and only empirical positive results are known (Bowling and Veloso, 2001).
7.1 Reﬂective Oracles
7.1.1 Deﬁnition
First we connect semimeasures as deﬁned in Deﬁnition 2.14 to Turing machines. In
Chapter 2 we used monotone Turing machines which naturally correspond to lower
semicomputable semimeasures (Li and Vitányi, 2008, Sec. 4.5.2) that describe the dis-
tribution that arises when piping fair coin ﬂips into the monotone machine. Here we
take a diﬀerent route.
Aprobabilistic Turing machine is a Turing machine that has access to an unlimited
number of uniformly random coin ﬂips. Let Tdenote the set of all probabilistic Turing
machines that take some input in Xand may query an oracle (formally deﬁned below).
We take a Turing machine T2Tto correspond to a semimeasure TwhereT(ajx)
is the probability that Toutputsa2Xwhen given x2Xas input. The value of
130 The Grain of Truth Problem
T(x)is then given by the chain rule
T(x) :=jxjY
k=1T(xkjx<k): (7.1)
ThusTgives rise to the set of semimeasures MLSCwhere the conditionals (ajx)are
lower semicomputable. In contrast, in Chapter 6 we considered semimeasures whose
jointprobability (7.1) is lower semicomputable. This set MLSCcontains all com-
putable measures. However, MLSCis a proper subset of the set of all lower semicom-
putable semimeasures because the product (7.1) is lower semicomputable, but there
are some lower semicomputable semimeasures whose conditional is not lower semicom-
putable (Theorem 6.6):
MCCM
compM LSCMCCS
LSC
In the following we assume that our alphabet is binary, i.e., X:=f0;1g.
Deﬁnition 7.2 (Oracle).Anoracleis a function O:Tf 0;1gQ!f0;1g.
Oracles are understood to be probabilistic: they randomly return 0or1. LetTOde-
notethemachine T2Twhenrunwiththeoracle O, andletO
Tdenotethesemimeasure
induced by TO. This means that drawing from O
Tinvolves two sources of randomness:
one from the distribution induced by the probabilistic Turing machine Tand one from
the oracle’s answers.
The intended semantics of an oracle are that it takes a query (T;x;p )and returns
1if the machine TOoutputs 1on inputxwith probability greater than pwhen run
with the oracle O, i.e., when O
T(1jx)> p. Furthermore, the oracle returns 0if the
machineTOoutputs 1on inputxwith probability less than pwhen run with the oracle
O, i.e., when O
T(1jx)< p. To fulﬁll this, the oracle Ohas to make statements
about itself, since the machine Tfrom the query may again query O. Therefore we call
oracles of this kind reﬂective oracles . This has to be deﬁned very carefully to avoid the
obvious diagonalization issues that are caused by programs that ask the oracle about
themselves. We impose the following self-consistency constraint.
Deﬁnition7.3 (ReﬂectiveOracle) .AnoracleOisreﬂective iﬀforallqueries (T;x;p )2
Tf 0;1gQ,
(a)O
T(1jx)>pimpliesO(T;x;p ) = 1, and
(b)O
T(0jx)>1pimpliesO(T;x;p ) = 0.
Ifpunder- or overshoots the true probability of O
T(jx), then the oracle must
reveal this information. However, in the critical case when p=O
T(1jx), the oracle is
allowed to return anything and may randomize its result. Furthermore, since Tmight
not output any symbol, it is possible that O
T(0jx) +O
T(1jx)<1. In this case
the oracle can reassign the non-halting probability mass to 0,1, or randomize; see
Figure 7.1.
§7.1Reﬂective Oracles 131
0 1O
T(0jx) O
T(1jx)Oreturns 1Omay randomize Oreturns 0
Figure 7.1: Answer options of a reﬂective oracle Ofor the query (T;x;p ); the rational
p2[0;1]falls into one of the three regions above. The values of O
T(0jx)andO
T(1jx)
are depicted as the length of the line segment under which they are written.
Example 7.4 (Reﬂective Oracles and Diagonalization) .LetT2Tbe a probabilistic
Turing machine that outputs 1O(T;;1=2)(Tcan know its own source code by
quining; Kleene, 1952, Thm. 27). In other words, Tqueries the oracle about whether it
is more likely to output 1or0, and then does whichever the oracle says is less likely. In
this case we can use an oracle O(T;;1=2) := 1=2(answer 0or1with equal probability),
which implies O
T(1j) =O
T(0j) = 1=2, so the conditions of Deﬁnition 7.3 are
satisﬁed. In fact, for this machine Twe must have O(T;;1=2) = 1=2for all reﬂective
oraclesO. 3
The following theorem establishes that reﬂective oracles exist.
Theorem 7.5 (Fallenstein et al., 2015c, App. B) .There is a reﬂective oracle.
Deﬁnition 7.6 (Reﬂective-Oracle-Computable) .A semimeasure is called reﬂective-
oracle-computable iﬀ it is computable on a probabilistic Turing machine with access to
a reﬂective oracle.
For any probabilistic Turing machine T2 Twe can complete the semimeasure
O
T(jx)into a reﬂective-oracle-computable measure O
T(jx): Using the oracle Oand
a binary search on the parameter pwe search for the crossover point pwhereO(T;x;p )
goes from returning 1to returning 0. The limit point p
x2Rof the binary search is
random since the oracle’s answers may be random. But the main point is that the
expectation of p
xexists, soO
T(1jx) =E[p
x] = 1O
T(0jx). HenceO
Tis a measure.
Moreover, if the oracle is reﬂective, then O
T(x)O
T(x)for allx2X. In this sense
the oracleOcan be viewed as a way of ‘completing’ all semimeasures O
Tto measures
by arbitrarily assigning the non-halting probability mass. If the oracle Ois reﬂective
this is consistent in the sense that Turing machines who run other Turing machines will
be completed in the same way. This is especially important for a universal machine
that runs all other Turing machines to induce a Solomonoﬀ prior (Example 3.5).
7.1.2 A Limit Computable Reﬂective Oracle
The proof of Theorem 7.5 given by Fallenstein et al. (2015c, App. B) is nonconstructive
and uses the axiom of choice. In Section 7.1.3 we give a new proof for the existence
of reﬂective oracles and provide a construction that there is a reﬂective oracle that is
limit computable.
132 The Grain of Truth Problem
Theorem 7.7 (A Limit Computable Reﬂective Oracle) .There is a reﬂective oracle
that is limit computable.
This theorem has the immediate consequence that reﬂective oracles cannot be used
as halting oracles. At ﬁrst, this result may seem surprising: according to the deﬁnition
of reﬂective oracles, they make concrete statements about the output of probabilistic
Turing machines. However, the fact that the oracles may randomize some of the time
actually removes enough information such that halting can no longer be decided from
the oracle output.
Corollary 7.8 (Reﬂective Oracles are not Halting Oracles) .There is no probabilistic
Turing machine Tsuch that for every preﬁx program pand every reﬂective oracle O,
we have that O
T(1jp)>1=2ifphalts andO
T(1jp)<1=2otherwise.
Proof.Assume there was such a machine Tand letObe the limit computable oracle
from Theorem 7.7. Since Ois reﬂective we can turn Tinto a deterministic halting oracle
bycallingO(T;p;1=2)whichdeterministicallyreturns 1ifphaltsand 0otherwise. Since
Oislimitcomputable, wecanﬁnitelycomputetheoutputof Oonanyquerytoarbitrary
ﬁnite precision using our deterministic halting oracle. We construct a probabilistic
Turing machine T0that uses our halting oracle to compute (rather than query) the
oracleOon(T0;;1=2)to a precision of 1=3in ﬁnite time. If O(T0;;1=2)1=3>1=2,
the machine T0outputs 0, otherwise T0outputs 1. Since our halting oracle is entirely
deterministic, the output of T0is entirely deterministic as well (and T0always halts), so
O
T0(0j) = 1orO
T0(1j) = 1. Therefore O(T0;;1=2) = 1orO(T0;;1=2) = 0because
Ois reﬂective. A precision of 1=3is enough to tell them apart, hence T0returns 0if
O(T0;;1=2) = 1andT0returns 1ifO(T0;;1=2) = 0. This is a contradiction.
A similar argument can also be used to show that reﬂective oracles are not com-
putable.
7.1.3 Proof of Theorem 7.7
The idea for the proof of Theorem 7.7 is to construct an algorithm that outputs an
inﬁnite series of partial oracles converging to a reﬂective oracle in the limit.
The set of queries is countable, so we can assume that we have some computable
enumeration of it:
Tf 0;1gQ=:fq1;q2;:::g
Deﬁnition 7.9 (k-Partial Oracle) .Ak-partial oracle ~Ois function from the ﬁrst k
queries to the multiples of 2kin[0;1]:
~O:fq1;q2;:::;qkg!fn2kj0n2kg
Deﬁnition 7.10 (Approximating an Oracle) .Ak-partial oracle ~Oapproximates an
oracleOiﬀjO(qi)~O(qi)j2k1for allik.
§7.1Reﬂective Oracles 133
Letk2N, let ~Obe ak-partial oracle, and let T2Tbe an oracle machine. The
machineT~Othat we get when we run Twith thek-partial oracle ~Ois deﬁned as follows
(this is with slight abuse of notation since kis taken to be understood implicitly).
1. RunTfor at most ksteps.
2. IfTcalls the oracle on qiforik,
(a) return 1with probability ~O(qi)2k1,
(b) return 0with probability 1~O(qi)2k1, and
(c) halt otherwise.
3. IfTcalls the oracle on qjforj >k, halt.
Furthermore, we deﬁne ~O
Tanalogously to O
Tas the distribution generated by the
machineT~O.
Lemma 7.11. If ak-partial oracle ~Oapproximates a reﬂective oracle O, thenO
T(1j
x)~O
T(1jx)andO
T(0jx)~O
T(0jx)for allx2f0;1gand allT2T.
Proof.This follows from the deﬁnition of T~O: when running Twith ~Oinstead ofO, we
can only lose probability mass. If Tmakes calls whose index is > kor runs for more
thanksteps, then the execution is aborted and no further output is generated. If T
makes calls whose index ik, then ~O(qi)2k1O(qi)since ~Oapproximates O.
Therefore the return of the call qiis underestimated as well.
Deﬁnition 7.12 (k-Partially Reﬂective) .Ak-partial oracle ~Oisk-partially reﬂective
iﬀ for the ﬁrst kqueries (T;x;p )
•~O
T(1jx)>pimplies ~O(T;x;p ) = 1, and
•~O
T(0jx)>1pimplies ~O(T;x;p ) = 0.
It is important to note that we can check whether a k-partial oracle is k-partially
reﬂective in ﬁnite time by running all machines Tfrom the ﬁrst kqueries for ksteps
and tallying up the probabilities to compute ~O
T.
Lemma 7.13. IfOis a reﬂective oracle and ~Ois ak-partial oracle that approximates
O, then ~Oisk-partially reﬂective.
Lemma 7.13 only holds because we use semimeasures whose conditionals are lower
semicomputable.
Proof.Assuming~O
T(1jx)> pwe get from Lemma 7.11 that O
T(1jx)~O
T(1j
x)> p. ThusO(T;x;p ) = 1becauseOis reﬂective. Since ~Oapproximates O, we get
1 =O(T;x;p )~O(T;x;p ) + 2k1, and since ~Oassigns values in a 2k-grid, it follows
that ~O(T;x;p ) = 1. The second implication is proved analogously.
134 The Grain of Truth Problem
Deﬁnition 7.14 (Extending Partial Oracles) .Ak+ 1-partial oracle ~O0extendsak-
partial oracle ~Oiﬀj~O(qi)~O0(qi)j2k1for allik.
Lemma 7.15. There is an inﬁnite sequence of partial oracles (~Ok)k2Nsuch that for
eachk,~Okis ak-partially reﬂective k-partial oracle and ~Ok+1extends ~Ok.
Proof.By Theorem 7.5 there is a reﬂective oracle O. For every k, there is a canonical
k-partial oracle ~Okthat approximates O: restrictOto the ﬁrst kqueries and for any
such query qpick the value in the 2k-grid which is closest to O(q). By construction,
~Ok+1extends ~Okand by Lemma 7.13, each ~Okisk-partially reﬂective.
Lemma 7.16. If thek+ 1-partial oracle ~Ok+1extends the k-partial oracle ~Ok, then
~Ok+1
T(1jx)~Ok
T(1jx)and~Ok+1
T(0jx)~Ok
T(0jx)for allx2f0;1gand all
T2T.
Proof.T~Ok+1runs for one more step than T~Ok, can answer one more query and has
increased oracle precision. Moreover, since ~Ok+1extends ~Ok, we havej~Ok+1(qi)
~Ok(qi)j2k1, and thus ~Ok+1(qi)2k1~Ok(qi)2k. Therefore the success to
answers to the oracle calls (case 2(a) and 2(b)) will not decrease in probability.
Now everything is in place to state the algorithm that constructs a reﬂective oracle
in the limit. It recursively traverses a tree of partial oracles. The tree’s nodes are the
partial oracles; level kof the tree contains all k-partial oracles. There is an edge in the
tree from the k-partial oracle ~Okto thei-partial oracle ~Oiif and only if i=k+ 1and
~Oiextends ~Ok.
For everyk, there are only ﬁnitely many k-partial oracles, since they are functions
from ﬁnite sets to ﬁnite sets. In particular, there are exactly two 1-partial oracles (so
the search tree has two roots). Pick one of them to start with, and proceed recursively
as follows. Given a k-partial oracle ~Ok, there are ﬁnitely many (k+ 1)-partial oracles
that extend ~Ok(ﬁnite branching of the tree). Pick one that is (k+1)-partially reﬂective
(which can be checked in ﬁnite time). If there is no (k+1)-partially reﬂective extension,
backtrack.
By Lemma 7.15 our search tree is inﬁnitely deep and thus the tree search does
not terminate. Moreover, it can backtrack to each level only a ﬁnite number of times
because at each level there is only a ﬁnite number of possible extensions. Therefore
the algorithm will produce an inﬁnite sequence of partial oracles, each extending the
previous. Because of ﬁnite backtracking, the output eventually stabilizes on a sequence
of partial oracles ~O1;~O2;:::. By the following lemma, this sequence converges to a
reﬂective oracle, which concludes the proof of Theorem 7.7.
Lemma 7.17. Let~O1;~O2;:::be a sequence where ~Okis ak-partially reﬂective k-partial
oracle and ~Ok+1extends ~Okfor allk2N. LetO:= limk!1~Okbe the pointwise limit.
Then
(a)~Ok
T(1jx)!O
T(1jx)and~Ok
T(0jx)!O
T(0jx)ask!1for allx2f0;1g
and allT2T, and
§7.2A Grain of Truth 135
(b)Ois a reﬂective oracle.
Proof.First note that the pointwise limit must exists because j~Ok(qi)~Ok+1(qi)j
2k1by Deﬁnition 7.14.
(a) Since ~Ok+1extends ~Ok, each ~Okapproximates O. Letx2f0;1gandT2Tand
considerthesequence ak:=~Ok
T(1jx)fork2N. ByLemma7.16, akak+1, sothe
sequence is monotone increasing. By Lemma 7.11, akO
T(1jx), so the sequence
is bounded. Therefore it must converge. But it cannot converge to anything strictly
belowO
T(1jx)by the deﬁnition of TO.
(b) By deﬁnition, Ois an oracle; it remains to show that Ois reﬂective. Let qi=
(T;x;p )be some query. If p<O
T(1jx), then by (a) there is a klarge enough such
thatp < ~Ot
T(1jx)for alltk. For anytmaxfk;ig, we have ~Ot(T;x;p ) = 1
since ~Otist-partially reﬂective. Therefore 1 = limk!1~Ok(T;x;p ) =O(T;x;p ).
The case 1p<O
T(0jx)is analogous.
7.2 A Grain of Truth
7.2.1 Reﬂective Bayesian Agents
FixOto be a reﬂective oracle. From now on, we assume that the action space A:=
f;gisbinary. Wecantreatcomputablemeasuresoverbinarystringsasenvironments:
the environment corresponding to a probabilistic Turing machine T2Tis deﬁned by
(etjæ<tat) :=O
T(yjx) =kY
i=1O
T(yijxy1:::yi1)
wherey1:kis a binary encoding of etandxis a binary encoding of æ<tat. The actions
a1:1are only contextual , and not part of the environment distribution. We deﬁne
(e<tka<t)analogously to (4.1).
LetT1;T2;:::be an enumeration of all probabilistic Turing machines in Tthat use
an oracle. We deﬁne the class of reﬂective environments
MO
re
:=n
O
T1;O
T2;:::o
:
This is the class of all environments computable on a probabilistic Turing machine with
reﬂective oracle O, that have been completed from semimeasures to measures using O.
Analogously to Section 4.3.1, we deﬁne a Bayesian mixture over the class MO
re
.
Letw2MO
re
be a lower semicomputable prior probability distribution on MO
re
.
Possible choices for the prior include the Solomonoﬀ prior w
O
T
:= 2K(T), where
K(T)denotes the length of the shortest input to some universal Turing machine that
encodesT. We deﬁne the corresponding Bayesian mixture
(etjæ<tat) :=X
2MO
re
w(jæ<t)(etjæ<tat) (7.2)
136 The Grain of Truth Problem
wherew(jæ<t)is the (renomalized) posterior,
w(jæ<t) :=w()(e<tka<t)
(e<tka<t): (7.3)
The mixture is lower semicomputable on an oracle Turing machine because the pos-
teriorw(jæ<t)is lower semicomputable. Hence there is an oracle machine Tsuch
that=O
T. We deﬁne its completion :=O
Tas the completion of O
T. This is the
distribution that is used to compute the posterior. There are no cyclic dependencies
sinceis called on the shorter history æ<t. We arrive at the following statement.
Proposition 7.18 (Bayes is in the Class) .2MO
re
.
Moreover, since Oisreﬂective, wehavethat dominatesallenvironments 2MO
re
:
(e1:tka1:t) =(etjæ<tat)(e<tka<t)
(etjæ<tat)(e<tja<t)
=(e<tka<t)X
2MO
re
w(jæ<t)(etjæ<tat)
=(e<tka<t)X
2MO
re
w()(e<tka<t)
(e<tka<t)(etjæ<tat)
=X
2MO
re
w()(e1:tka1:t)
w()(e1:tka1:t)
Therefore we get on-policy value convergence according to Corollary 4.20: for all
2MO
re
and all policies
V
(æ<t)V
(æ<t)!0ast!1-almost surely. (7.4)
7.2.2 Reﬂective-Oracle-Computable Policies
This subsection is dedicated to the following result that was previously stated by Fall-
enstein et al. (2015a, Alg. 6) but not proved. It contrasts results on arbitrary semicom-
putable environments where optimal policies are not limit computable (see Section 6.3).
Theorem 7.19 (Optimal Policies are Oracle Computable) .For every2MO
re
, there
is a-optimal (stochastic) policy
that is reﬂective-oracle-computable.
Note that even though deterministic optimal policies always exist, those policies are
typically not reﬂective-oracle-computable.
To prove Theorem 7.19 we need the following lemma.
Lemma 7.20 (Reﬂective-Oracle-Computable Optimal Value Function) .For every en-
vironment2MO
re
the optimal value function V
is reﬂective-oracle-computable.
§7.3Multi-Agent Environments 137
Proof.This proof follows the proof of Corollary 6.14. We write the optimal value explic-
itly as in (4.2). For a ﬁxed m, all involved quantities are reﬂective-oracle-computable.
Moreover, this quantity is monotone increasing in mand the tail sum from m+ 1to1
is bounded by m+1which is computable according to Assumption 4.6a and converges
to0asm!1. Therefore we can enumerate all rationals above and below V
.
Proof of Theorem 7.19. According to Lemma 7.20 the optimal value function V
is
reﬂective-oracle-computable. Hence there is a probabilistic Turing machine Tsuch that
O
T(1jæ<t) =
V
(æ<t)V
(æ<t) + 1
=2:
We deﬁne the policy
(æ<t) :=(
ifO(T;æ<t;1=2) = 1;and
ifO(T;æ<t;1=2) = 0
This policy is stochastic because the answer of the oracle Ois stochastic.
It remains to show that is a-optimal policy. If V
(æ<t)> V
(æ<t), then
O
T(1jæ<t)>1=2, thusO(T;æ<t;1=2) = 1 sinceOis reﬂective, and hence
takes action . Conversely, if V
(æ<t)< V
(æ<t), thenO
T(1jæ<t)<1=2,
thusO(T;æ<t;1=2) = 0sinceOis reﬂective, and hence takes action . Lastly,
ifV
(æ<t) =V
(æ<t), then both actions are optimal and thus it does not matter
which action is returned by policy . (This is the case where the oracle may random-
ize.)
7.2.3 Solution to the Grain of Truth Problem
Together, Proposition 7.18 and Theorem 7.19 provide the necessary ingredients to solve
the grain of truth problem (Problem 7.1).
Corollary 7.21 (Solution to the Grain of Truth Problem) .For every lower semicom-
putable prior w2MO
re
the Bayes optimal policy
is reﬂective-oracle-computable
whereis the Bayes-mixture corresponding to wdeﬁned in (7.2).
Proof.From Proposition 7.18 and Theorem 7.19.
Hence the environment class MO
re
contains any reﬂective-oracle-computable modi-
ﬁcation of the Bayes optimal policy
. In particular, this includes computable multi-
agent environments that contain other Bayesian agents over the class MO
re
. So any
Bayesian agent over the class MO
re
has a grain of truth even though the environment
may contain other Bayesian agents of equal power . We proceed to sketch the implica-
tions for multi-agent environments in the next section.
7.3 Multi-Agent Environments
In amulti-agent environment there arenagents each taking sequential actions from the
ﬁnite action space A. In each time step t= 1;2;:::, the environment receives action ai
t
138 The Grain of Truth Problem
agent1
agent2
...
agentnmulti-agent
environment
a1
t
e1
t
a2
t
e2
t
an
t
en
t
Figure 7.2: Agents1;:::;ninteracting in a multi-agent environment.
from agent iand outputs nperceptse1
t;:::;en
t2E, one for each agent. Each percept
ei
t= (oi
t;ri
t)contains an observation oi
tand a reward ri
t2[0;1]. Importantly, agent i
only sees its own action ai
tand its own percept ei
t(see Figure 7.2). We use the shorthand
notationat:= (a1
t;:::;an
t)andet:= (e1
t;:::;en
t)and denote æi
<t=ai
1ei
1:::ai
t1ei
t1
andæ<t=a1e1:::at1et1. Formally, multi-agent environments are deﬁned as follows.
Deﬁnition 7.22 (Multi-Agent Environment) .Amulti-agent environment is a function
: (AnEn)An!(En):
Together with the policies 1;:::;nthe multi-agent environment induces a history
distribution 1:nwhere
1:n() : = 1
1:n(æ1:t) : =1:n(æ<tat)(etjæ<tat)
1:n(æ<tat) : =1:n(æ<t)nY
i=1i(ai
tjæi
<t):
Agentiacts in a subjective environment igiven by joining the multi-agent en-
vironmentwith the policies 1;:::;nand marginalizing over the histories that i
does not see. Together with policy i, the environment iyields a distribution over the
histories of agent i
i
i(æi
<t) :=X
æj
<t;j6=i1:n(æ<t):
Wegetthedeﬁnitionofthesubjectiveenvironment iwiththeidentity i(ei
tjæi
<tai
t) :=
i
i(ei
tjæi
<tai
t). The subjective environment idepends on ibecause other policies’
actions may depend on the actions of i. It is crucial to note that the subjective
environment iand the policy iare ordinary environments and policies, so we can use
the notation from Chapter 4.
§7.3Multi-Agent Environments 139
Our deﬁnition of a multi-agent environment is very general and encompasses most
of game theory. It allows for cooperative, competitive, and mixed games; inﬁnitely
repeatedgamesorany(inﬁnite-length)extensiveformgameswithﬁnitelymanyplayers.
Example 7.23 (Matching Pennies) .In the game of matching pennies there are two
agents (n= 2), and two actions A=f;grepresenting the two sides of a penny. In
each time step agent 1wins if the two actions are identical and agent 2wins if the two
actions are diﬀerent. The payoﬀ matrix is as follows.

1,0 0,1
0,1 1,0
We useE=f0;1gto be the set of rewards (observations are vacuous) and deﬁne the
multi-agent environment to give reward 1to agent 1iﬀa1
t=a2
t(0otherwise) and
reward 1to agent 2iﬀa1
t6=a2
t(0otherwise). Formally,
(r1
tr2
tjæ<tat) :=8
>><
>>:1ifr1
t= 1;r2
t= 0;a1
t=a2
t;
1ifr1
t= 0;r2
t= 1;a1
t6=a2
t;and
0otherwise:
Letdenote the policy that always takes action . If two agents each using policy
play matching pennies, agent 1wins in every step. Formally, setting 1:=2:=we
get a history distribution that assigns probability one to the history
1010::::
The subjective environment of agent 1is
1(r1
tjæ1
<ta1
t) =8
>><
>>:1ifr1
t= 1;a1
t=;
1ifr1
t= 0;a1
t=;and
0otherwise:
Therefore policy is optimal in agent 1’s subjective environment. 3
Deﬁnition 7.24 ("-Best Response) .A policyiacting in multi-agent environment
with policies 1;:::;nis an"-best response after history æi
<tiﬀ
V
i(æi
<t)Vii(æi
<t)<":
If at some time step t, all agents’ policies are "-best responses, we have an "-
Nash equilibrium . The property of multi-agent systems that is analogous to asymptotic
optimality is convergence to an "-Nash equilibrium.
140 The Grain of Truth Problem
7.4 Informed Reﬂective Agents
Letbe a multi-agent environment and let
1;:::
nbe such that for each ithe
policy
iis an optimal policy in agent i’s subjective environment i. At ﬁrst glance
this seems ill-deﬁned: The subjective environment idepends on each policy
j, which
depends on the subjective environment j, which in turn depends on the policy
i.
However, this circular deﬁnition actually has a well-deﬁned solution.
Theorem 7.25 (Optimal Multi-Agent Policies) .For any reﬂective-oracle-computable
multi-agent environment , the optimal policies
1;:::;
nexist and are reﬂective-
oracle-computable.
To prove Theorem 7.25, we need the following proposition.
Proposition 7.26 (Reﬂective-Oracle-Computability) .If the multi-agent environment
and the policies 1;:::;nare reﬂective-oracle-computable, then 1:nandi
iare
reﬂective-oracle-computable, and i2MO
re
.
Proof.All involved quantities in the deﬁnition of 1:nare reﬂective-oracle-computable
by assumption, therefore also their marginalizations i
iandi.
Proof of Theorem 7.25. According to Theorem 7.19 the optimal policy
iin agenti’s
subjective environment is reﬂective-oracle-computable if the subjective environment i
is. In particular the process that takes iin form of a probabilistic Turing machine and
returns
iis reﬂective-oracle-computable. Moreover, iis reﬂective-oracle-computable
ifand1;:::;nare, according to Proposition 7.26. Again, this construction is itself
reﬂective-oracle-computable. Connecting these two constructions we get probabilistic
Turing machines T1;:::;Tn2Twhere each Titakes the multi-agent environment
and1;:::;nin form of probabilistic Turing machines and returns
i. We deﬁne the
probabilistic Turing machines T0
1;:::;T0
nwhereT0
irunsTO
ion(;T0
1;:::;T0
n); henceT0
i
computes
i. Note that this construction only works because we relied on the reﬂective
oracle in the proof of Theorem 7.19. Since the machines TO
ialways halt, so do T0
iO
despite their inﬁnitely recursive construction.
Note the strength of Theorem 7.25: each of the policies
iis acting optimally
given the knowledge of everyone else’s policies . Hence optimal policies play 0-best re-
sponses by deﬁnition, so if every agent is playing an optimal policy, we have a Nash
equilibrium. Moreover, this Nash equilibrium is also a subgame perfect Nash equi-
librium, because each agent also acts optimally on the counterfactual histories that
do not end up being played. In other words, Theorem 7.25 states the existence and
reﬂective-oracle-computability of a subgame perfect Nash equilibrium in any reﬂective-
oracle-computable multi-agent environment. The following immediate corollary states
that these subgame perfect Nash equilibria are limit computable.
Corollary 7.27 (Solution to Computable Multi-Agent Environments) .For any com-
putable multi-agent environment , the optimal policies
1;:::;
nexist and are limit
computable.
§7.5Learning Reﬂective Agents 141
Proof.From Theorem 7.25 and Theorem 7.7.
Example 7.28 (Nash Equilibrium in Matching Pennies) .Consider the matching pen-
nies game from Example 7.23. The only pair of optimal policies is the pair of two
uniformly random policies that play andwith equal probability in every time step:
if one of the agents picks a policy that plays one of the actions with probability >1=2,
then the other agent’s best response is to play the other action with probability 1. But
now the ﬁrst agent’s policy is no longer a best response. 3
7.5 Learning Reﬂective Agents
Since our classMO
re
solves the grain of truth problem, the result by Kalai and Lehrer
(1993) immediately implies that for any Bayesian agents 1;:::;ninteracting in an
inﬁnitely repeated game and for all ">0and alli2f1;:::;ngthere is almost surely a
t02Nsuch that for all tt0the policyiis an"-best response. However, this hinges
on the important fact that every agent has to know the game and also that all other
agents are Bayesian agents. Otherwise the convergence to an "-Nash equilibrium may
fail, as illustrated by the following example.
At the core of the construction is a dogmatic prior (Section 5.2.2). A dogmatic prior
assigns very high probability to going to hell (reward 0forever) if the agent deviates
from a given computable policy . For a Bayesian agent it is thus only worth deviating
from the policy if the agent thinks that the prospects of following are very poor
already. This implies that for general multi-agent environments and without additional
assumptions on the prior, we cannot prove any meaningful convergence result about
Bayesian agents acting in an unknown multi-agent environment.
Example 7.29 (Reﬂective Bayesians Playing Matching Pennies) .Consider the multi-
agent environment matching pennies from Example 7.23. Let 1be the policy that
takes the action sequence ()1and let2:=be the policy that always takes
action. The average reward of policy 1is2=3and the average reward of policy
2is1=3. Letbe a universal mixture (7.2). By on-policy value convergence (7.4),
V1
!c12=3andV2
!c21=3almost surely when following policies (1;2).
Therefore there is an " >0such thatV1
> "andV2
> "for all time steps. Now
we can apply Theorem 5.5 to conclude that there are (dogmatic) mixtures 0
1and0
2
such that
0
1always follows policy 1and
0
2always follows policy 2. This does not
converge to a ( "-)Nash equilibrium. 3
An important property required for the construction in Example 7.29 is that the
environment class contains environments that threaten the agent with going to hell,
which is outside of the class of matching pennies environments. In other words, since
the agent does not know a priori that it is playing a matching pennies game, it might
behave more conservatively than appropriate for the game.
The following theorem is our main convergence result. It states that for asymptot-
ically optimal agents we get convergence to "-Nash equilibria in any reﬂective-oracle-
computable multi-agent environment.
142 The Grain of Truth Problem
Theorem 7.30 (Convergence to Equilibrium) .Letbe an reﬂective-oracle-computable
multi-agent environment and let 1;:::;nbe reﬂective-oracle-computable policies that
are asymptotically optimal in mean in the class MO
re
. Then for all " > 0and all
i2f1;:::;ngthe1:n-probability that the policy iis an"-best response converges to
1ast!1.
Proof.Leti2 f1;:::;ng. By Proposition 7.26, the subjective environment iis
reﬂective-oracle-computable, therefore i2MO
re
. Sinceiis asymptotically optimal
in mean in the class MO
re
, we get that E[V
i(æ<t)Vii(æ<t)]!0. Convergence in
mean implies convergence in probability for bounded random variables, hence for all
">0we have
i
i[V
i(æi
<t)Vii(æi
<t)"]!0ast!1:
Therefore the probability that the policy iplays an"-best response converges to 1as
t!1.
In contrast to Theorem 7.25 which yields policies that play a subgame perfect equi-
librium, this is not the case for Theorem 7.30: the agents typically do not learn to
predict oﬀ-policy and thus will generally not play "-best responses in the counterfac-
tual histories that they never see. This weaker form of equilibrium is unavoidable if the
agents do not know the environment because it is impossible to learn the parts that
they do not interact with.
Corollary 7.31 (Convergence to Equilibrium) .There are limit computable policies
1;:::;nsuch that for any computable multi-agent environment and for all">0and
alli2f1;:::;ngthe1:n-probability that the policy iis an"-best response converges
to1ast!1.
Proof.Pick1;:::;nto be the Thompson sampling policy Tdeﬁned in Algorithm 2
over the countable class MO
re
. By Theorem 5.25 these policies are asymptotically op-
timal in mean. By Theorem 7.32 below they are reﬂective-oracle-computable and by
Theorem 7.7 they are also limit computable. The statement now follows from Theo-
rem 7.30.
Theorem 7.32 (Thompson Sampling is Reﬂective-Oracle-Computable) .The policyT
deﬁned in Algorithm 2 over the class MO
re
is reﬂective-oracle-computable.
Proof.The posterior w(jæ<t)is reﬂective-oracle-computable by the deﬁnition (7.3)
and according to Theorem 7.19 the optimal policies
are reﬂective-oracle-computable.
On resampling steps we can compute the action probabilities of Tby enumerating all
2MO
re
and computing
weighted by the posterior w(jæ<t). Between resampling
steps we need to condition the policy Tcomputed above by the actions it has already
taken since the last resampling step (compare Example 5.28).
Because the posterior w(jæ<t)is a-martingale when acting according to the
policy, it converges -almost surely according to the martingale convergence theo-
rem (Theorem 2.8). Since dominates the subjective environment i, it also converges
§7.6Impossibility Results 143

i-almost surely (see Example 3.20). Hence all the Thompson sampling agents even-
tually ‘calm down’ and settle on some posterior belief.
According to Theorem 7.30, the policies 1;:::;nonly need to be asymptotically
optimal in mean. For Thompson sampling this is independent of the discount function
according to Theorem 5.25 (but the discount function has to be known to the agent).
So the diﬀerent agents may use diﬀerent discount functions, resample at diﬀerent time
steps and converge at diﬀerent speeds.
Example 7.33 (Thompson Samplers Playing Matching Pennies) .Consider the match-
ing pennies game from Example 7.23 and let both agents use the Thompson sampling
policy deﬁned in Algorithm 2, i.e., deﬁne 1:=Tand2:=T.
The value of the uniformly random policy Ris always 1=2, soV
iVRi= 1=2.
According to Theorem 7.30, for every ">0, each agent will eventually always play an
"-best response, i.e., Vii> V
i"1=2". Since matching pennies is a zero-sum
game,V11+V22= 1, soV22= 1V11<1=2 +".
Therefore each agent will end up randomizing their actions; i(atjæ<t)1=2most
of the time: If one of the agents (say agent 1) does not suﬃciently randomize their
actions in some time steps, then agent 2 could exploit this by picking a deterministic
adversarial policy in those time steps. Suppose that this way it can gain a value of
"compared to the random policy R, i.e.,V22VR2+"= 1=2 +". But this is a
contradiction because then agent 1 is not playing an "-best response:
V
1V11=V
11 +V221=21 + 1=2 +"=" 3
7.6 Impossibility Results
Why does our solution to the grain of truth problem not violate the impossibility results
from the literature? Assume we are playing an inﬁnitely repeated game where in the
stagegamenoagenthasaweaklydominantactionandthepureactionmaxminrewardis
strictly less then the minmax reward. The impossibility result of Nachbar (1997, 2005)
states that there is no class of policies such that the following are simultaneously
satisﬁed.
•Learnability. Each agent learns to predict the other agent’s actions.
•Caution and Symmetry. The set is closed under simple policy modiﬁcations
such as renaming actions.
•Purity.There is an " >0such that for any stochastic policy 2there is a
deterministic policy 02such that if 0(æ<t) =a, then(ajæ<t)>".
•Consistency. Each agent always has an "-best response available in .
In order to converge to an "-Nash equilibrium, each agent has to have an "-best response
available to them, so consistency is our target. Learnability is immediately satisﬁed for
any environment in our class if we have a dominant prior according to Corollary 4.20.
144 The Grain of Truth Problem
ForMO
re
caution and symmetry are also satisﬁed since this set is closed under any
computable modiﬁcations to policies. However, our class MO
re
avoids this impossibility
result because it violates the purity condition: Let T1;T2;:::be an enumeration of T.
Consider the policy that maps history æi
<tto the action 1O(Tt;æi
<t;1=2). IfTtis
deterministic, then will take a diﬀerent action than Ttfor any history of length t1.
Therefore no deterministic reﬂective-oracle-computable policy can take an action that
assigns positive probability to in every time step.
Foster and Young (2001) present a condition that makes convergence to a Nash
equilibrium impossible: if the player’s rewards are perturbed by a small real number
drawn from some continuous density , then for-almost all realizations the players do
not learn to predict each other and do not converge to a Nash equilibrium. For example,
in a matching pennies game, rational agents randomize only if the (subjective) values
of both actions are exactly equal. But this happens only with -probability zero, since
is a density. Thus with -probability one the agents do not randomize. If the agents
do not randomize, they either fail to learn to predict each other, or they are not acting
rationally according to their beliefs: otherwise they would seize the opportunity to
exploit the other player’s deterministic action.
But this does not contradict our convergence result: the class MO
re
is countable
and each2MO
re
has positive prior probability. Perturbation of rewards with ar-
bitrary real numbers is not possible. Even more, the argument given by Foster and
Young (2001) cannot work in our setting: the Bayesian mixture mixes overTfor all
probabilistic Turing machines T. For Turing machines Tthat sometimes do not halt,
the oracle decides how to complete Tinto a measure T. Thus the oracle has enough
inﬂuence on the exact values in the Bayesian mixture that the values of two actions in
matching pennies can be made exactly equal.
7.7 Discussion
Thischapterintroducedtheclassofallreﬂective-oracle-computableenvironments MO
re
.
This class fully solves the grain of truth problem (Problem 7.1) because it contains (any
computablemodiﬁcationof)Bayesianagentsdeﬁnedover MO
re
: theoptimalagentsand
Bayes optimal agents over the class are all reﬂective-oracle-computable (Theorem 7.19
and Corollary 7.21).
If the environment is unknown, then a Bayesian agent may end up playing subop-
timally (Example 7.29). However, if each agent uses a policy that is asymptotically
optimal in mean (such as the Thompson sampling policy from Section 4.3.4) then for
every" >0the agents converge to an "-Nash equilibrium (Theorem 7.30 and Corol-
lary 7.31).
However, Corollary 7.31 does notimply that two Thompson sampling policies will
converge to cooperation in an iterated prisoner’s dilemma since always defecting is also
a Nash equilibrium. The exact outcome will depend on the priors involved and the
randomness of the policies.
Our solution to the grain of truth problem is purely theoretical. However, Theo-
rem 7.7 shows that our class MO
re
allows for computable approximations. This suggests
§7.7Discussion 145
that practical approaches can be derived from this result, and reﬂective oracles have
already seen applications in one-shot games (Fallenstein et al., 2015b).
146 The Grain of Truth Problem
Chapter 8
Conclusion
The biggest existential risk is that future superintelligences stop simulating us.
— Nick Bostrom
Today computer programs exceeding humans in general intelligence are known only
from science ﬁction. But research on AI has progressed steadily over the last decades
and there is good reason to believe that we will be able to build HLAI eventually, and
even sooner than most people think (Müller and Bostrom, 2016).
The advent of strong AI would be the biggest event in human history. Potential
beneﬁts are huge, as the new level of automation would free us from any kind of un-
desirable labor. But there is no reason to believe that humans are at the far end of
the intelligence spectrum. Rather, humans barely cross the threshold for general intel-
ligence to be able to use language and do science. Once we engineer HLAI, it seems
unlikely that progress is going to stop; why not build even smarter machines?
This could lead to an intelligence explosion (Good, 1965; Vinge, 1993; Kurzweil,
2005; Chalmers, 2010; Hutter, 2012a; Schmidhuber, 2012; Muehlhauser and Salamon,
2012; Eden et al., 2013; Shanahan, 2015; Eden, 2016; Walsh, 2016): a (possibly very
rapid)increaseinintelligence, e.g., throughself-ampliﬁcationeﬀectsfromAIsimproving
themselves (if doing AI research is one of humans’ capabilities, then a machine that can
do everything humans can do can also do AI research). Once machine intelligence is
above or far above human level, machines would steer the course of history. There is no
reason to believe that machines would be adversarial to us, but nevertheless humanity’s
fate might rest at the whims of the machines, just as chimpanzees today have no say
in the large-scale events on this planet.
Bostrom (2002) deﬁnes:
Existential risk — One where an adverse outcome would either annihilate
Earth-originating intelligent life or permanently and drastically curtail its
potential.
Existential risks are events that have the power to extinguish human life as we know
it. Examples are cosmic events such as an asteroid colliding with Earth. But cosmic
eventsareunlikelyonhumantimescalescomparedtohuman-madeexistentialrisksfrom
nuclear weapons, synthetic biology, and nanotechnology.
It is possible that artiﬁcial intelligence also falls into this category. Vinge (1993)
was the ﬁrst person to recognize this:
147
148 Conclusion
Within thirty years, we will have the technological means to create super-
human intelligence. Shortly after, the human era will be ended.
After Vinge, Yudkowsky (2001, 2008) can be regarded as one of the key people to
popularize the potential dangers of AI technology. Bostrom (2003) picked up on this
issue very early and gave the topic credibility through his well-researched and carefully
written book Superintelligence (Bostrom, 2014). He argues that we need to solve the
Control Problem —the unique principal-agent problem that arises with the creation of
strong AI (Bostrom, 2014, Ch. 9). In other words: How do we align strong AI with
human values? How do we ensure that AI remains robust and beneﬁcial? This research
is collected under the umbrella term AI safety . Currently, we have no idea how to solve
these problems even in theory.
Through Yudkowsky’s and, more importantly, Bostrom’s eﬀorts, AI-related long-
term safety concerns have now entered the mainstream media. In 2014 high-proﬁle
scientists such as Stephen Hawking, Max Tegmark, Stuart Russell, and Frank Wilczek
have warned against the dangers posed by AI (Hawking et al., 2014). (See also Alexan-
der (2015) for a collection of positions by prominent AI researchers.) Many scientists
inside and outside the ﬁeld have signed an open letter that research ensuring that AI
systems remain robust and beneﬁcial is both important and timely (Future of Life In-
stitute, 2015c). This lead entrepreneur Elon Musk to donate $10 million to kick-start
research in this ﬁeld (Future of Life Institute, 2015a); most of this money has now
been distributed across the planet to 37 diﬀerent projects (Future of Life Institute,
2015b). Moreover, the Future of Life Institute and the Machine Intelligence Research
Institute have formulated concrete technical research priorities to make AI more robust
and beneﬁcial (Soares and Fallenstein, 2014; Russell et al., 2015).
At the end of 2015 followed the announcement of OpenAI, a nonproﬁt organization
with ﬁnancial backing from several famous silicon valley billionaires (OpenAI, 2015):
OpenAI is a non-proﬁt artiﬁcial intelligence research company. Our goal
is to advance digital intelligence in the way that is most likely to beneﬁt
humanity as a whole, unconstrained by a need to generate ﬁnancial return.
ThemissionofOpenAIistoenableeveryonetobeneﬁtfromAItechnology. But, despite
the name, OpenAI is not committed to publish all of their research freely, and Bostrom
(2016) argues that unrestricted publication might not be the best idea.
Despite all of the recent eﬀorts in AI safety research, critical voices within the AI
community remain. Prominently, Davis (2014), Ng (2016), Walsh (2016), and Lawrence
(2016) have proposed counterarguments that range from ‘HLAI is so far away that any
worry is misplaced’ to claims that ‘the safety problem would not be so hard’. See Sotala
and Yampolskiy (2014) for a discussion.
If AI poses an existential risk then a formal theory of strong AI is paramount to
develop technical approaches to mitigate this risk. Which path will ultimately lead us
to HLAI is in the realm of speculation at this time; therefore we should make as few and
as weak assumptions as possible and abstract away from possible future implementation
details.
149
This thesis lays some of the groundwork for this endeavor. We built on top of Hut-
ter’s theory of universal artiﬁcial intelligence. Chapter 3 discussed the formal theory of
learning. Chapter 4 presented several approaches to acting in unknown environments
(Bayes, Thompson sampling, knowledge-seeking agents, and BayesExp). Chapter 5
analysed these approaches and discussed notions of optimality and principled prob-
lems with acting under uncertainty in general environment. Chapter 6 provided the
mathematical tools to analyze the computational properties of these models. Finally,
Chapter 7 solved the grain of truth problem, which lead to convergence to Nash equi-
libria in unknown general multi-agent environments.
Our work is theoretical by nature and there is still a long way to go until these
results make their way into applications. But a solution in principle is a crucial ﬁrst
step towards solving a problem in practice. Consider the research paper by Shannon
(1950) on how to solve chess in principle. The algorithm he describes expands the full
game tree of chess (until some speciﬁed depth), which is completely infeasible even with
today’s computation power. His contribution was to show that winning at chess is a
feat that computers can achieve in principle , which was not universally accepted at the
time. Even more, his approach already considered the correct ideas (minimax-search
over the game tree) that ultimately lead to the defeat of the chess champion Garry
Kasparov by the computer Deep Blue 46 years later (IBM, 2012a).
The theory of general reinforcement learning can serve and has served as a starting
point for future investigation in AI safety. In particular, value learning (Dewey, 2011),
self-reﬂection (Soares, 2015; Fallenstein et al., 2015b), self-modiﬁcation (Orseau and
Ring, 2011, 2012a; Everitt et al., 2016), interruptibility (Orseau and Armstrong, 2016;
Armstrong and Orseau, 2016), decision theory (Everitt et al., 2015), memory manip-
ulation (Orseau and Ring, 2012b), wireheading (Ring and Orseau, 2011; Everitt and
Hutter, 2016), and questions of identity (Orseau, 2014b,c).
It is possible that HLAI is decades or centuries away. It might also be a few years
around the corner. Whichever is the case, we are currently completely unprepared for
the consequences. As an AI researcher, it is tempting to devote your life to increasing
the capability of AI, advancing it domain after domain, and showing oﬀ with ﬂashy
demos and impressive victories over human contestants. But every technology incurs
risks, and the more powerful the technology, the higher the risks. The potential power
of AI technology is enormous, and correspondingly we need to consider the risks, take
them seriously, and proceed to mitigate them.
150 Conclusion
Appendix A
Measures and Martingales
In this chapter we provide the proofs for Theorem 3.18 and Theorem 3.19.
Proof of Theorem 3.18. Xtis only undeﬁned if P(v1:t) = 0. The set
fv21j9t:P(v1:t) = 0g
hasP-measure 0and hence (Xt)t2Nis well-deﬁned almost everywhere.
Xtis constant on ufor allu2t, andFtis generated by a collection of ﬁnitely
many disjoint sets:
1=]
u2tu:
(a) Therefore XtisFt-measurable.
(b)u=U
a2uafor allu2tandv2u, and therefore
E[Xt+1jFt](v) =1
P(u)X
a2Xt+1(ua)P(ua) =1
P(u)X
a2Q(ua)
P(ua)P(ua)
()=1
P(u)X
a2Q(ua) =Q(u)
P(u)=Xt(v):
At()we used the fact that Pis locally absolutely continuous with respect to Q.
(IfPwere not locally absolutely continuous with respect to Q, then there are cases
whereP(u)>0,P(ua) = 0, andQ(ua)6= 0. Therefore Xt+1(ua)does not
contribute to the expectation and thus Xt+1(ua)P(ua) = 06=Q(ua).)
P0andQ0by deﬁnition, thus Xt0. SinceP() =Q() = 1, we have
E[X0] = 1.
The following lemma gives a convenient condition for the existence of a measure on
(!;F1). It is a special case of the Daniell-Kolmogorov Extension Theorem (Rogers
and Williams, 1994, Thm. 26.1).
151
152 Measures and Martingales
Lemma A.1 (Extending measures) .Letq: ![0;1]be a function such that q() = 1
andP
a2q(ua) =q(u)for allu2. Then there exists a unique probability measure
Qon(1;F1)such thatq(u) =Q(u)for allu2.
To prove this lemma, we need the following two ingredients.
Deﬁnition A.2 (Semiring) .A setR 2
is called semiring over
iﬀ
(a);2R,
(b) for allA;B2R, the setA\B2R, and
(c) for allA;B2R, there are pairwise disjoint sets C1;:::;Cn2Rsuch thatAnB=Un
i=1Ci.
Theorem A.3 (Carathéodory’s Extension Theorem; Durrett, 2010, Thm. A.1.1) .Let
Rbe a semiring over
and let:R! [0;1]be a function such that
(a)(
) = 1(normalization) ,
(b)(Un
i=1Ai) =Pn
i=1(Ai)for pairwise disjoint sets A1;:::;An2 Rsuch thatUn
i=1Ai2R(ﬁnite additivity) , and
(c)(S
i0Ai)P
i0(Ai)for any collection (Ai)i0such that each Ai2RandS
i0Ai2R(-subadditivity) .
Then there is a unique extension ofthat is a probability measure on (
;(R))such
that(A) =(A)for allA2R.
Proof of Lemma A.1. We show the existence of Qusing Carathéodory’s Extension The-
orem. DeﬁneR:=fuju2g[f;g.
(a);2R.
(b) For any u;v2R, either
•uis a preﬁx of vandu\v= v2R, or
•vis a preﬁx of uandu\v= u2R, or
•u\v=;2R.
(c) For any u;v2R,
•unv=U
w2jvjjujnfxguwifv=ux, i.e.,uis a preﬁx of v, and
•unv=;otherwise.
ThereforeRis a semiring. By deﬁnition of R, we have(R) =F1.
The function q: ![0;1]naturally gives rise to a function :R! [0;1]with
(;) := 0and(u) :=q(u)for allu2. We will now check the prerequisites of
Carathéodory’s Extension Theorem.
153
(a) (Normalization.) (1) =() =q() = 1.
(b) (Finite additivity.) Let u1;:::; uk2Rbe pairwise disjoint sets such that w:=Uk
i=1ui2 R. Let`:= maxfjuij j1ikg, then w=U
v2`wv. By
assumption,P
a2q(ua) =q(u), thusP
a2(ua) =(u)and inductively we
have
(ui) =X
s2`juij(uis); (A.1)
and
(w) =X
v2`(wv): (A.2)
For every string v2`, the concatenation wv2w=Uk
i=1ui, so there is a
uniqueisuch thatwv2ui. Hence there is a unique string s2`juijsuch that
wv=uis. Together with (A.1) and (A.2) this yields
 k]
i=1ui!
=(w) =X
v2`(wv) =kX
i=1X
s2`juij(uis) =kX
i=1(ui):
(c) (-subadditivity.) We will show that each uis compact with respect to the topol-
ogyOgenerated byR.-subadditivity then follows from (b) because every count-
able union is in fact a ﬁnite union.
We will show that the topology Ois the product topology of the discrete topology
on. (This establishes that (!;O)is a Cantor Space.) Every projection k:
1!selecting the k-th symbol is continuous, since 1
k(a) =S
u2k1uafor
everya2. Moreover,Ois the coarsest topology with this property, since we can
generate every open set u2Rin the base of the topology by
u=juj\
i=11
i(fuig):
The set is ﬁnite and thus compact. By Tychonoﬀ’s Theorem, 1is also compact.
Therefore uis compact since it is homeomorphic to 1via the canonical map
u: 1!u,v7!uv.
From (a), (b), and (c) Carathéodory’s Extension Theorem yields a unique probability
measureQon(1;F1)such thatQ(u) =(u) =q(u)for allu2.
Using Lemma A.1, the proof of Theorem 3.19 is now straightforward.
Proof of Theorem 3.19. We deﬁne a function q: !R, with
q(u) :=Xjuj(v)P(u)
for anyv2u. The choice of vis irrelevant because Xjujis constant on usince it is
Ft-measurable. In the following, we also write Xt(u)ifjuj=tto simplify notation.
154 Measures and Martingales
The function qis non-negative because XtandPare both non-negative. Moreover,
for anyu2t,
1 =E[Xt] =Z
1XtdPZ
uXtdP=P(u)Xt(u) =q(u):
Hence the range of qis a subset of [0;1].
We haveq() =X0()P() =E[X0] = 1sincePis a probability measure and
F0=f;;1gis the trivial -algebra. Let u2t.
X
a2q(ua) =X
a2Xt+1(ua)P(ua) =Z
uXt+1dP
=Z
uE[Xt+1jFt]dP=Z
uXtdP=P(u)Xt(u) =q(u):
By Lemma A.1, there is a probability measure Qon(1;F1)such thatq(u) =Q(u)
of allu2. Therefore, for all v21and for allt2NwithP(v1:t)>0,
Xt(v) =q(v1:t)
P(v1:t)=Q(v1:t)
P(v1:t):
Moreover,Pis locally absolutely continuous with respect to QsinceP(u) = 0implies
Q(u) =q(u) =Xjuj(u)P(u) = 0:
Bibliography
Shipra Agrawal and Navin Goyal. Analysis of Thompson sampling for the multi-armed
bandit problem. In Conference on Learning Theory , 2011.
Scott Alexander. AI researchers on AI risk. http://slatestarcodex.com/2015/05/
22/ai-researchers-on-ai-risk/ , May 2015. Accessed: 2016-04-25.
Stuart Armstrong and Laurent Orseau. Interruptibility and corrigibility for AIXI and
Monte Carlo agents. 2016. To appear.
Frank Arntzenius, Adam Elga, and John Hawthorne. Bayesianism, inﬁnite decisions,
and binding. Mind, 113(450):251–283, 2004.
Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for re-
inforcement learning. In Advances in Neural Information Processing Systems , pages
89–96, 2009.
Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for rein-
forcement learning. Journal of Machine Learning Research , 11:1563–1600, 2010.
Leemon Baird. Residual algorithms: Reinforcement learning with function approxima-
tion. InInternational Conference on Machine Learning , pages 30–37, 1995.
Jacob D Bekenstein. Universal upper bound on the entropy-to-energy ratio for bounded
systems. Physical Review D , 23(2):287, 1981.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade
learningenvironment: Anevaluationplatformforgeneralagents. Journal of Artiﬁcial
Intelligence Research , 47:253–279, 2013.
Marc G Bellemare, Georg Ostrovski, Arthur Guez, Philip S Thomas, and Rémi Munos.
Increasing the action gap: New operators for reinforcement learning. In AAAI, 2016.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies
with gradient descent is diﬃcult. IEEE Transactions on Neural Networks , 5(2):157–
166, 1994.
Dimitri P Bertsekas and John Tsitsiklis. Dynamic Programming and Optimal Control .
Athena Scientiﬁc, 1995.
Christopher M Bishop. Pattern Recognition and Machine Learning . Springer, 2006.
David Blackwell and Lester Dubins. Merging of opinions with increasing information.
The Annals of Mathematical Statistics , pages 882–886, 1962.
155
156 BIBLIOGRAPHY
Nick Bostrom. Existential risks. Journal of Evolution and Technology , 9(1):1–31, 2002.
Nick Bostrom. Ethical issues in advanced artiﬁcial intelligence. Science Fiction and
Philosophy: From Time Travel to Superintelligence , pages 277–284, 2003.
Nick Bostrom. Superintelligence: Paths, Dangers, Strategies . Oxford University Press,
2014.
Nick Bostrom. Strategic implications of openness in AI development. Technical re-
port, Future of Humanity Institute, 2016. http://www.nickbostrom.com/papers/
openness.pdf .
Michael Bowling and Manuela Veloso. Rational and convergent learning in stochastic
games. In International Joint Conference on Artiﬁcial Intelligence , pages 1021–1026,
2001.
Sébastien Bubeck and Cesa-Nicolò Bianchi. Regret analysis of stochastic and non-
stochastic multi-armed bandit problems. Foundations and Trends in Machine Learn-
ing, 5(1):1–122, 2012.
Lucian Busoniu, Robert Babuska, and Bart De Schutter. A comprehensive survey
of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and
Cybernetics, Part C: Applications and Reviews , 38(2):156–172, 2008.
Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, Learning, and Games . Cambridge
University Press, 2006.
David Chalmers. The singularity: A philosophical analysis. Journal of Consciousness
Studies, 17(9-10):7–65, 2010.
Olivier Chapelle and Lihong Li. An empirical evaluation of Thompson sampling. In
Advances in Neural Information Processing Systems , pages 2249–2257, 2011.
Paul Christiano, Eliezer Yudkowsky, Marcello Herreshoﬀ, and Mihaly Barasz. Deﬁn-
ability of truth in probabilistic logic. Technical report, Machine Intelligence Research
Institute, 2013. https://intelligence.org/files/DefinabilityTruthDraft.pdf .
Thomas M Cover and Joy A Thomas. Elements of Information Theory . John Wiley &
Sons, 2nd edition, 2006.
Mayank Daswani. Generic Reinforcement Learning Beyond Small MDPs . PhD thesis,
Australian National University, 2015.
Mayank Daswani and Jan Leike. A deﬁnition of happiness for reinforcement learning
agents. In Artiﬁcial General Intelligence , pages 231–240. Springer, 2015.
Ernest Davis. Ethical guidelines for a superintelligence. Technical report, New York
University, 2014. https://cs.nyu.edu/davise/papers/Bostrom.pdf .
BIBLIOGRAPHY 157
Adam Day. Increasing the gap between descriptional complexity and algorithmic proba-
bility.Transactions of the American Mathematical Society , 363(10):5577–5604, 2011.
Richard Dearden, Nir Friedman, and Stuart Russell. Bayesian Q-learning. In AAAI,
pages 761–768, 1998.
DanielDewey. Learningwhattovalue. In Artiﬁcial General Intelligence , pages309–314.
Springer, 2011.
Joseph L. Doob. Stochastic Processes . Wiley, New York, 1953.
Finale Doshi-Velez. Bayesian Nonparametric Approaches for Reinforcement Learning
in Partially Observable Domains . PhD thesis, Massachusetts Institute of Technology,
2012.
Rick Durrett. Probability: Theory and Examples . Cambridge University Press, 4th
edition, 2010.
Amnon H Eden. The singularity controversy. Technical report, Sapience Project, 2016.
http://arxiv.org/abs/1601.05977 .
Amnon H Eden, James H Moor, Johnny H Søraker, and Eric Steinhart, editors. Sin-
gularity Hypotheses: A Scientiﬁc and Philosophical Assessment . Springer, 2013.
Tom Everitt and Marcus Hutter. Avoiding wireheading with value reinforcement learn-
ing. InArtiﬁcial General Intelligence , pages 12–22, 2016.
Tom Everitt, Jan Leike, and Marcus Hutter. Sequential extensions of causal and ev-
idential decision theory. In Algorithmic Decision Theory , pages 205–221. Springer,
2015.
Tom Everitt, Daniel Filan, Mayank Daswani, and Marcus Hutter. Self-modiﬁcation of
policy and utility function in rational agents. In Artiﬁcial General Intelligence , 2016.
Benja Fallenstein, Nate Soares, and Jessica Taylor. Reﬂective variants of Solomonoﬀ
induction and AIXI. In Artiﬁcial General Intelligence . Springer, 2015a.
Benja Fallenstein, Jessica Taylor, and Paul F Christiano. Reﬂective oracles: A founda-
tion for game theory in artiﬁcial intelligence. In Logic, Rationality, and Interaction ,
pages 411–415. Springer, 2015b.
Benja Fallenstein, Jessica Taylor, and Paul F Christiano. Reﬂective oracles: A foun-
dation for classical game theory. Technical report, Machine Intelligence Research
Institute, 2015c. http://arxiv.org/abs/1508.04145 .
DanielFilan. Agentsusingspeedpriors. Honoursthesis, AustralianNationalUniversity,
October 2015.
Daniel Filan, Jan Leike, and Marcus Hutter. Loss bounds and time complexity for
speed priors. In Artiﬁcial Intelligence and Statistics , 2016.
158 BIBLIOGRAPHY
Jakob N Foerster, Yannis M Assael, Nando de Freitas, and Shimon Whiteson. Learning
to communicate to solve riddles with deep distributed recurrent Q-networks. Tech-
nical report, University of Oxford, 2016. http://arxiv.org/abs/1602.02672 .
Dean P Foster and H Peyton Young. On the impossibility of predicting the behavior
of rational agents. Proceedings of the National Academy of Sciences , 98(22):12848–
12853, 2001.
Drew Fudenberg and David K Levine. The Theory of Learning in Games . MIT press,
1998.
Future of Life Institute. Elon musk donates $10m to keep AI beneﬁcial. http://
futureoflife.org/misc/AI , January 2015a. Accessed: 2015-03-28.
Future of Life Institute. 2015 project grants recommended for funding. http:
//futureoflife.org/first-ai-grant-recipients/ , 2015b. Accessed: 2016-03-28.
Future of Life Institute. Research priorities for robust and beneﬁcial artiﬁcial intel-
ligence: An open letter. http://futureoflife.org/ai-open-letter/ , January
2015c. Accessed: 2016-04-26.
John Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal
Statistical Society. Series B (Methodological) , pages 148–177, 1979.
Irving John Good. The paradox of conﬁrmation. British Journal for the Philosophy of
Science, pages 145–149, 1960.
Irving John Good. Speculations concerning the ﬁrst ultraintelligent machine. Advances
in Computers , 6:31–88, 1965.
Irving John Good. The white shoe is a red herring. The British Journal for the
Philosophy of Science , 17(4):322–322, 1967.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. Book in prepa-
ration for MIT Press, http://www.deeplearningbook.org , 2016.
Google. The latest chapter for the self-driving car: mastering
city street driving. http://googleblog.blogspot.com.au/2014/04/
the-latest-chapter-for-self-driving-car.html , April 2014. Accessed:
2015-03-27.
Google. What we learned in Seoul with AlphaGo. https://googleblog.blogspot.
com.au/2016/03/what-we-learned-in-seoul-with-alphago.html , March 2016.
Accessed: 2016-03-28.
Aditya Gopalan and Shie Mannor. Thompson sampling for learning parameterized
Markov decision processes. In Conference on Learning Theory , pages 861–898, 2015.
Geoﬀrey J Gordon. Reinforcement learning with function approximation converges to
a region. In Advanced in Neural Information Processing Systems , 2001.
BIBLIOGRAPHY 159
Peter D. Grünwald. The Minimum Description Length Principle . The MIT Press,
Cambridge, 2007.
Péter Gács. On the relation between descriptional complexity and algorithmic proba-
bility.Theoretical Computer Science , 22(1):71–93, 1983.
Kurt Gödel. Über formal unentscheidbare Sätze der Principia Mathematica und ver-
wandter Systeme I. Monatshefte für Mathematik und Physik , 38(1):173–198, 1931.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical
Learning . Springer, 2nd edition, 2009.
Matthew Hausknecht and Peter Stone. Deep recurrent Q-learning for partially observ-
able MDPs. In 2015 AAAI Fall Symposium Series , 2015.
Stephen Hawking, Max Tegmark, Stuart Russell, and Frank Wilczek. Transcend-
ing complacency on superintelligent machines. http://www.huffingtonpost.com/
stephen-hawking/artificial-intelligence_b_5174265.html , April 2014. Ac-
cessed: 2015-03-28.
Nicholas J Hay. Universal semimeasures: An introduction. Master’s thesis, University
of Auckland, 2007.
Nicolas Heess, Jonathan J Hunt, Timothy P Lillicrap, and David Silver. Memory-based
control with recurrent neural networks. Technical report, Google DeepMind, 2015.
http://arxiv.org/abs/1512.04455 .
Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in
imperfect-information games. Technical report, DeepMind, 2016. http://arxiv.
org/abs/1603.01121 .
Matthias Heizmann, Daniel Dietsch, Jan Leike, Betim Musa, and Andreas Podelski.
UltimateAutomizerwitharrayinterpolation(competitioncontribution). In Tools and
Algorithms for the Construction and Analysis of Systems , pages 455–457. Springer,
2015.
Matthias Heizmann, Daniel Dietsch, Marius Greitschus, Jan Leike, Betim Musa, Claus
Schätzle, and Andreas Podelski. Ultimate Automizer with two-track proofs (compe-
tition contribution). In Tools and Algorithms for the Construction and Analysis of
Systems, pages 950–953. Springer, 2016.
Carl G Hempel. Studies in the logic of conﬁrmation (I.). Mind, pages 1–26, 1945.
Carl G Hempel. The white shoe: No red herring. The British Journal for the Philosophy
of Science , 18(3):239–240, 1967.
Marcus Hutter. A theory of universal artiﬁcial intelligence based on algorithmic com-
plexity. Technical report, 2000. http://arxiv.org/abs/cs.AI/0004001 .
160 BIBLIOGRAPHY
Marcus Hutter. Universal sequential decisions in unknown environments. In European
Workshop on Reinforcement Learning , pages 25–26, 2001a.
Marcus Hutter. New error bounds for Solomonoﬀ prediction. Journal of Computer and
System Sciences , 62(4):653–667, 2001b.
Marcus Hutter. Self-optimizing and Pareto-optimal policies in general environments
based on Bayes-mixtures. In Computational Learning Theory , pages 364–379.
Springer, 2002a.
Marcus Hutter. The fastest and shortest algorithm for all well-deﬁned problems. In-
ternational Journal of Foundations of Computer Science , 13(03):431–443, 2002b.
MarcusHutter. AgentleintroductiontotheuniversalalgorithmicagentAIXI. Technical
report, IDSIA, 2003. ftp://ftp.idsia.ch/pub/techrep/IDSIA-01-03.ps.gz .
Marcus Hutter. Universal Artiﬁcial Intelligence . Springer, 2005.
Marcus Hutter. Sequential predictions based on algorithmic complexity. Journal of
Computer and System Sciences , 72(1):95–117, 2006a.
Marcus Hutter. General discounting versus average reward. In Algorithmic Learning
Theory, pages 244–258. Springer, 2006b.
Marcus Hutter. 50’000 €prize for compressing human knowledge. http://prize.
hutter1.net/ , 2006c. Accessed: 2015-03-29.
Marcus Hutter. Universal algorithmic intelligence: A mathematical top !down ap-
proach. In Artiﬁcial General Intelligence , pages 227–290. Springer, 2007a.
Marcus Hutter. On universal prediction and Bayesian conﬁrmation. Theoretical Com-
puter Science , 384(1):33–48, 2007b.
Marcus Hutter. Discrete MDL predicts in total variation. In Advances in Neural
Information Processing Systems , pages 817–825, 2009a.
Marcus Hutter. Open problems in universal induction & intelligence. Algorithms , 3(2):
879–906, 2009b.
Marcus Hutter. Feature dynamic Bayesian networks. In Artiﬁcial General Intelligence ,
pages 67–73. Atlantis Press, 2009c.
Marcus Hutter. Feature reinforcement learning: Part I: Unstructured MDPs. Journal
of Artiﬁcial General Intelligence , 1:3–24, 2009d.
Marcus Hutter. Can intelligence explode? Journal of Consciousness Studies , 19(1–2):
143–166, 2012a.
Marcus Hutter. One decade of universal artiﬁcial intelligence. In Theoretical Founda-
tions of Artiﬁcial General Intelligence , pages 67–88. Springer, 2012b.
BIBLIOGRAPHY 161
Marcus Hutter. Extreme state aggregation beyond MDPs. In Algorithmic Learning
Theory. Springer, 2014.
Marcus Hutter and Andrej A. Muchnik. On semimeasures predicting Martin-Löf ran-
dom sequences. Theoretical Computer Science , 382(3):247–261, 2007.
IBM. Deep Blue. http://www-03.ibm.com/ibm/history/ibm100/us/en/icons/
deepblue/ , March 2012a. Accessed: 2015-03-27.
IBM. A computer called Watson. http://www-03.ibm.com/ibm/history/ibm100/us/
en/icons/watson/ , March 2012b. Accessed: 2015-03-27.
Edwin T Jaynes. Probability Theory: The Logic of Science . Cambridge University
Press, 2003.
Sham Machandranath Kakade. On the Sample Complexity of Reinforcement Learning .
PhD thesis, University College London, 2003.
Ehud Kalai and Ehud Lehrer. Rational learning leads to Nash equilibrium. Economet-
rica, pages 1019–1045, 1993.
Ehud Kalai and Ehud Lehrer. Weak and strong merging of opinions. Journal of Math-
ematical Economics , 23(1):73–86, 1994.
EmilieKaufmann, NathanielKorda, andRémiMunos. Thompsonsampling: Anasymp-
toticallyoptimalﬁnite-timeanalysis. In Algorithmic Learning Theory , pages199–213.
Springer, 2012.
HJ Kim, Michael I Jordan, Shankar Sastry, and Andrew Y Ng. Autonomous helicopter
ﬂight via reinforcement learning. In Advances in Neural Information Processing Sys-
tems, page None, 2003.
Stephen Cole Kleene. Introduction to Metamathematics . Wolters-Noordhoﬀ Publishing,
1952.
Tejas D Kulkarni, Karthik R Narasimhan, Ardavan Saeedi, and Joshua B Tenenbaum.
Hierarchical deep reinforcement learning: Integrating temporal abstraction and in-
trinsic motivation. Technical report, Massachusetts Institute of Technology, 2016.
https://arxiv.org/abs/1604.06057 .
Ray Kurzweil. The Singularity is Near: When Humans Transcend Biology . Viking
Books, 2005.
Tor Lattimore. Theory of General Reinforcement Learning . PhD thesis, Australian
National University, 2013.
Tor Lattimore. Regret analysis of the ﬁnite-horizon Gittins index strategy for multi-
armed bandits. In Conference on Learning Theory , 2016.
162 BIBLIOGRAPHY
Tor Lattimore and Marcus Hutter. Asymptotically optimal agents. In Algorithmic
Learning Theory , pages 368–382. Springer, 2011.
Tor Lattimore and Marcus Hutter. PAC bounds for discounted MDPs. In Algorithmic
Learning Theory , pages 320–334. Springer, 2012.
TorLattimoreandMarcusHutter. OnMartin-LöfconvergenceofSolomonoﬀ’smixture.
InTheory and Applications of Models of Computation , volume 7876, pages 212–223.
Springer, 2013.
Tor Lattimore and Marcus Hutter. General time consistent discounting. Theoretical
Computer Science , 519:140–154, 2014.
Tor Lattimore and Marcus Hutter. On Martin-Löf (non-)convergence of Solomonoﬀ’s
universal mixture. Theoretical Computer Science , 588:2–15, 2015.
Tor Lattimore, Marcus Hutter, and Vaibhav Gavane. Universal prediction of selected
bits. InAlgorithmic Learning Theory , pages 262–276. Springer, 2011.
Neil Lawrence. Future of AI 6. Discussion of ‘Superintelligence: Paths,
Dangers, Strategies’. http://inverseprobability.com/2016/05/09/
machine-learning-futures-6 , May 2016. Accessed: 2016-05-10.
Yann LeCun, Yoshua Bengio, and Geoﬀrey Hinton. Deep learning. Nature, 521(7553):
436–444, 2015.
Shane Legg. Is there an elegant universal theory of prediction? In Algorithmic Learning
Theory, pages 274–287, 2006.
Shane Legg. Machine Super Intelligence . PhD thesis, University of Lugano, 2008.
Shane Legg and Marcus Hutter. A collection of deﬁnitions of intelligence. Frontiers in
Artiﬁcial Intelligence and Applications , 157:17–24, 2007a.
Shane Legg and Marcus Hutter. Universal intelligence: A deﬁnition of machine intelli-
gence.Minds & Machines , 17(4):391–444, 2007b.
Shane Legg and Joel Veness. An approximation of the universal intelligence measure. In
Algorithmic Probability and Friends. Bayesian Prediction and Artiﬁcial Intelligence ,
pages 236–249. Springer, 2013.
Ehud Lehrer and Rann Smorodinsky. Merging and learning. Statistics, Probability and
Game Theory , pages 147–168, 1996.
Jan Leike and Matthias Heizmann. Ranking templates for linear loops. In Tools and
Algorithms for the Construction and Analysis of Systems , pages 172–186. Springer,
2014a.
Jan Leike and Matthias Heizmann. Geometric series as nontermination arguments
for linear lasso programs. Technical report, University of Freiburg, 2014b. http:
//arxiv.org/abs/1405.4413 .
BIBLIOGRAPHY 163
Jan Leike and Matthias Heizmann. Ranking templates for linear loops. Logical Methods
in Computer Science , 11(1):1–27, March 2015.
Jan Leike and Matthias Heizmann. Geometric nontermination arguments. 2016. Under
preparation.
Jan Leike and Marcus Hutter. Indeﬁnitely oscillating martingales. In Algorithmic
Learning Theory , pages 321–335, 2014a.
Jan Leike and Marcus Hutter. Indeﬁnitely oscillating martingales. Technical report,
Australian National University, 2014b. http://arxiv.org/abs/1408.3169 .
JanLeikeandMarcusHutter. OnthecomputabilityofAIXI. In Uncertainty in Artiﬁcial
Intelligence , pages 464–473, 2015a.
Jan Leike and Marcus Hutter. On the computability of Solomonoﬀ induction and
knowledge-seeking. In Algorithmic Learning Theory , pages 364–378, 2015b.
Jan Leike and Marcus Hutter. Bad universal priors and notions of optimality. In
Conference on Learning Theory , pages 1244–1259, 2015c.
Jan Leike and Marcus Hutter. Solomonoﬀ induction violates Nicod’s criterion. In
Algorithmic Learning Theory , pages 349–363. Springer, 2015d.
Jan Leike and Marcus Hutter. On the computability of Solomonoﬀ induction and AIXI.
2016. Under review.
Jan Leike, Tor Lattimore, Laurent Orseau, and Marcus Hutter. Thompson sampling is
asymptotically optimal in general environments. In Uncertainty in Artiﬁcial Intelli-
gence, 2016a.
Jan Leike, Jessica Taylor, and Benya Fallenstein. A formal solution to the grain of
truth problem. In Uncertainty in Artiﬁcial Intelligence , 2016b.
Leonid A Levin. On the notion of a random sequence. Soviet Mathematics Doklady , 14
(5):1413–1416, 1973.
Ming Li and Paul M. B. Vitányi. An Introduction to Kolmogorov Complexity and Its
Applications . Texts in Computer Science. Springer, 3rd edition, 2008.
Yitao Liang, Marlos C Machado, Erik Talvitie, and Michael Bowling. State of the art
control of Atari games using shallow reinforcement learning. In Autonomous Agents
and Multiagent Systems , 2016.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez,
Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep rein-
forcement learning. In International Conference on Learning Representations , 2016.
John L Mackie. The paradox of conﬁrmation. British Journal for the Philosophy of
Science, pages 265–277, 1963.
164 BIBLIOGRAPHY
Omid Madani, Steve Hanks, and Anne Condon. On the undecidability of probabilis-
tic planning and inﬁnite-horizon partially observable Markov decision problems. In
AAAI, pages 541–548, 1999.
Omid Madani, Steve Hanks, and Anne Condon. On the undecidability of probabilistic
planning and related stochastic optimization problems. Artiﬁcial Intelligence , 147
(1):5–34, 2003.
Sridhar Mahadevan. Optimality criteria in reinforcement learning. In AAAI Fall Sym-
posium on Learning Complex Behaviors in Adaptive Intelligent Systems , 1996.
Patrick Maher. Inductive logic and the ravens paradox. Philosophy of Science , pages
50–70, 1999.
A Rupam Mahmood, Huizhen Yu, Martha White, and Richard Sutton. Emphatic
temporal-diﬀerence learning. Technical report, University of Alberta, 2015. http:
//arxiv.org/abs/1507.01569 .
Norman Margolus and Lev B Levitin. The maximum speed of dynamical evolution.
Physica D: Nonlinear Phenomena , 120(1):188–195, 1998.
Jarryd Martin, Tom Everitt, and Marcus Hutter. Death and suicide in universal artiﬁ-
cial intelligence. In Artiﬁcial General Intelligence , 2016.
JohnMcCarthy, MarvinMinsky, NathanielRochester, andClaudeShannon. Aproposal
for the Dartmouth summer research project on artiﬁcial intelligence. 1955.
Ronald I Miller and Chris William Sanchirico. The role of absolute continuity in “merg-
ing of opinions” and “rational learning”. Games and Economic Behavior , 29(1):170–
190, 1999.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou,
Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learn-
ing. Technical report, Google DeepMind, 2013. http://arxiv.org/abs/1312.5602 .
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness,
Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Os-
trovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King,
Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level
control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P
Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods
for deep reinforcement learning. In International Conference on Machine Learning ,
2016.
Luke Muehlhauser and Anna Salamon. Intelligence explosion: Evidence and import.
InSingularity Hypotheses , pages 15–42. Springer, 2012.
BIBLIOGRAPHY 165
Martin Mundhenk, Judy Goldsmith, Christopher Lusena, and Eric Allender. Complex-
ity of ﬁnite-horizon Markov decision process problems. Journal of the ACM , 47(4):
681–720, 2000.
Markus Müller. Stationary algorithmic probability. Theoretical Computer Science , 411
(1):113–130, 2010.
Vincent C Müller and Nick Bostrom. Future progress in artiﬁcial intelligence: A survey
of expert opinion. Fundamental Issues of Artiﬁcial Intelligence , pages 553–571, 2016.
John H Nachbar. Prediction, optimization, and learning in repeated games. Economet-
rica, 65(2):275–309, 1997.
John H Nachbar. Beliefs in repeated games. Econometrica , 73(2):459–480, 2005.
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessan-
dro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig
Petersen, Shane Legg, Volodymyr Mnih, Koray Kavukcuoglu, and David Silver. Mas-
sively parallel methods for deep reinforcement learning. Technical report, Google
DeepMind, 2015. http://arxiv.org/abs/1507.04296 .
Andrew Ng. Is A.I. an existential threat to humanity? https://www.quora.com/
Is-A-I-an-existential-threat-to-humanity/answer/Andrew-Ng , January 2016.
Accessed: 2016-05-19.
Phuong Nguyen, Odalric-Ambrym Maillard, Daniil Ryabko, and Ronald Ortner. Com-
peting with an inﬁnite set of models in reinforcement learning. In Artiﬁcial Intelli-
gence and Statistics , pages 463–471, 2013.
Jean Nicod. Le Problème Logique de L’Induction . Presses Universitaires de France,
1961.
André Nies. Computability and Randomness . Oxford University Press, 2009.
James Olds and Peter Milner. Positive reinforcement produced by electrical stimulation
ofseptalareaandotherregionsofratbrain. Journal of Comparative and Physiological
Psychology , 47(6):419, 1954.
Stephen M Omohundro. The basic AI drives. In Artiﬁcial General Intelligence , pages
483–492, 2008.
OpenAI. About OpenAI. https://openai.com/about/ , December 2015. Accessed:
2016-04-27.
Laurent Orseau. Optimality issues of universal greedy agents with static priors. In
Algorithmic Learning Theory , pages 345–359. Springer, 2010.
Laurent Orseau. Universal knowledge-seeking agents. In Algorithmic Learning Theory ,
pages 353–367. Springer, 2011.
166 BIBLIOGRAPHY
Laurent Orseau. Asymptotic non-learnability of universal agents with computable hori-
zon functions. Theoretical Computer Science , 473:149–156, 2013.
Laurent Orseau. Universal knowledge-seeking agents. Theoretical Computer Science ,
519:127–139, 2014a.
Laurent Orseau. The multi-slot framework: A formal model for multiple, copiable AIs.
InArtiﬁcial General Intelligence , pages 97–108. Springer, 2014b.
Laurent Orseau. Teleporting universal intelligent agents. In Artiﬁcial General Intelli-
gence, pages 109–120. Springer, 2014c.
Laurent Orseau and Stuart Armstrong. Safely interruptible agents. In Uncertainty in
Artiﬁcial Intelligence , pages 557–566, 2016.
Laurent Orseau and Mark Ring. Self-modiﬁcation and mortality in artiﬁcial agents. In
Artiﬁcial General Intelligence , pages 1–10. Springer, 2011.
LaurentOrseauandMarkRing. Space-timeembeddedintelligence. In Artiﬁcial General
Intelligence , pages 209–218. Springer, 2012a.
Laurent Orseau and Mark Ring. Memory issues of intelligent agents. In Artiﬁcial
General Intelligence , pages 219–231. Springer, 2012b.
Laurent Orseau, Tor Lattimore, and Marcus Hutter. Universal knowledge-seeking
agents for stochastic environments. In Algorithmic Learning Theory , pages 158–172.
Springer, 2013.
Pedro A Ortega and Daniel A Braun. A minimum relative entropy principle for learning
and acting. Journal of Artiﬁcial Intelligence Research , pages 475–511, 2010.
Pedro A Ortega and Daniel A Braun. Generalized Thompson sampling for sequential
decision-making and causal inference. Complex Adaptive Systems Modeling , 2(1):2,
2014.
Ian Osband, Dan Russo, and Benjamin van Roy. (More) eﬃcient reinforcement learning
via posterior sampling. In Neural Information Processing Systems , pages 3003–3011,
2013.
Christos H Papadimitriou and John N Tsitsiklis. The complexity of Markov decision
processes. Mathematics of Operations Research , 12(3):441–450, 1987.
Martin L Puterman. Markov Decision Processes . John Wiley & Sons, 2014.
SamuelRathmannerandMarcusHutter. Aphilosophicaltreatiseofuniversalinduction.
Entropy, 13(6):1076–1136, 2011.
Mark Ring and Laurent Orseau. Delusion, survival, and intelligent agents. In Artiﬁcial
General Intelligence , pages 11–20. Springer, 2011.
BIBLIOGRAPHY 167
Chris Rogers and David Williams. Diﬀusions, Markov Processes, and Martingales:
Volume 1, Foundations . Cambridge University Press, 2nd edition, 1994.
Stuart Russell, Daniel Dewey, and Max Tegmark. Research priorities for robust and
beneﬁcialartiﬁcialintelligence. Technicalreport, FutureofLifeInstitute, 2015. http:
//arxiv.org/abs/1602.03506 .
Stuart J Russell and Peter Norvig. Artiﬁcial Intelligence. A Modern Approach . Prentice
Hall, 3rd edition, 2010.
Daniil Ryabko. Characterizing predictable classes of processes. In Uncertainty in Arti-
ﬁcial Intelligence , pages 471–478, 2009.
Daniil Ryabko. On ﬁnding predictors for arbitrary families of processes. The Journal
of Machine Learning Research , 11:581–602, 2010.
Daniil Ryabko. On the relation between realizable and nonrealizable cases of the se-
quence prediction problem. The Journal of Machine Learning Research , 12:2161–
2180, 2011.
Daniil Ryabko and Marcus Hutter. On sequence prediction for arbitrary measures. In
IEEE International Symposium on Information Theory , pages 2346–2350, 2007.
Daniil Ryabko and Marcus Hutter. Predicting non-stationary processes. Applied Math-
ematics Letters , 21(5):477–482, 2008.
Régis Sabbadin, Jérôme Lang, and Nasolo Ravoanjanahry. Purely epistemic Markov
decision processes. In AAAI, pages 1057–1062, 2007.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience
replay. In International Conference on Learning Representations , 2016.
Jürgen Schmidhuber. The speed prior: A new simplicity measure yielding near-optimal
computablepredictions. In Computational Learning Theory , pages216–228.Springer,
2002.
Jürgen Schmidhuber. Philosophers & futurists, catch up! Journal of Consciousness
Studies, 19(1-2):173–182, 2012.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks ,
61:85–117, 2015.
Murray Shanahan. The Technological Singularity . MIT Press, 2015.
ClaudeEShannon. Programmingacomputerforplayingchess. The London, Edinburgh,
and Dublin Philosophical Magazine and Journal of Science , 41(314):256–275, 1950.
Joseph R Shoenﬁeld. Mathematical Logic . Addison-Wesley Reading, 1967.
Yoav Shoham and Kevin Leyton-Brown. Multiagent Systems: Algorithmic, Game-
Theoretic, and Logical Foundations . Cambridge University Press, 2009.
168 BIBLIOGRAPHY
DavidSilver,AjaHuang,ChrisJMaddison,ArthurGuez,LaurentSifre,Georgevanden
Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc
Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya
Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel,
and Demis Hassabis. Mastering the game of Go with deep neural networks and tree
search.Nature, 529(7587):484–489, 2016.
Satinder Singh, Michael L Littman, Nicholas K Jong, David Pardoe, and Peter Stone.
Learning predictive state representations. In International Conference on Machine
Learning , pages 712–719, 2003.
Satinder Singh, Michael R James, and Matthew R Rudary. Predictive state represen-
tations: A new theory for modeling dynamical systems. In Uncertainty in Artiﬁcial
Intelligence , pages 512–519, 2004.
Nate Soares. Formalizing two problems of realistic world-models. Technical report,
Machine Intelligence Research Institute, 2015. http://intelligence.org/files/
RealisticWorldModels.pdf .
Nate Soares and Benja Fallenstein. Aligning superintelligence with human interests: A
technical research agenda. Technical report, Machine Intelligence Research Institute,
2014. http://intelligence.org/files/TechnicalAgenda.pdf .
Ray Solomonoﬀ. A formal theory of inductive inference. Parts 1 and 2. Information
and Control , 7(1):1–22 and 224–254, 1964.
Ray Solomonoﬀ. Complexity-based induction systems: Comparisons and convergence
theorems. IEEE Transactions on Information Theory , 24(4):422–432, 1978.
Kaj Sotala and Roman V Yampolskiy. Responses to catastrophic AGI risk: A survey.
Physica Scripta , 90(1):018001, 2014.
Tom F Sterkenburg. Putnam’s diagonal argument and the impossibility of a universal
learning machine. Technical report, Centrum Wiskunde & Informatica, 2016. http:
//philsci-archive.pitt.edu/12096/ .
Jordan M Stoyanov. Counterexamples in Probability . Courier Corporation, 3rd edition,
2013.
Malcolm Strens. A Bayesian framework for reinforcement learning. In International
Conference on Machine Learning , pages 943–950, 2000.
Peter Sunehag and Marcus Hutter. Consistency of feature Markov processes. In Algo-
rithmic Learning Theory , pages 360–374. Springer, 2010.
Peter Sunehag and Marcus Hutter. Optimistic agents are asymptotically optimal. In
Australasian Joint Conference on Artiﬁcial Intelligence , pages15–26.Springer, 2012a.
BIBLIOGRAPHY 169
Peter Sunehag and Marcus Hutter. Optimistic AIXI. In Artiﬁcial General Intelligence ,
pages 312–321. Springer, 2012b.
Peter Sunehag and Marcus Hutter. Rationality, optimism and guarantees in general
reinforcement learning. Journal of Machine Learning Research , 16:1345–1390, 2015.
Richard Sutton. Learning to predict by the methods of temporal diﬀerences. Machine
Learning , 3(1):9–44, 1988.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction .
MIT Press, 1998.
Richard G Swinburne. The paradoxes of conﬁrmation: A survey. American Philosoph-
ical Quarterly , pages 318–330, 1971.
Csaba Szepesvári. Algorithms for Reinforcement Learning . Morgan & Claypool Pub-
lishers, 2010.
Gerald Tesauro. Temporal diﬀerence learning and TD-Gammon. Communications of
the ACM , 38(3):58–68, 1995.
William R Thompson. On the likelihood that one unknown probability exceeds another
in view of the evidence of two samples. Biometrika , pages 285–294, 1933.
John N Tsitsiklis and Benjamin Van Roy. An analysis of temporal-diﬀerence learning
with function approximation. IEEE Transactions on Automatic Control , 42(5):674–
690, 1997.
Alexandre Tsybakov. Introduction to Nonparametric Estimation . Springer, 2008.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with
double Q-learning. In AAAI, 2016.
Joel Veness, Kee Siong Ng, Marcus Hutter, William Uther, and David Silver. A Monte-
Carlo AIXI approximation. Journal of Artiﬁcial Intelligence Research , 40(1):95–142,
2011.
Joel Veness, Marc G Bellemare, Marcus Hutter, Alvin Chua, and Guillaume Desjardins.
Compress and control. In AAAI, 2015.
Vernor Vinge. The coming technological singularity. Vision 21: Interdisciplinary Sci-
ence and Engineering in the Era of Cyberspace , 1:11–22, 1993. http://www.rohan.
sdsu.edu/faculty/vinge/misc/singularity.html .
Paul MB Vitányi, Frank J Balbach, Rudi L Cilibrasi, and Ming Li. Normalized in-
formation distance. In Information Theory and Statistical Learning , pages 45–82.
Springer, 2009.
Nikos Vlassis, Mohammad Ghavamzadeh, Shie Mannor, and Pascal Poupart. Bayesian
reinforcement learning. In Marco Wiering and Martijn van Otterlo, editors, Rein-
forcement Learning , pages 359–386. Springer, 2012.
170 BIBLIOGRAPHY
PeterBMVranas. Hempel’sravenparadox: AlacunainthestandardBayesiansolution.
The British Journal for the Philosophy of Science , 55(3):545–560, 2004.
Toby Walsh. The singularity may never be near. Technical report, University of New
South Wales, 2016. http://arxiv.org/abs/1602.06462 .
Ziyu Wang, Nando de Freitas, Tom Schaul, Matteo Hessel, Hado van Hasselt, and
Marc Lanctot. Dueling network architectures for deep reinforcement learning. In
International Conference on Machine Learning , 2016.
Larry Wasserman. All of Statistics . Springer, 2004.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning , 8(3-4):
279–292, 1992.
Eric W Weisstein. Random walk—1-dimensional. In MathWorld—A Wolfram
Web Resource . Wolfram Research, Inc., 2002. http://mathworld.wolfram.com/
RandomWalk1-Dimensional.html .
Marco Wiering and Martijn van Otterlo, editors. Reinforcement Learning . Springer,
2012.
Ian Wood, Peter Sunehag, and Marcus Hutter. (Non-)equivalence of universal priors.
InSolomonoﬀ 85th Memorial Conference , pages 417–425. Springer, 2011.
Huizhen Yu. On convergence of emphatic temporal-diﬀerence learning. In Conference
on Learning Theory , pages 1724–1751, 2015.
Eliezer Yudkowsky. Creating friendly AI 1.0: The analysis and design of benevolent
goal architectures. Technical report, Singularity Institute for Artiﬁcial Intelligence,
2001.
Eliezer Yudkowsky. Artiﬁcial intelligence as a positive and negative factor in global
risk. InGlobal Catastrophic Risks , pages 308–345. Oxford University Press, 2008.
Tom Zahavy, Nir Ben Zrihem, and Shie Mannor. Graying the black box: Understanding
DQNs. Technical report, Israel Institute of Technology, 2016. http://arxiv.org/
abs/1602.02658 .
JacobZivandAbrahamLempel. Auniversalalgorithmforsequentialdatacompression.
IEEE Transactions on information theory , 23(3):337–343, 1977.
List of Notation
Abbreviations
AIXI Bayesian RL agent with a Solomonoﬀ prior, see Section 4.3.1
AI artiﬁcial intelligence
HLAI human-level artiﬁcial intelligence
MDL minimum description length, see Example 3.14
MDP Markov decision process, see Section 4.1.3
POMDP partially observable Markov decision process, see Section 4.1.3
RL reinforcement learning
UTM universal Turing machine, Section 2.4
Mathematical notation
:= deﬁned to be equal
:2 deﬁned to be an element of
A,B,
 sets
#A the cardinality of the set A, i.e., the number of elements

 the set of probability distributions over a ﬁnite or countable set
1x the characteristic function that is 1forxand0otherwise.
f,g functions
fg there is a constant c>0such thatfcg
f=g fgandgf
N the set of natural numbers, starting with 1
Q the set of rational numbers
R the set of real numbers
n,k,t,m,i,jnatural numbers
t (current) time step, t2N
k some other time step, k2N
q;q0rational numbers
r real number
X a ﬁnite nonempty alphabet
Xthe set of all ﬁnite strings over the alphabet X
X1the set of all inﬁnite strings over the alphabet X
X]X]:=X[X1, the set of all ﬁnite and inﬁnite strings over the
alphabetX
x;y;z (typically ﬁnite) strings from X]
x<t the ﬁrstt1symbols of the string x
171
172 List of Notation
xvy the stringxis a preﬁx of the string y
zeros(x) the number of zeros in the binary string x2f0;1g
ones(x) the number of ones in the binary string x2f0;1g
, computable functions
' formula of ﬁrst-order logic
 computable relation/quantiﬁer-free formula
T a Turing machine
p;p0programs on a universal Turing machine in the form of ﬁnite binary
strings
jpj length of the program pin bits
K the Kolmogorov complexity of a string or a semimeasure
Km the monotone Kolmogorov complexity of a string
Ent entropy
KLm KL-divergence
Dm total variation distance
IG information gain
F expected total variation distance
F,Ft,F1-algebras
x the cylinder set of all strings starting with x
A,H,E measurable sets
X,Y real-valued random variables
P a distribution over X1, thetruedistribution
Q adistributionover X1, thelearningalgorithmorbeliefdistribution
Bernoulli (r)a Bernoulli process with parameter r
 the uniform measure or Lebesgue measure
L Laplace rule
M Solomonoﬀ’s prior
M the measure mixture
SKt the speed prior
 a semimeasure
norm the Solomonoﬀ normalization of the semimeasure
 absolute continuity
W weak dominance
L local absolute continuity
A the ﬁnite set of possible actions
O the ﬁnite set of possible observations
E the ﬁnite set of possible percepts, EO R
; two diﬀerent actions, ;2A
at the action in time step t
ot the observation in time step t
rt the reward in time step t, bounded between 0and1
et the percept in time step t, we useet= (ot;rt)implicitly
æ<t the ﬁrstt1interactions, a1e1a2e2:::at1et1(a history of length
t1)
List of Notation 173
h a history,h2(AE )
 the history of length 0
", small positive real numbers

 the discount function
:N!R0, deﬁned in Deﬁnition 4.3
t a discount normalization factor, t:=P1
k=t
(k)
m horizon of the agent (how many steps it plans ahead)
Ht(") an"-eﬀective horizon
;; environments
;~ policies,;~: (AE )!A

 an optimal policy for environment
the history distribution generated by policy in environment
E
 the expectation with respect to the history distribution
V
 the-expected value of the policy
V
 the optimal value in environment
W
 the iterative value of the policy in environment
W
 the optimal iterative value in environment Pmax the max-sum-operator
Rm(;) regret of policy in environment for horizon m
() the Legg-Hutter intelligence of policy measured in the universal
mixture
 the minimal Legg-Hutter intelligence measured in the universal
mixture
 the maximal Legg-Hutter intelligence measured in the universal
mixture
M a class of environments
MCCSthe class of all chronological contextual semimeasures
MCCS
LSC the class of all lower semicomputable chronological contextual
semimeasures
MCCM
comp the class of all computable chronological contextual measures
MO
re
the class of all reﬂective-oracle-computable environments
U reference universal Turing machine
U0a ‘bad’ universal Turing machine
w a positive prior over the environment class
w0a ‘bad’ positive prior over the environment class
 the universal mixture over all environments MCCS
LSCgiven by the
reference UTM U
0a ‘bad’ universal mixture over all environments MCCS
LSCgiven by the
‘bad’ UTM U0
T the set of all probabilistic Turing machines
O an oracle
~O a partial oracle
T the semimeasure generated by Turing machine T
O
T the semimeasure generated by Turing machine Trunning with or-
acleO
174 List of Notation
O
T the completion of O
Tinto a measure using oracle O
 a multi-agent environment
i the subjective environment of agent iacting in multi-agent envi-
ronment
Index
absolute continuity, 25, 26, 26, 28–33,
38–40
local, 25,28, 28–31, 33
action, 2, 50, 53
adversarial
environment, 112
sequence, 36
agent, 2, 49, 51
agent model
dualistic, 50, 52
physicalistic, 52
AI
human-level, 1
narrow, 1
strong, 1
AI safety, 148
AIMU, 57, 111
AINU, 57, 110, 112, 116, 118, 119
AIXI, 7,62, 69, 78, 81, 96, 97, 111, 114,
120
AIXItl, 9, 74, 75
alphabet, 15
anytime algorithm, 101, 104
argmax tie, 57
arithmetical hierarchy, 101, 103, 103
Atari 2600, 4, 13, 54
bandit,seemulti-armed bandit
BayesExp, 65, 65, 84, 122, 123
Bayesian
controlrule, seeThompsonsampling
mixture,26, 45, 63, 136
Bellman equation, 57
black ravens, 25, 30, 32, 47
Carathéodory’s extension theorem, 152
Cesàro average, 18, 79
chain rule, 130compatibility, 24, 25, 31, 45
compression, seeuniversal compression
computable, 7, 16, 16, 103, 104, 106
reﬂective-oracle, 131, 136, 140, 142
conditional expectation, 17
conﬁrmation, 24
Control Problem, 148
convergence
almost surely, 18, 60, 79
in mean,18, 79
in probability, 18, 79
martingale, 19, 30, 31, 84, 86, 142
cylinder set, 17, 21, 29, 51
deep learning, 4
diameter, 3, 90
disconﬁrmation, 24
discount
function, 10, 52, 52, 53, 81, 84, 92
normalization factor, 52
discounting
ﬁnite horizon, 53, 70, 73
geometric, 3, 53, 88, 89, 92, 95
power, 53
subgeometric, 53, 92
dominance, 25, 25, 26, 28, 31, 37, 40
weak, 25, 27, 27, 28, 31, 34, 35, 40
with coeﬃcients, 25, 27, 28, 31
DQN, 4
dualistic model, 52
"-best response, 139, 140–143
eﬀective horizon, 52, 80–83, 86, 89, 91,
98
bounded, 52, 83
entropy,19, 38, 63
relative, seeKL-divergence
environment, 2, 49, 51
175
176 Index
deterministic, 51
environment class, 51, 52, 54
Epicurus’ principle, 26, 41
equivalence condition, 47
ergodic, 3, 6, 7, 55
error
cumulative, 35, 36, 38
instantaneous, 35
estimable in polynomial time, 43, 43
event,17
existential risk, 147
exploration vs. exploitation, 2, 8, 49, 54,
63, 65, 83, 96, 124
falsify, 24
feature reinforcement learning, 7
ﬁcticious play, 129
ﬁltration, 17, 51
function approximation, 3, 5
general reinforcement learning problem,
2, 49,51
Gibb’s inequality, 19
goal, 2, 4, 6, 8, 49
good enough eﬀect, 81
grain of truth, 11, 25, 127
problem, 11, 128, 137
heaven, 76, 78, 91, 103
hell, 76, 78, 91, 103
Hempel’s paradox, seeparadox of con-
ﬁrmation
history, 2, 24, 50
distribution, 51,138
horizon,56
Hutter prize, 44
Hutter search, 75
hypothesis, 24, 25, 30
information gain, 64, 85, 98
intelligence, 4, 8, 75, 76–79
explosion, 147
maximal, 76
minimal, 76
invariance theorem, 42, 97KL-divergence, 19, 20, 34, 36, 38, 40, 64
knowledge-seeking, 6, 50, 63
Kolmogorov complexity, 20
monotone, 20, 43
Laplace rule, 28, 33, 45
learnable, 90
Legg-Hutterintelligence, seeintelligence
limit computable, 101, 103, 123, 140,
142
lower semicomputable, 16, 16, 21, 103
M,seeenvironment class
Markov decision process, seeMDP
martingale, 18, 19, 29–31
matching pennies, 139, 141, 143
MDL,28, 28, 33, 45
MDP, 3,54
measurable
set, 16,17, 17
space,16
measure,21, 30
Bernoulli, 25
compatibility, seecompatibility
deterministic, 17, 37
Lebesgue, 21, 37
mixture,42, 106–108
probability, 17
uniform, seeLebesgue measure
merging, 23, 32, 36, 59
almost weak, 32, 34, 34, 35, 43, 60
of opinions, 32
strong,32, 32–34, 41, 44, 60
weak, 32, 33, 33, 34, 60
model-based, 3
model-free, 3
multi-agentenvironment, 137, 138, 138,
140, 142
multi-armed bandit, 54, 89
Nash equilibrium, 139
subgame perfect, 140
Nicod’s criterion, 47
nonparametric, 7, 52
Index 177
observation, 2, 50
Ockham’s razor, 26, 41
oﬀ-policy, 2, 49, 65, 83, 98
on-policy, 2, 112
value convergence, 49, 59–62
optimality
action,57
asymptotic, 7, 68, 79, 81, 95, 98,
99, 128
in mean, 79, 84, 89, 142
in probability, 79
strong, 79, 81, 82, 88
weak, 79, 81, 84, 89, 123
balanced Pareto, 67, 76, 76
Bayes, 67, 73, 76, 82
Pareto, 67, 69, 69
oracle,130
halting, 101, 103, 132
partial,132, 132–134
partially reﬂective, 133, 133, 134
reﬂective, 129, 130, 130–134
0
n-formula,103
PAC, 3, 68, 79
paradox of conﬁrmation, 47
partially observable, 5–7, 49, 55, 80
Peano arithmetic, 74
percept, 2, 50, 53
physicalistic model, 52
Pinsker’s inequality, 20
planning, 3, 49
policy, 49, 51, 109
Bayes optimal, 62
computable, 73, 77, 79
consistent with history, 51
deterministic, 51, 78, 81
"-optimal,58, 110, 111, 114, 120
optimal,57, 95, 109, 136, 140
POMDP, 6, 55, 89, 91, 92
Post’s theorem, 103
posterior, 24, 30, 62
sampling, seeThompson sampling
power set, 16
prediction, 23expected regret, 35, 36–38, 40, 41,
43, 44
loss, 35
regret,35, 35–37, 39, 40
with expert advice, 23
prior, 24
dogmatic, 10, 70, 71, 73, 97, 141
Gödel, 70, 74, 97
indiﬀerence, 70, 70, 97
positive,26
speed,27, 43, 45
prisoner’s dilemma, 127, 144
program, 20
Q-learning, 2–5, 7
query, 130
Radon-Nikodym derivative, 31
random variable, 17
realizable case, 7, 23, 52
recoverability, 6, 9, 91, 91, 92
refute, 24
regret, 3, 68, 90, 91, 92, 95, 96
reward, 2, 50, 53
-algebra,16, 51
Borel,16
0
n-formula,103
SARSA, 2
self-optimizing, 82
semimeasure, 21
chronological, 51
contextual, 51
semiprior, 26
semiring, 152
Solomonoﬀ
induction, 41
mixture, 26
normalization, 21, 48
prior, 7,26, 26, 41, 42, 45, 106, 108,
135
state, 3, 7, 54
stochastic process, 18
subjective enviroment, 138
178 Index
t0-value,56
tail event, 32, 44, 100
TD-learning, 3
Thompson sampling, 10, 65, 66, 84, 88,
96, 142, 143
time consistent, 52, 62
total variation distance, 19, 20, 59
expected, 85, 85
trap, 2, 49, 55, 90, 99
Turing machine
monotone, 20, 21, 129
natural, 97
probabilistic, 129
universal, 20, 26, 42, 98
universal
artiﬁcial intelligence, 62
compression, 43–45
induction, seeSolomonoﬀ induction
universal artiﬁcial intelligence, 149
value function, 49, 56, 58, 59, 109, 136
entropy-seeking, 63, 64, 122
information-seeking, 64, 122
iterative, 115, 116, 118–120
recursive, 56, 56
reward-seeking, 63
weakly communicating, 55, 89–92
wireheading, 6, 149