Chapter 5
Agent Foundations for Aligning Machine
Intelligence with Human Interests:
A Technical Research Agenda
Nate Soares and Benya Fallenstein
5.1 Introduction
The property that has given humans a dominant advantage over other species is
not strength or speed, but intelligence. If progress in artiﬁcial intelligence continuesunabated, AI systems will eventually exceed humans in general reasoning ability. A
system that is “superintelligent” in the sense of being “smarter than the best human
brains in practically every ﬁeld” could have an enormous impact upon humanity
(Bostrom 2014 ). Just as human intelligence has allowed us to develop tools and
strategies for controlling our environment, a superintelligent system would likely becapable of developing its own tools and strategies for exerting control (Muehlhauser
and Salamon 2012 ). In light of this potential, it is essential to use caution when devel-
oping AI systems that can exceed human levels of general intelligence, or that canfacilitate the creation of such systems.
Since artiﬁcial agents would not share our evolutionary history, there is no reason
to expect them to be driven by human motivations such as lust for power. However,nearly all goals can be better met with more resources (Omohundro 2008 ). This
suggests that, by default, superintelligent agents would have incentives to acquire
resources currently being used by humanity. (Just as artiﬁcial agents would not auto-matically acquire a lust for power, they would not automatically acquire a human
sense of fairness, compassion, or conservatism.) Thus, most goals would put the
agent at odds with human interests, giving it incentives to deceive or manipulate itshuman operators and resist interventions designed to change or debug its behavior
(Bostrom 2014 , Chap. 8).
N. Soares ( ✉)⋅B. Fallenstein
Machine Intelligence Research Institute, Berkeley, USAe-mail: nate@intelligence.org
B. Fallenstein
e-mail: benya@intelligence.org
© Springer-Verlag GmbH Germany 2017
V. Callaghan et al. (eds.), The Technological Singularity ,
The Frontiers Collection, DOI 10.1007/978-3-662-54033-6\_5103
104 N. Soares and B. Fallenstein
Care must be taken to avoid constructing systems that exhibit this default behav-
ior. In order to ensure that the development of smarter-than-human intelligence has
a positive impact on the world, we must meet three formidable challenges: How canwe create an agent that will reliably pursue the goals it is given? How can we for-
mally specify beneﬁcial goals? And how can we ensure that this agent will assist and
cooperate with its programmers as they improve its design, given that mistakes inearly AI systems are inevitable?
This agenda discusses technical research that is tractable today, which the authors
think will make it easier to confront these three challenges in the future. Sections 5.2
through 5.4motivate and discuss six research topics that we think are relevant to
these challenges.
We call a smarter-than-human system that reliably pursues beneﬁcial goals
“aligned with human interests” or simply “aligned.”
1To become conﬁdent that an
agent is aligned in this way, a practical implementation that merely seems to meet
the challenges outlined above will not suﬃce. It is also important to gain a solid for-mal understanding of why that conﬁdence is justiﬁed. This technical agenda argues
that there is foundational research we can make progress on today that will make it
easier to develop aligned systems in the future, and describes ongoing work on some
of these problems.
Of the three challenges, the one giving rise to the largest number of currently
tractable research questions is the challenge of ﬁnding an agent architecture that will
reliably and autonomously pursue a set of objectives—that is, an architecture that
can at least be aligned with some end goal. This requires theoretical knowledge of
how to design agents which reason well and behave as intended even in situations
never envisioned by the programmers. The problem of highly reliable agent designs
is discussed in Sect. 5.2.
The challenge of developing agent designs which are tolerant of human error also
gives rise to a number of tractable problems. We expect that smarter-than-human
systems would by default have incentives to manipulate and deceive human opera-tors, and that special care must be taken to develop agent architectures which avert
these incentives and are otherwise tolerant of programmer error. This problem and
some related open questions are discussed in Sect. 5.3.
Reliable and error-tolerant agent designs are only beneﬁcial if the resulting agent
actually pursues desirable outcomes. The diﬃculty of concretely specifying what
is meant by “beneﬁcial behavior” implies a need for some way to construct agentsthat reliably learn what to value (Bostrom 2014 , Chap. 12). A solution to this
“value learning” problem is vital; attempts to start making progress are reviewed
in Sect. 5.4.
1A more careful wording might be “aligned with the interests of sentient beings.” We would not
want to beneﬁt humans at the expense of sentient non-human animals—or (if we build them) at theexpense of sentient machines.
5 Agent Foundations for Aligning Machine Intelligence . . . 105
Why work on these problems now, if smarter-than-human AI is likely to be
decades away? This question is touched upon brieﬂy below, and is discussed fur-
ther in Sect. 5.5. In short, the authors believe that there are theoretical prerequisites
for designing aligned smarter-than-human systems over and above what is required
to design misaligned systems. We believe that research can be done today that will
make it easier to address alignment concerns in the future.
5.1.1 Why These Problems?
This technical agenda primarily covers topics that the authors believe are tractable ,
uncrowded ,focused , and unable to be outsourced to forerunners of the target AI
system.
Bytractable problems, we mean open problems that are concrete and admit
immediate progress. Signiﬁcant eﬀort will ultimately be required to align real
smarter-than-human systems with beneﬁcial values, but in the absence of working
designs for smarter-than-human systems, it is diﬃcult if not impossible to begin most
of that work in advance. This agenda focuses on research that can help us gain a better
understanding today of the problems faced by almost any suﬃciently advanced AIsystem. Whether practical smarter-than-human systems arise in ten years or in one
hundred years, we expect to be better able to design safe systems if we understand
solutions to these problems.
This agenda further limits attention to uncrowded domains, where there is not
already an abundance of research being done, and where the problems may not be
solved over the course of “normal” AI research. For example, program veriﬁcationtechniques are absolutely crucial in the design of extremely reliable programs (Sotala
and Yampolskiy 2015, Sect. 5.5, this volume), but program veriﬁcation is not covered
in this agenda primarily because a vibrant community is already actively studyingthe topic.
This agenda also restricts consideration to focused tools, ones that would be
useful for designing aligned systems in particular (as opposed to intelligent sys-tems in general). It might be possible to design generally intelligent AI systems
before developing an understanding of highly reliable reasoning suﬃcient for con-
structing an aligned system. This could lead to a risky situation where powerful AIsystems are built long before the tools needed to safely utilize them. Currently, sig-
niﬁcant research eﬀort is focused on improving the capabilities of artiﬁcially intel-
ligent systems, and comparatively little eﬀort is focused on superintelligence align-
ment (Bostrom 2014 , Chap. 14). For that reason, this agenda focuses on research that
improves our ability to design aligned systems in particular.
Lastly, we focus on research that cannot be safely delegated to machines .A sA I
algorithms come to rival humans in scientiﬁc inference and planning, new possibili-
ties will emerge for outsourcing computer science labor to AI algorithms themselves.
106 N. Soares and B. Fallenstein
This is a consequence of the fact that intelligence is the technology we are designing:
on the path to great intelligence, much of the work may be done by smarter-than-
human systems.2
As a result, the topics discussed in this agenda are ones that we believe are diﬃcult
to safely delegate to AI systems. Error-tolerant agent design is a good example: no AI
problem (including the problem of error-tolerant agent design itself) can be safelydelegated to a highly intelligent artiﬁcial agent that has incentives to manipulate
or deceive its programmers. By contrast, a suﬃciently capable automated engineer
would be able to make robust contributions to computer vision or natural languageprocessing even if its own visual or linguistic abilities were initially lacking. Most
intelligent agents optimizing for some goal would also have incentives to improve
their visual and linguistic abilities so as to enhance their ability to model and interactwith the world.
It would be risky to delegate a crucial task before attaining a solid theoretical
understanding of exactly what task is being delegated. It may be possible to useour understanding of ideal Bayesian inference to task a highly intelligent system
with developing increasingly eﬀective approximations of a Bayesian reasoner, but
it would be far more diﬃcult to delegate the task of “ﬁnding good ways to revise
how conﬁdent you are about claims” to an intelligent system before gaining a solid
understanding of probability theory. The theoretical understanding is useful to ensurethat the right questions are being asked.
5.2 Highly Reliable Agent Designs
Bird and Layzell ( 2002 ) describe a genetic algorithm which, tasked with making
an oscillator, re-purposed the printed circuit board tracks on the motherboard as a
makeshift radio to amplify oscillating signals from nearby computers. This is notkind of solution the algorithm would have found if it had been simulated on a vir-
tual circuit board possessing only the features that seemed relevant to the problem.
Intelligent search processes in the real world have the ability to use resources in unex-pected ways, e.g., by ﬁnding “shortcuts” or “cheats” not accounted for in a simpliﬁed
model.
When constructing intelligent systems which learn and interact with all the com-
plexities of reality, it is not suﬃcient to verify that the algorithm behaves well in test
settings. Additional work is necessary to verify that the system will continue work-
ing as intended in application. This is especially true of systems possessing gen-
eral intelligence at or above the human level: superintelligent machines might ﬁnd
2Since the Dartmouth Proposal (McCarthy et al. 1955 ), it has been a standard idea in AI that a
suﬃciently smart machine intelligence could be intelligent enough to improve itself. In 1965, I.J.Good observed that this might create a positive feedback loop leading to an “intelligence explosion”(Good 1965 ). Sotala and Yampolskiy (2015, Sect. 2.3, this volume) and Bostrom ( 2014 , Chap. 14)
has observed that an intelligence explosion is especially likely if the agent has the ability to acquiremore hardware, improve its software, or design new hardware.
5 Agent Foundations for Aligning Machine Intelligence . . . 107
strategies and execute plans beyond both the experience and imagination of the
programmers, making the clever oscillator of Bird and Layzell look trite. At the
same time, unpredictable behavior from smarter-than-human systems could causecatastrophic damage, if they are not aligned with human interests (Yudkowsky 2008 ).
Because the stakes are so high, testing combined with a gut-level intuition that the
system will continue to work outside the test environment is insuﬃcient, even if thetesting is extensive. It is important to also have a formal understanding of precisely
why the system is expected to behave well in application.
What constitutes a formal understanding? It seems essential to us to have both
(1) an understanding of precisely what problem the system is intended to solve; and
(2) an understanding of precisely why thispractical system is expected to solve that
abstract problem. The latter must wait for the development of practical smarter-than-human systems, but the former is a theoretical research problem that we can already
examine.
A full description of the problem would reveal the conceptual tools needed to
understand why practical heuristics are expected to work. By analogy, consider the
game of chess. Before designing practical chess algorithms, it is necessary to possess
not only a predicate describing checkmate, but also a description of the problem in
term of trees and backtracking algorithms: Trees and backtracking do not immedi-
ately yield a practical solution—building a full game tree is infeasible—but they arethe conceptual tools of computer chess. It would be quite diﬃcult to justify conﬁ-
dence in a chess heuristic before understanding trees and backtracking.
While these conceptual tools may seem obvious in hindsight, they were not clear
to foresight. Consider the famous essay by Edgar Allen Poe about Maelzel’s Mechan-
ical Turk (Poe 1836 ). It is in many ways remarkably sophisticated: Poe compares the
Turk to “the calculating machine of Mr. Babbage” and then remarks on how the twosystems cannot be of the same kind, since in Babbage’s algebraical problems each
step follows of necessity, and so can be represented by mechanical gears making
deterministic motions; while in a chess game, no move follows with necessity fromthe position of the board, and even if our own move followed with necessity, the
opponent’s would not. And so (argues Poe) we can see that chess cannot possibly
be played by mere mechanisms, only by thinking beings. From Poe’s state of knowl-edge, Shannon’s ( 1950 ) description of an idealized solution in terms of backtracking
and trees constitutes a great insight. Our task it to put theoretical foundations under
the ﬁeld of general intelligence, in the same sense that Shannon put theoretical foun-dations under the ﬁeld of computer chess.
It is possible that these foundations will be developed over time, during the normal
course of AI research: in the past, theory has often preceded application. But the
converse is also true: in many cases, application has preceded theory. The claim of
this technical agenda is that, in safety-critical applications where mistakes can putlives at risk, it is crucial that certain theoretical insights come ﬁrst.
A smarter-than-human agent would be embedded within and computed by a com-
plex universe, learning about its environment and bringing about desirable states of
108 N. Soares and B. Fallenstein
aﬀairs. How is this formalized? What metric captures the question of how well an
agent would perform in the real world?3
Not all parts of the problem must be solved in advance: the task of designing
smarter, safer, more reliable systems could be delegated to early smarter-than-human
systems, if the research done by those early systems can be suﬃciently trusted. It is
important, then, to focus research eﬀorts particularly on parts of the problem wherean increased understanding is necessary to construct a minimal reliable generally
intelligent system. Moreover, it is important to focus on aspects which are currently
tractable, so that progress can in fact be made today, and on issues relevant to align-ment in particular, which would not otherwise be studied over the course of “normal”
AI research.
In this section, we discuss four candidate topics meeting these criteria: (1) real-
istic world-models , the study of agents learning and pursuing goals while embedded
within a physical world; (2) decision theory , the study of idealized decision-making
procedures; (3) logical uncertainty , the study of reliable reasoning with bounded
deductive capabilities; and (4) Vingean reﬂection , the study of reliable methods for
reasoning about agents that are more intelligent than the reasoner. We will now dis-
cuss each of these topics in turn.
5.2.1 Realistic World-Models
Formalizing the problem of computer intelligence may seem easy in theory: encode
some set of preferences as a utility function, and evaluate the expected utility that
would be obtained if the agent were implemented. However, this is not a full speciﬁ-cation: What is the set of “possible realities” used to model the world? Against what
distribution over world models is the agent evaluated? How is a given world model
used to score an agent? To ensure that an agent would work well in reality, it is ﬁrstuseful to formalize the problem faced by agents learning (and acting in) arbitrary
environments.
Solomonoﬀ ( 1964 ) made an early attempt to tackle these questions by specifying
an “induction problem” in which an agent must construct world models and promote
correct hypotheses based on the observation of an arbitrarily complex environment,
in a manner reminiscent of scientiﬁc induction. In this problem, the agent and envi-ronment are separate. The agent gets to see one bit from the environment in each
turn, and must predict the bits which follow.
Solomonoﬀ’s induction problem answers all three of the above questions in
a simpliﬁed setting: The set of world models is any computable environment
3Legg and Hutter ( 2007 ) provide a preliminary answer to this question, by deﬁning a “universal
measure of intelligence” which scores how well an agent can learn the features of an external envi-ronment and maximize a reward function. This is the type of formalization we are looking for: ascoring metric which describes how well an agent would achieve some set of goals. However, whilethe Legg-Hutter metric is insightful, it makes a number of simplifying assumptions, and many dif-ﬁcult open questions remain (Soares 2015 ).
5 Agent Foundations for Aligning Machine Intelligence . . . 109
(e.g., any Turing machine). In reality, the simplest hypothesis that predicts the data
is generally correct, so agents are evaluated against a simplicity distribution. Agents
are scored according to their ability to predict their next observation. These answerswere insightful, and led to the development of many useful tools, including algorith-
mic probability and Kolmogorov complexity.
However, Solomonoﬀ’s induction problem does not fully capture the problem
faced by an agent learning about an environment while embedded within it, as a
subprocess. It assumes that the agent and environment are separated, save only for
the observation channel. What is the analog of Solomonoﬀ induction for agents thatare embedded within their environment?
This is the question of naturalized induction (Bensinger 2013 ). Unfortunately,
the insights of Solomonoﬀ do not apply in the naturalized setting. In Solomonoﬀ’ssetting, where the agent and environment are separated, one can consider arbitrary
Turing machines to be “possible environments.” But when the agent is embedded
in the environment, consideration must be restricted to environments which embedthe agent. Given an algorithm, what is the set of environments which embed that
algorithm? Given that set, what is the analogue of a simplicity prior which captures
the fact that simpler hypotheses are more often correct?
Technical problem (Naturalized Induction) What, formally, is the induction prob-
lem faced by an intelligent agent embedded within and computed by its environment?What is the set of environments which embed the agent? What constitutes a simplicity
prior over that set? How is the agent scored? For discussion, see Soares ( 2015 ).
Just as a formal description of Solomonoﬀ induction answered the above three ques-
tions in the context of an agent learning an external environment, a formal descrip-
tion of naturalized induction may well yield answers to those questions in the contextwhere agents are embedded in and computed by their environment.
Of course, the problem of computer intelligence is not simply an induction
problem: the agent must also interact with the environment. Hutter ( 2000 ) extends
Solomonoﬀ’s induction problem to an “interaction problem,” in which an agent must
both learn and act upon its environment. In each turn, the agent both observes one
input and writes one output, and the output aﬀects the behavior of the environ-ment. In this problem, the agent is evaluated in terms of its ability to maximize a
reward function speciﬁed in terms of inputs. While this model does not capture the
diﬃculties faced by agents which are embedded within their environment, it doescapture a large portion of the problem faced by agents interacting with arbitrar-
ily complex environments. Indeed, the interaction problem (and AIXI Hutter 2000 ,
its solution) are the basis for the “universal measure of intelligence” developed by
Legg and Hutter ( 2007 ).
However, even barring problems arising from the agent/environment separation,
the Legg-Hutter metric does not fully characterize the problem of computer intel-
ligence. It scores agents according to their ability to maximize a reward function
speciﬁed in terms of observation. Agents scoring well by the Legg-Hutter metricare extremely eﬀective at ensuring their observations optimize a reward function,
but these high-scoring agents are likely to be the type that ﬁnd clever ways to seize
110 N. Soares and B. Fallenstein
control of their observation channel rather than the type that identify and manipulate
the features in the world that the reward function was intended to proxy for (Soares
2015 ). Reinforcement learning techniques which punish the agent for attempting to
take control would only incentivize the agent to deceive and mollify the program-
mers until it found a way to gain a decisive advantage (Bostrom 2014 , Chap. 8).
The Legg-Hutter metric does not characterize the question of how well an algo-
rithm would perform if implemented in reality: to formalize that question, a scoring
metric must evaluate the resulting environment histories, not just the agent’s obser-
vations (Soares 2015 ).
But human goals are not speciﬁed in terms of environment histories, either: they
are speciﬁed in terms of high-level notions such as “money” or “ﬂourishing humans.”
Leaving aside problems of philosophy, imagine rating a system according to howwell it achieves a straightforward, concrete goal, such as by rating how much dia-
mond is in an environment after the agent has acted on it, where “diamond” is speci-
ﬁed concretely in terms of an atomic structure. Now the goals are speciﬁed in termsof atoms, and the environment histories are speciﬁed in terms of Turing machines
paired with an interaction history. How is the environment history evaluated in terms
of atoms? This is the ontology identiﬁcation problem.
Technical problem (Ontology Identiﬁcation) Given goals speciﬁed in some ontol-
ogy and a world model, how can the ontology of the goals be identiﬁed in the worldmodel? What types of world models are amenable to ontology identiﬁcation? For a
discussion, see Soares ( 2015 ).
To evaluate world models, the world models must be evaluated in terms of the ontol-
ogy of the goals. This may be diﬃcult in cases where the ontology of the goals
does not match reality: it is one thing to locate atoms in a Turing machine using anatomic model of physics, but it is another thing altogether to locate atoms in a Turing
machine modeling quantum physics. De Blanc ( 2011 ) further motivates the idea that
explicit mechanisms are needed to deal with changes in the ontology of the system’s
world model.
Agents built to solve the wrong problem—such as optimizing their observations—
may well be capable of attaining superintelligence, but it is unlikely that those agents
could be aligned with human interests (Bostrom 2014 , Chap. 12). A better under-
standing of naturalized induction and ontology identiﬁcation is needed to fully spec-ify the problem that intelligent agents would face when pursuing human goals while
embedded within reality, and this increased understanding could be a crucial tool
when it comes to designing highly reliable agents.
5.2.2 Decision Theory
Smarter-than-human systems must be trusted to make good decisions, but what doesit mean for a decision to be “good”? Formally, given a description of an environment
5 Agent Foundations for Aligning Machine Intelligence . . . 111
and an agent embedded within, how is the “best available action” identiﬁed, with
respect to some set of preferences? This is the question of decision theory.
The answer may seem trivial, at least in theory: simply iterate over the agent’s
available actions, evaluate what would happen if the agent took that action, and then
return whichever action leads to the most expected utility. But this is not a full spec-
iﬁcation: How are the “available actions” identiﬁed? How is what “would happen”deﬁned?
The diﬃculty is easiest to illustrate in a deterministic setting. Consider a deter-
ministic agent embedded in a deterministic environment. There is exactly one actionthat the agent will take. Given a set of actions that it “could take,” it is necessary
to evaluate, for each action, what would happen if the agent took that action. But
the agent will not take most of those actions. How is the counterfactual environmentconstructed, in which a deterministic algorithm “does something” that, in the real
environment, it doesn’t do? Answering this question requires a theory of counterfac-
tual reasoning, and counterfactual reasoning is not well understood.
Technical problem (Theory of Counterfactuals) What theory of counterfactual rea-
soning can be used to specify a procedure which always identiﬁes the best action
available to a given agent in a given environment, with respect to a given set ofpreferences? For discussion, see Soares and Fallenstein ( 2014 ).
Decision theory has been studied extensively by philosophers. The study goes back
to Pascal, and has been picked up in modern times by Lehmann ( 1950 ), Wald ( 1939 ),
Jeﬀrey ( 1983 ), Joyce ( 1999 ), Lewis ( 1981 ), Pearl ( 2000 ), and many others. However,
no satisfactory method of counterfactual reasoning yet answers this particular ques-
tion. To give an example of why counterfactual reasoning can be diﬃcult, consider
a deterministic agent playing against a perfect copy of itself in the classic prisoner’s
dilemma (Rapoport and Chammah 1965 ). The opponent is guaranteed to do the same
thing as the agent, but the agents are “causally separated,” in that the action of onecannot physically aﬀect the action of the other.
What is the counterfactual world in which the agent on the left cooperates? It is
not suﬃcient to consider changing the action of the agent on the left while hold-ing the action of the agent on the right constant, because while the two are causally
disconnected, they are logically constrained to behave identically. Standard causal
reasoning, which neglects these logical constraints, misidentiﬁes “defection” as thebest strategy available to each agent even when they know they have identical source
codes (Lewis 1979 ).
4Satisfactory counterfactual reasoning must respect these log-
ical constraints, but how are logical constraints formalized and identiﬁed? It is ﬁneto say that, in the counterfactual where the agent on the left cooperates, all identical
copies of it also cooperate; but what counts as an identical copy? What if the right
4As this is a multi-agent scenario, the problem of counterfactuals can also be thought of as game-
theoretic. The goal is to deﬁne a procedure which reliably identiﬁes the best available action; thelabel of “decision theory” is secondary. This goal subsumes both game theory and decision the-ory: the desired procedure must identify the best action in all settings, even when there is no clear
demarcation between “agent” and “environment.” Game theory informs, but does not deﬁne, this
area of research.
112 N. Soares and B. Fallenstein
agent runs the same algorithm written in a diﬀerent programming language? What
if it only does the same thing 98% of the time?
These questions are pertinent in reality: practical agents must be able to identify
good actions in settings where other actors base their actions on imperfect (but well-
informed) predictions of what the agent will do. Identifying the best action available
to an agent requires taking the non-causal logical constraints into account. A satis-factory formalization of counterfactual reasoning requires the ability to answer ques-
tions about how other deterministic algorithms behave in the counterfactual world
where the agent’s deterministic algorithm does something it doesn’t. However, theevaluation of “logical counterfactuals” is not yet well understood.
Technical problem (Logical Counterfactuals) Consider a counterfactual in which
a given deterministic decision procedure selects a diﬀerent action from the one it
selects in reality. What are the implications of this counterfactual on other algo-
rithms? Can logical counterfactuals be formalized in a satisfactory way? A methodfor reasoning about logical counterfactuals seems necessary in order to formalize a
more general theory of counterfactuals. For a discussion, see Soares and Fallenstein
(2014 ).
Unsatisfactory methods of counterfactual reasoning (such as the causal reasoning of
Pearl ( 2000 )) seem powerful enough to support smarter-than-human intelligent sys-
tems, but systems using those reasoning methods could systematically act in unde-
sirable ways (even if otherwise aligned with human interests).
To construct practical heuristics that are known to make good decisions, even
when acting beyond the oversight and control of humans, it is essential to under-
stand what is meant by “good decisions.” This requires a formulation which, given
a description of an environment, an agent embedded in that environment, and someset of preferences, identiﬁes the best action available to the agent. While modern
methods of counterfactual reasoning do not yet allow for the speciﬁcation of such
a formula, recent research has pointed the way towards some promising paths for
future research.
For example, Wei Dai’s “updateless decision theory” (UDT) is a new take on deci-
sion theory that systematically outperforms causal decision theory (Hintze 2014 ),
and two of the insights behind UDT highlight a number of tractable open problems
(Soares and Fallenstein 2014 ).
Recently, Barasz et al. ( 2014 ) developed a concrete model, together with a Haskell
implementation, of multi-agent games where agents have access to each others’
source code and base their decisions on what they can prove about their opponent.They have found that it is possible for some agents to achieve robust cooperation in
the one-shot prisoner’s dilemma while remaining unexploitable Barasz et al. ( 2014 ).
These results suggest a number of new ways to approach the problem of counter-
factual reasoning, and we are optimistic that continued study will prove fruitful.
5 Agent Foundations for Aligning Machine Intelligence . . . 113
5.2.3 Logical Uncertainty
Consider a reasoner encountering a black box with one input chute and two output
chutes. Inside the box is a complex Rube Goldberg machine that takes an input balland deposits it in one of the two output chutes. A probabilistic reasoner may have
uncertainty about where the ball will exit, due to uncertainty about which Rube Gold-
berg machine is in the box. However, standard probability theory assumes that if thereasoner didknow which machine the box implemented, they would know where
the ball would exit: the reasoner is assumed to be logically omniscient , i.e., to know
all logical consequences of any hypothesis they entertain.
By contrast, a practical bounded reasoner may be able to know exactly which
Rube Goldberg machine the box implements without knowing where the ball will
come out, due to the complexity of the machine. This reasoner is logically uncertain .
Almost all practical reasoning is done under some form of logical uncertainty (Gaif-
man 2004 ), and almost all reasoning done by a smarter-than-human agent must be
some form of logically uncertain reasoning. Any time an agent reasons about the con-
sequences of a plan, the eﬀects of running a piece of software, or the implications of
an observation, it must do some sort of reasoning under logical uncertainty. Indeed,the problem of an agent reasoning about an environment in which it is embedded as
a subprocess is inherently a problem of reasoning under logical uncertainty.
Thus, to construct a highly reliable smarter-than-human system, it is vitally
important to ensure that the agent’s logically uncertain reasoning is reliable and
trustworthy. This requires a better understanding of the theoretical underpinnings of
logical uncertainty, to more fully characterize what it means for logically uncertainreasoning to be “reliable and trustworthy” (Soares and Fallenstein 2015 ).
It is natural to consider extending standard probability theory to include the con-
sideration of worlds which are “logically impossible” (e.g., where a deterministicRube Goldberg machine behaves in a way that it doesn’t). This gives rise to two
questions: What, precisely, are logically impossible possibilities? And, given some
means of reasoning about impossible possibilities, what is a reasonable prior prob-
ability distribution over impossible possibilities?
The problem is diﬃcult to approach in full generality, but a study of logical uncer-
tainty in the restricted context of assigning probabilities to logical sentences goes
back at least to Łoś ( 1955 ) and Gaifman ( 1964 ), and has been investigated by many,
including Halpern ( 2003 ), Hutter et al. ( 2013 ), Demski ( 2012 ), Russell ( 2014 ), and
others. Though it isn’t clear to what degree this formalism captures the kind of logi-
cally uncertain reasoning a realistic agent would use, logical sentences in, for exam-
ple, the language of Peano Arithmetic are quite expressive: for example, given theRube Goldberg machine discussed above, it is possible to form a sentence which is
true if and only if the machine deposits the ball into the top chute. Thus, considering
reasoners which are uncertain about logical sentences is a useful starting point. Theproblem of assigning probabilities to sentences of logic naturally divides itself into
two parts.
114 N. Soares and B. Fallenstein
First, how can probabilities consistently be assigned to sentences? An agent
assigning probability 1 to short contradictions is hardly reasoning about the
sentences as if they are logical sentences: some of the logical structure must bepreserved. But which aspects of the logical structure? Preserving all logical impli-
cations requires that the reasoner be deductively omnipotent, as some implications
𝜙→𝜓may be very involved. The standard answer in the literature is that a coher-
ent assignment of probabilities to sentences corresponds to a probability distribution
over complete, consistent logical theories (Gaifman 1964 ; Christiano 2014a ); that is,
an “impossible possibility” is any consistent assignment of truth values to all sen-tences. Deductively limited reasoners cannot have fully coherent distributions, but
they can approximate these distributions: for a deductively limited reasoner, “impos-
sible possibilities” can be any assignment of truth values to sentences that looksconsistent so far, so long as the assignment is discarded as soon as a contradiction is
introduced.
Technical problem (Impossible Possibilities) How can deductively limited reason-
ers approximate reasoning according to a probability distribution over complete the-
ories of logic? For a discussion, see Christiano ( 2014a ).
Second, what is a satisfactory prior probability distribution over logical sentences?
If the system is intended to reason according to a theory at least as powerful as PeanoArithmetic ( 𝖯𝖠), then that theory will be incomplete (Gödel et al. 1934 ). What prior
distribution places nonzero probability on the set of complete extensions of 𝖯𝖠?
Deductively limited agents would not be able to literally use such a prior, but if itwere computably approximable, then they could start with a rough approximation
of the prior and reﬁne it over time. Indeed, the process of reﬁning a logical prior—
getting better and better probability estimates for given logical sentences—capturesthe whole problem of reasoning under logical uncertainty in miniature. Hutter et al.
(2013 ) have deﬁned a desirable prior, but Sawin and Demski ( 2013 )h a v es h o w n
that it cannot be computably approximated. Demski ( 2012 ) and Christiano ( 2014a )
have also proposed logical priors, but neither seems fully satisfactory. The speciﬁca-
tion of satisfactory logical priors is diﬃcult in part because it is not yet clear whichproperties are desirable in a logical prior, nor which properties are possible.
Technical problem (Logical Priors) What is a satisfactory set of priors over logical
sentences that a bounded reasoner can approximate? For a discussion, see Soares
and Fallenstein ( 2015 ).
Many existing tools for studying reasoning, such as game theory, standard proba-
bility theory, and Bayesian networks, all assume that reasoners are logically omni-
scient. A theory of reasoning under logical uncertainty seems necessary to formalizethe problem of naturalized induction, and to generate a satisfactory theory of coun-
terfactual reasoning. If these tools are to be developed, extended, or improved, then
a better understanding of logically uncertain reasoning is required.
5 Agent Foundations for Aligning Machine Intelligence . . . 115
5.2.4 Vingean Reﬂection
Instead of specifying superintelligent systems directly, it seems likely that humans
may instead specify generally intelligent systems that go on to create or attainsuperintelligence. In this case, the reliability of the resulting superintelligent sys-
tem depends upon the reasoning of the initial system which created it (either anew
or via self-modiﬁcation).
If the agent reasons reliably under logical uncertainty, then it may have a generic
ability to evaluate various plans and strategies, only selecting those which seem ben-
eﬁcial. However, some scenarios put that logically uncertain reasoning to the testmore than others. There is a qualitative diﬀerence between reasoning about simple
programs and reasoning about human-level intelligent systems. For example, mod-
ern program veriﬁcation techniques could be used to ensure that a “smart” military
drone obeys certain rules of engagement, but it would be a diﬀerent problem alto-
gether to verify the behavior of an artiﬁcial military general which must run an entirewar. A general has far more autonomy, ability to come up with clever unexpected
strategies, and opportunities to impact the future than a drone.
A self-modifying agent (or any that constructs new agents more intelligent than
itself) must reason about the behavior of a system that is more intelligent than the rea-
soner. This type of reasoning is critically important to the design of self-improving
agents: if a system will attain superintelligence through self-modiﬁcation, then theimpact of the system depends entirely upon the correctness of the original agent’s
reasoning about its self-modiﬁcations (Fallenstein and Soares 2015 ).
Before trusting a system to attain superintelligence, it seems prudent to ensure
that the agent uses appropriate caution when reasoning about successor agents.
5That
is, it seems necessary to understand the mechanisms by which agents reason about
smarter systems.
Naive tools for reasoning about plans including smarter agents, such as backwards
induction (Ben-Porath 1997 ), would have the reasoner evaluate the smarter agent by
simply checking what the smarter agent would do. This does not capture the diﬃculty
of the problem: a parent agent cannot simply check what its successor agent would do
in all scenarios, for if it could, then it would already know what actions its successorwill take, and the successor would not in any way be smarter.
Yudkowsky and Herreshoﬀ ( 2013 ) call this observation the “Vingean principle,”
after Vernor Vinge ( 1993 ), who emphasized how diﬃcult it is for humans to predict
the behavior of smarter-than-human agents. Any agent reasoning about more intelli-
gent successor agents must do so abstractly , without pre-computing all actions that
the successor would take in every scenario. We refer to this kind of reasoning asVingean reﬂection .
5Of course, if an agent reasons perfectly under logical uncertainty, it would also reason well about
the construction of successor agents. However, given the fallibility of human reasoning and the factthat this path is critically important, it seems prudent to verify the agent’s reasoning methods in thisscenario speciﬁcally.
116 N. Soares and B. Fallenstein
Technical problem (Vingean Reﬂection) How can agents reliably reason about
agents which are smarter than themselves, without violating the Vingean principle?
For discussion, see Fallenstein and Soares ( 2015 ).
It may seem premature to worry about how agents reason about self-improvements
before developing a theoretical understanding of reasoning under logical uncertainty
in general. However, it seems to us that work in this area can inform understandingof what sort of logically uncertain reasoning is necessary to reliably handle Vingean
reﬂection.
Given the high stakes when constructing systems smarter than themselves, artiﬁ-
cial agents might use some form of extremely high-conﬁdence reasoning to verify the
safety of potentially dangerous self-modiﬁcations. When humans desire extremely
high reliability, as is the case for (e.g.) autopilot software, we often use formal logical
systems (United States Department of Defense 1985 ; United Kingdom Ministry of
Defense 1991 ). High-conﬁdence reasoning in critical situations may require some-
thing akin to formal veriﬁcation (even if most reasoning is done using more generic
logically uncertain reasoning), and so studying Vingean reﬂection in the domain of
formal logic seems like a good starting point.
Logical models of agents reasoning about agents that are “more intelligent,” how-
ever, run into a number of obstacles. By Gödel’s second incompleteness theorem
(1934 ), suﬃciently powerful formal systems cannot rule out the possibility that they
may be inconsistent. This makes it diﬃcult for agents using formal logical reason-
ing to verify the reasoning of similar agents which also use formal logic for high-
conﬁdence reasoning; the ﬁrst agent cannot verify that the latter agent is consistent.Roughly, it seems desirable to be able to develop agents which reason as follows:
This smarter successor agent uses reasoning similar to mine, and my own reasoning is sound,
so its reasoning is sound as well.
However, Gödel et al. ( 1934 ) showed that this sort of reasoning leads to inconsis-
tency, and these problems do in fact make Vingean reﬂection diﬃcult in a logical
setting (Fallenstein and Soares 2015 ; Yudkowsky 2013 ).
Technical problem (Löbian Obstacle) How can agents gain very high conﬁdence
in agents that use similar reasoning systems, while avoiding paradoxes of self-
reference? For discussion, see Fallenstein and Soares ( 2015 ).
These results may seem like artifacts of models rooted in formal logic, and may
seem irrelevant given that practical agents must eventually use logical uncertainty
rather than formal logic to reason about smarter successors. However, it has been
shown that many of the Gödelian obstacles carry over into early probabilistic logics
in a straightforward way, and some results have already been shown to apply in thedomain of logical uncertainty (Fallenstein 2014 ).
Studying toy models in this formal logical setting has led to partial solutions
(Fallenstein and Soares 2014 ). Recent work has pushed these models towards prob-
abilistic settings (Fallenstein and Soares 2014 ; Yudkowsky 2014 ; Soares 2014 ).
Further research may continue driving the development of methods for reasoning
5 Agent Foundations for Aligning Machine Intelligence . . . 117
under logical uncertainty which can handle Vingean reﬂection in a reliable way (Fal-
lenstein and Soares 2015 ).
5.3 Error-Tolerant Agent Designs
Incorrectly speciﬁed superintelligent agents could be dangerous (Yudkowsky 2008 ).
Correcting a modern AI system involves simply shutting the system down and mod-ifying its source code. Modifying a smarter-than-human system may prove more
diﬃcult: a system attaining superintelligence could acquire new hardware, alter its
software, create subagents, and take other actions that would leave the original pro-grammers with only dubious control over the agent. This is especially true if the
agent has incentives to resist modiﬁcation or shutdown. If intelligent systems are to
be safe, they must be constructed in such a way that they are amenable to correction,even if they have the ability to prevent or avoid correction.
This does not come for free: by default, agents have incentives to preserve their
own preferences, even if those conﬂict with the intentions of the programmers
(Omohundro 2008 ; Soares and Fallenstein 2015 ). Special care is needed to specify
agents that avoid the default incentives to manipulate and deceive (Bostrom 2014 ,
Chap. 8).
Restricting the actions available to a superintelligent agent may be quite diﬃ-
cult (Bostrom 2014 , Chap. 9). Intelligent optimization processes often ﬁnd unex-
pected ways to fulﬁll their optimization criterion using whatever resources are at
their disposal; recall the evolved oscillator of Bird and Layzell ( 2002 ). Superintel-
ligent optimization processes may well use hardware, software, and other resourcesin unanticipated ways, making them diﬃcult to contain if they have incentives to
escape.
We must learn how to design agents which do not have incentives to escape,
manipulate, or deceive in the ﬁrst place: agents which reason as if they are incom-
plete and potentially ﬂawed in dangerous ways, and which are therefore amenable to
online correction. Reasoning of this form is known as “corrigible reasoning.”
Technical problem (Corrigibility) What sort of reasoning can reﬂect the fact that
an agent is incomplete and potentially ﬂawed in dangerous ways? For discussion,see Soares and Fallenstein ( 2015 ).
Naïve attempts at specifying corrigible behavior are unsatisfactory. For example,
“moral uncertainty” frameworks could allow agents to learn values through obser-
vation and interaction, but would still incentivize agents to resist changes to the
underlying moral uncertainty framework if it happened to be ﬂawed. Simple “penaltyterms” for manipulation and deception also seem doomed to failure: agents subject
to such penalties would have incentives to resist modiﬁcation while cleverly avoid-
ing the technical deﬁnitions of “manipulation” and “deception.” The goal is not to
118 N. Soares and B. Fallenstein
design systems that fail in their attempts to deceive the programmers; the goal is to
construct reasoning methods that do not give rise to deception incentives in the ﬁrst
place.
A formalization of the intuitive notion of corrigibility remains elusive. Active
research is currently focused on small toy problems, in the hopes that insight gained
there will generalize. One such toy problem is the “shutdown problem,” whichinvolves designing a set of preferences that incentivize an agent to shut down upon
the press of a button without also incentivizing the agent to either cause or prevent
the pressing of that button (Soares and Fallenstein 2015 ). Stuart Armstrong’s utility
indiﬀerence technique ( 2015 ) provides a partial solution, but not a fully satisfactory
one.
Technical problem (Utility Indiﬀerence) Can a utility function be speciﬁed such
that agents maximizing that utility function switch their preferences on demand,
without having incentives to cause or prevent the switching? For discussion, seeArmstrong ( 2015 ).
A better understanding of corrigible reasoning is essential to design agent archi-
tectures that are tolerant of human error. Other research could also prove fruit-
ful, including research into reliable containment mechanisms. Alternatively, agent
designs could somehow incentivize the agent to have a “low impact” on the world.Specifying “low impact” is trickier than it may seem: How do you tell an agent that
it can’t aﬀect the physical world, given that its RAM is physical? How do you tell
it that it can only use its own hardware, without allowing it to use its motherboardas a makeshift radio? How do you tell it not to cause big changes in the world when
its behavior inﬂuences the actions of the programmers, who inﬂuence the world in
chaotic ways?
Technical problem (Domesticity) How can an intelligent agent be safely incen-
tivized to have a low impact? Specifying such a thing is not as easy as it seems.For a discussion, see Armstrong et al. ( 2012 ).
Regardless of the methodology used, it is crucial to understand designs for agents
that could be updated and modiﬁed during the development process, so as to ensure
that the inevitable human errors do not lead to catastrophe.
5.4 Value Speciﬁcation
A highly-reliable, error-tolerant agent design does not guarantee a positive impact;
the eﬀects of the system still depend upon whether it is pursuing appropriate goals.
A superintelligent system may ﬁnd clever, unintended ways to achieve the spe-
ciﬁc goals that it is given. Imagine a superintelligent system designed to cure cancerwhich does so by stealing resources, proliferating robotic laboratories at the expense
of the biosphere, and kidnapping test subjects: the intended goal may have been
5 Agent Foundations for Aligning Machine Intelligence . . . 119
“cure cancer without doing anything bad,” but such a goal is rooted in cultural con-
text and shared human knowledge.
It is not suﬃcient to construct systems that are smart enough to ﬁgure out the
intended goals. Human beings, upon learning that natural selection “intended” sex
to be pleasurable only for purposes of reproduction, do not suddenly decide that con-
traceptives are abhorrent. While one should not anthropomorphize natural selection,humans are capable of understanding the process which created them while being
completely unmotivated to alter their preferences. For similar reasons, when devel-
oping AI systems, is not suﬃcient to develop a system intelligent enough to ﬁgureout the intended goals; the system must also somehow be deliberately constructed
to pursue them (Bostrom 2014 , Chap. 8).
However, the “intentions” of the operators are a complex, vague, fuzzy, context-
dependent notion (Yudkowsky 2011 ; cf. Sotala and Yampolskiy 2015, Sects. 2.2 and
5.2.5, this volume). Concretely writing out the full intentions of the operators in a
machine-readable format is implausible if not impossible, even for simple tasks. Anintelligent agent must be designed to learn and act according to the preferences of
its operators.
6This is the value learning problem .
Directly programming a rule which identiﬁes cats in images is implausibly diﬃ-
cult, but specifying a system that inductively learns how to identify cats in images
is possible. Similarly, while directly programming a rule capturing complex humanintentions is implausibly diﬃcult, intelligent agents could be constructed to induc-
tively learn values from training data.
Inductive value learning presents unique diﬃculties. The goal is to develop a sys-
tem which can classify potential outcomes according to their value, but what sort of
training data allows this classiﬁcation? The labeled data could be given in terms of
the agent’s world-model, but this is a brittle solution if the ontology of the world-model is liable to change. Alternatively, the labeled data could come in terms of
observations, in which case the agent would have to ﬁrst learn how the labels in
the observations map onto objects in the world-model, and then learn how to clas-
sify outcomes. Designing algorithms which can do this likely requires a better under-
standing of methods for constructing multi-level world-models from sense data.
Technical problem (Multi-Level World-Models) How can multi-level world-
models be constructed from sense data in a manner amenable to ontology identi-ﬁcation? For a discussion, see Soares ( 2016 ).
Standard problems of inductive learning arise, as well: how could a training set be
constructed which allows the agent to fully learn the complexities of value? It is easy
to imagine a training set which labels many observations of happy humans as “good”and many observations of needlessly suﬀering humans as “bad,” but the simplest
generalization from this data set may well be that humans value human-shaped things
6Or of all humans, or of all sapient creatures, etc. There are many philosophical concerns surround-
ing what sort of goals are ethical when aligning a superintelligent system, but a solution to the valuelearning problem will be a practical necessity regardless of which philosophical view is the correctone.
120 N. Soares and B. Fallenstein
mimicking happy emotions: after training on this data, an agent may be inclined to
construct many simple animatronics mimicking happiness. Creating a training set
that covers all relevant dimensions of human value is diﬃcult for the same reasonthat specifying human value directly is diﬃcult. In order for inductive value learning
to succeed, it is necessary to construct a system which identiﬁes ambiguities in the
training set—dimensions along which the training set gives no information—andqueries the operators accordingly.
Technical problem (Ambiguity Identiﬁcation) Given a training data set and a
world model, how can dimensions which are neglected by the training data be iden-
tiﬁed? For discussion, see Soares ( 2016 ).
This problem is not unique to value learning, but it is especially important for it.
Research into the programmatic identiﬁcation of ambiguities, and the generation of
“queries” which are similar to previous training data but diﬀer along the ambiguous
axis, would assist in the development of systems which can safely perform inductivevalue learning.
Intuitively, an intelligent agent should be able to use some of its intelligence to
assist in this process: it does not take a detailed understanding of the human psyche
to deduce that humans care more about some ambiguities (are the human-shaped
things actually humans?) than others (does it matter if there is a breeze?). To build asystem that acts as intended, the system must model the intentions of the operators
and act accordingly. This adds another layer of indirection: the system must model
the operators in some way, and must extract “preferences” from the operator-modeland update its preferences accordingly (in a manner robust against improvements
to the model of the operator). Techniques such as inverse reinforcement learning
(Ng and Russell 2000 ), in which the agent assumes that the operator is maximizing
some reward function speciﬁed in terms of observations, are a good start, but many
questions remain unanswered.
Technical problem (Operator Modeling) By what methods can an operator be mod-
eled in such a way that (1) a model of the operator’s preferences can be extracted;
and (2) the model may eventually become arbitrarily accurate and represent theoperator as a subsystem embedded within the larger world? For a discussion, see
Soares ( 2016 ).
A system which acts as the operators intend may still have signiﬁcant diﬃculty
answering questions that the operators themselves cannot answer: imagine humans
trying to design an artiﬁcial agent to do what they would want, if they were better peo-
ple. How can normative uncertainty (uncertainty about moral claims) be resolved?
Bostrom ( 2014 , Chap. 13) suggests an additional level of indirection: task the system
with reasoning about what sorts of conclusions the operators would come to if theyhad more information and more time to think. Formalizing this is diﬃcult, and the
problems are largely still in the realm of philosophy rather than technical research.
However, Christiano ( 2014b ) has sketched one possible method by which the voli-
tion of a human could be extrapolated, and Soares ( 2016 ) discusses some potential
pitfalls.
5 Agent Foundations for Aligning Machine Intelligence . . . 121
Philosophical problem (Normative Uncertainty) What ought one do when one is
uncertain about what one ought to do? What norms govern uncertainty about nor-
mative claims? For a discussion, see MacAskill ( 2014 ).
Human operators with total control over a superintelligent system could give
rise to a moral hazard of extraordinary proportions by putting unprecedented power
into the hands of a small few (Bostrom 2014 , Chap. 6). The extraordinary poten-
tial of superintelligence gives rise to many ethical questions. When constructing
autonomous agents that will have a dominant ability to determine the future, it isimportant to design the agents to not only act according to the wishes of the oper-
ators, but also in others’ common interest. Here we largely leave the philosophical
questions aside, and remark only that those who design systems intended to surpasshuman intelligence will take on a responsibility of unprecedented scale.
5.5 Discussion
Sections 5.2through 5.4discussed six research topics where the authors think that
further research could make it easier to develop aligned systems in the future. This
section discusses reasons why we think useful progress can be made today.
5.5.1 Toward a Formal Understanding of the Problem
Are the problems discussed above tractable, uncrowded, focused, and unlikely to be
solved automatically in the course of developing increasingly intelligent AI systems?
They are certainly not very crowded. They also appear amenable to progress in
the near future, though it is less clear whether they can be fully solved.
When it comes to focus, some think that problems of decision theory and logical
uncertainty sound more like generic theoretical AI research than alignment-speciﬁcresearch. A more intuitive set of alignment problems might put greater emphasis on
AI constraint (see Chap. 4in this book) or value learning.
Progress on the topics outlined in this agenda might indeed make it easier to
design intelligent systems in general. Just as the intelligence metric of Legg and
Hutter ( 2007 ) lent insight into the ideal priors for agents facing Hutter’s interaction
problem, a full description of the naturalized induction problem could lend insightinto the ideal priors for agents embedded within their universe. A satisfactory theory
of logical uncertainty could lend insight into general intelligence more broadly. An
ideal decision theory could reveal an ideal decision-making procedure for real agents
to approximate.
122 N. Soares and B. Fallenstein
But while these advancements may provide tools useful for designing intelligent
systems in general, they would make it markedly easier to design aligned systems in
particular. Developing a general theory of highly reliable decision-making, even if itis too idealized to be directly implemented, gives us the conceptual tools needed to
design and evaluate safe heuristic approaches. Conversely, if we must evaluate real
systems composed of practical heuristics before formalizing the theoretical problemsthat those heuristics are supposed to solve, then we will be forced to rely on our
intuitions.
This theoretical understanding might not be developed by default. Causal coun-
terfactual reasoning, despite being suboptimal, might be good enough to enable the
construction of a smarter-than-human system. Systems built from poorly understood
heuristics might be capable of creating or attaining superintelligence for reasons wedon’t quite understand—but it is unlikely that such systems could then be aligned
with human interests.
Sometimes theory precedes application, but sometimes it does not. The goal of
much of the research outlined in this agenda is to ensure, in the domain of superin-
telligence alignment—where the stakes are incredibly high—that theoretical under-
standing comes ﬁrst.
5.5.2 Why Start Now?
It may seem premature to tackle the problem of AI goal alignment now, with super-
intelligent systems still ﬁrmly in the domain of futurism. However, the authors think
it is important to develop a formal understanding of AI alignment well in advance ofmaking design decisions about smarter-than-human systems. By beginning our work
early, we inevitably face the risk that it may turn out to be irrelevant; yet failing to
make preparations at all poses substantially larger risks.
We have identiﬁed a number of unanswered foundational questions relating to the
development of general intelligence, and at present it seems possible to make some
promising inroads. We think that the most responsible course, then, is to begin assoon as possible.
Weld and Etzioni ( 1994 ) directed a “call to arms” to computer scientists, noting
that “society will reject autonomous agents unless we have some credible means ofmaking them safe.” We are concerned with the opposite problem: what if society
fails to reject systems that are unsafe? What will be the consequences if someone
believes a smarter-than-human system is aligned with human interests when it is
not?
This is our call to arms: regardless of whether research eﬀorts follow the path laid
out in this document, signiﬁcant eﬀort must be focused on the study of superintelli-
gence alignment as soon as possible.
5 Agent Foundations for Aligning Machine Intelligence . . . 123
References
Armstrong S (2015) AI motivated value selection, accepted to the 1st International Workshop on
AI and Ethics, held within the 29th AAAI Conference on Artiﬁcial Intelligence (AAAI-2015),Austin, TX
Armstrong S, Sandberg A, Bostrom N (2012) Thinking inside the box: Controlling and using an
oracle AI. Minds and Machines 22(4):299–324
Bárász M, Christiano P, Fallenstein B, Herreshoﬀ M, LaVictoire P, Yudkowsky E (2014) Robust
cooperation in the Prisoner’s Dilemma: Program equilibrium via provability logic, unpublishedmanuscript. Available via arXiv. http://arxiv.org/abs/1401.5577
Ben-Porath E (1997) Rationality, Nash equilibrium, and backwards induction in perfect-information
games. Review of Economic Studies 64(1):23–46
Bensinger R (2013) Building phenomenological bridges. Less Wrong Blog http://lesswrong.com/
lw/jd9/building\_phenomenological\_bridges/
Bird J, Layzell P (2002) The evolved radio and its implications for modelling the evolution of novel
sensors. In: Proceedings of the 2002 Congress on Evolutionary Computation. Vol. 2, IEEE,Honolulu, HI, pp 1836–1841
Bostrom N (2014) Superintelligence: Paths, Dangers, Strategies. Oxford University Press, New
York
Christiano P (2014a) Non-omniscience, probabilistic inference, and metamathematics. Tech. Rep.
2014–3, Machine Intelligence Research Institute, Berkeley, CA, http://intelligence.org/ﬁles/
Non-Omniscience.pdf
Christiano P (2014b) Specifying “enlightened judgment” precisely (reprise). Ordinary Ideas Blog
http://ordinaryideas.wordpress.com/2014/08/27/specifying-enlightened-judgment-precisely-reprise/
de Blanc P (2011) Ontological crises in artiﬁcial agents’ value systems. Tech. rep., The Singularity
Institute, San Francisco, CA, http://arxiv.org/abs/1105.3821
Demski A (2012) Logical prior probability. In: Bach J, Goertzel B, Iklé M (eds) Artiﬁcial General
Intelligence, Springer, New York, 7716, pp 50–59, 5th International Conference, AGI 2012,Oxford, UK, December 8–11, 2012. Proceedings
Fallenstein B (2014) Procrastination in probabilistic logic. Working paper, Machine Intelligence
Research Institute, Berkeley, CA, http://intelligence.org/ﬁles/ProbabilisticLogicProcrastinates.
pdf
Fallenstein B, Soares N (2014) Problems of self-reference in self-improving space-time embedded
intelligence. In: Goertzel B, Orseau L, Snaider J (eds) Artiﬁcial General Intelligence, Springer,New York, 8598, pp 21–32, 7th International Conference, AGI 2014, Quebec City, QC, Canada,August 1–4, 2014. Proceedings
Fallenstein B, Soares N (2015) Vingean reﬂection: Reliable reasoning for self-improving agents.
Tech. Rep. 2015–2, Machine Intelligence Research Institute, Berkeley, CA, https://intelligence.
org/ﬁles/VingeanReﬂection.pdf
Gaifman H (1964) Concerning measures in ﬁrst order calculi. Israel Journal of Mathematics 2(1):
1–18
Gaifman H (2004) Reasoning with limited resources and assigning probabilities to arithmetical
statements. Synthese 140(1–2):97–119
Gödel K, Kleene SC, Rosser JB (1934) On Undecidable Propositions of Formal Mathematical
Systems. Institute for Advanced Study, Princeton, NJ
Good IJ (1965) Speculations concerning the ﬁrst ultraintelligent machine. In: Alt FL, Rubinoﬀ M
(eds) Advances in Computers, vol 6, Academic Press, New York, pp 31–88
Halpern JY (2003) Reasoning about Uncertainty. MIT Press, Cambridge, MAHintze D (2014) Problem class dominance in predictive dilemmas. Tech. rep., Machine Intelligence
Research Institute, Berkeley, CA, http://intelligence.org/ﬁles/ProblemClassDominance.pdf
Hutter M (2000) A theory of universal artiﬁcial intelligence based on algorithmic complexity,
unpublished manuscript. Available via arXiv. http://arxiv.org/abs/cs/0004001
124 N. Soares and B. Fallenstein
Hutter M, Lloyd JW, Ng KS, Uther WTB (2013) Probabilities on sentences in an expressive logic.
Journal of Applied Logic 11(4):386–420
Jeﬀrey RC (1983) The Logic of Decision, 2nd edn. Chicago University Press, Chicago, ILJoyce JM (1999) The Foundations of Causal Decision Theory. Cambridge Studies in Probability,
Induction and Decision Theory, Cambridge University Press, New York, NY
Legg S, Hutter M (2007) Universal intelligence: A deﬁnition of machine intelligence. Minds and
Machines 17(4):391–444
Lehmann EL (1950) Some principles of the theory of testing hypotheses. Annals of Mathematical
Statistics 21(1):1–26
Lewis D (1979) Prisoners’ dilemma is a Newcomb problem. Philosophy & Public Aﬀairs 8(3):
235–240, http://www.jstor.org/stable/2265034
Lewis D (1981) Causal decision theory. Australasian Journal of Philosophy 59(1):5–30Łoś J (1955) On the axiomatic treatment of probability. Colloquium Mathematicae 3(2):125–137,
http://eudml.org/doc/209996
MacAskill W (2014) Normative uncertainty. PhD thesis, St Anne’s College, University of Oxford,
http://ora.ox.ac.uk/objects/uuid:8a8b60af-47cd-4abc-9d29-400136c89c0f
McCarthy J, Minsky M, Rochester N, Shannon C (1955) A proposal for the Dartmouth summer
research project on artiﬁcial intelligence. Proposal, Formal Reasoning Group, Stanford Univer-
sity, Stanford, CA
Muehlhauser L, Salamon A (2012) Intelligence explosion: Evidence and import. In: Eden A,
Søraker J, Moor JH, Steinhart E (eds) Singularity Hypotheses: A Scientiﬁc and PhilosophicalAssessment, Springer, Berlin, the Frontiers Collection
Ng AY, Russell SJ (2000) Algorithms for inverse reinforcement learning. In: Langley P (ed) Pro-
ceedings of the Seventeenth International Conference on Machine Learning (ICML-’00), Mor-gan Kaufmann, San Francisco, pp 663–670
Omohundro SM (2008) The basic AI drives. In: Wang P, Goertzel B, Franklin S (eds) Artiﬁcial
General Intelligence 2008, IOS, Amsterdam, no. 171 in Frontiers in Artiﬁcial Intelligence andApplications, pp 483–492, proceedings of the First AGI Conference
Pearl J (2000) Causality: Models, Reasoning, and Inference, 1st edn. Cambridge University Press,
New York, NY
Poe EA (1836) Maelzel’s chess-player. Southern Literary Messenger 2(5):318–326Rapoport A, Chammah AM (1965) Prisoner’s Dilemma: A Study in Conﬂict and Cooperation, Ann
Arbor Paperbacks, vol 165. University of Michigan Press, Ann Arbor, MI
Russell S (2014) Unifying logic and probability: A new dawn for AI? In: Information Processing
and Management of Uncertainty in Knowledge-Based Systems: 15th International Conference,IPMU 2014, Montpellier, France, July 15–19, 2014, Proceedings, Part I, Springer, no. 442 inCommunications in Computer and Information Science, pp 10–14
Sawin W, Demski A (2013) Computable probability distributions which converge on 𝜋
1will disbe-
lieve true 𝜋2sentences. Tech. rep., Machine Intelligence Research Institute, Berkeley, CA, http://
intelligence.org/ﬁles/Pi1Pi2Problem.pdf
Shannon CE (1950) XXII. Programming a computer for playing chess. Philosophical Magazine
41(314):256–275
Soares N (2014) Tiling agents in causal graphs. Tech. Rep. 2014–5, Machine Intelligence Research
Institute, Berkeley, CA, http://intelligence.org/ﬁles/TilingAgentsCausalGraphs.pdf
Soares N (2015) Formalizing two problems of realistic world-models. Tech. Rep.
2015–3, Machine Intelligence Research Institute, Berkeley, CA, https://intelligence.org/
ﬁles/RealisticWorldModels.pdf
Soares N (2016) The value learning problem. In: Ethics for Artiﬁcial Intelligence Workshop at the
25th International Joint Conference on Artiﬁcial Intelligence (IJCAI-16). New York, NY, July9th-15th
Soares N, Fallenstein B (2014) Toward idealized decision theory. Tech. Rep.
2014–7, Machine Intelligence Research Institute, Berkeley, CA, https://intelligence.org/
ﬁles/TowardIdealizedDecisionTheory.pdf
5 Agent Foundations for Aligning Machine Intelligence . . . 125
Soares N, Fallenstein B (2015) Questions of reasoning under logical uncertainty. Tech. Rep.
2015–1, Machine Intelligence Research Institute, Berkeley, CA, https://intelligence.org/ﬁles/
QuestionsLogicalUncertainty.pdf
Solomonoﬀ RJ (1964) A formal theory of inductive inference. Part I. Information and Control
7(1):1–22
United Kingdom Ministry of Defense (1991) Requirements for the procurement of safety critical
software in defence equipment. Interim Defence Standard 00-55, United Kingdom Ministry ofDefense
United States Department of Defense (1985) Department of Defense trusted computer system eval-
uation criteria. Department of Defense Standard DOD 5200.28-STD, United States Departmentof Defense, http://csrc.nist.gov/publications/history/dod85.pdf
Vinge V (1993) The coming technological singularity: How to survive in the post-human era. In:
Vision-21: Interdisciplinary Science and Engineering in the Era of Cyberspace, NASA LewisResearch Center, no. 10129 in NASA Conference Publication, pp 11–22, http://ntrs.nasa.gov/
archive/nasa/casi.ntrs.nasa.gov/19940022856.pdf
Wald A (1939) Contributions to the theory of statistical estimation and testing hypotheses. Annals
of Mathematical Statistics 10(4):299–326
Weld D, Etzioni O (1994) The ﬁrst law of robotics (a call to arms). In: Hayes-Roth B, Korf RE (eds)
Proceedings of the Twelfth National Conference on Artiﬁcial Intelligence, AAAI Press, MenloPark, CA, pp 1042–1047, http://www.aaai.org/Papers/AAAI/1994/AAAI94-160.pdf
Yudkowsky E (2008) Artiﬁcial intelligence as a positive and negative factor in global risk. In:
Bostrom N, Ćirković MM (eds) Global Catastrophic Risks, Oxford University Press, New York,pp 308–345
Yudkowsky E (2011) Complex value systems in Friendly AI. In: Schmidhuber J, Thórisson KR,
Looks M (eds) Artiﬁcial General Intelligence, Springer, Berlin, no. 6830 in Lecture Notes inComputer Science, pp 388–393, 4th International Conference, AGI 2011, Mountain View, CA,USA, August 3–6, 2011. Proceedings
Yudkowsky E (2013) The procrastination paradox. Brief technical note, Machine Intelligence
Research Institute, Berkeley, CA, http://intelligence.org/ﬁles/ProcrastinationParadox.pdf
Yudkowsky E (2014) Distributions allowing tiling of staged subjective EU maximizers. Tech.
rep., Machine Intelligence Research Institute, Berkeley, CA, http://intelligence.org/ﬁles/
DistributionsAllowingTiling.pdf
Yudkowsky E, Herreshoﬀ M (2013) Tiling agents for self-modifying AI, and the Löbian obstacle.
Early draft, Machine Intelligence Research Institute, Berkeley, CA, http://intelligence.org/ﬁles/
TilingAgents.pdf