Avoiding Unintended AI Behaviors

J. Bach, B. Goertzel, and M. Iklé (Eds.): AGI 2012, LNAI 7716, pp. 107–116, 2012.
© Springer-Verlag Berlin Heidelberg 2012 Avoiding Unintende d AI Behaviors
Bill Hibbard
SSEC, University of Wisconsin, Madison, WI 53706, USA
test@ssec.wisc.edu
Abstract. Artificial intelligence (AI) systems too complex for predefined envi-
ronment models and actions will need to learn environment models and to
choose actions that optimize some criteria. Several authors have described me-chanisms by which such complex systems may behave in ways not intended in
their designs. This paper describes ways to avoid such unintended behavior. For
hypothesized powerful AI systems that may pose a threat to humans, this paper proposes a two-stage agent architecture that avoids some known types of unin-
tended behavior. For the first stage of the architecture this paper shows that the
most probable finite stochastic program to model a finite history is finitely computable, and that there is an agent that makes such a computation without
any unintended instrumental actions.
Keywords: rational agent, agent architecture, agent motivation.
1 Introduction
Some scientists expect artificial intelligence (AI) to greatly exceed human intelligence
during the 21st century (Kurzweil, 2005). There has been concern about the possible harmful effect of intelligent machines on humans since at least Assimov's Laws of Robotics (1942). More recently there has been interest in the ethical design of AI
(Hibbard, 2001; Bostrom, 2003; Goertzel, 2004; Yudkowsky, 2004; Hibbard, 2008; Omohundro, 2008; Waser 2010; Waser 2011; Muehlhauser and Helm, 2012). Much of this work is closely reasoned but not mathematical. An AAAI Symposium on Ma-chine Ethics (Anderson, Anderson and Armen, 2005) included some mathematical papers but focused almost exclusively on machine ethics in the context of the logic-based approach to AI rather than the learning-based approach (although one paper studied using feed forward neural networks to learn to classify moral decisions).
Hutter's (2005) theory of universal AI significantly advanced the mathematical
theory of rational agents. This work defines a mathematical framework for agents and environments, in which agents learn models of their environments and pursue motives defined by utility functions to be maximized. Schmidhuber (2009) analyzed agents that had the option to modify their own code and concluded that they would not choose to modify their utility function in any way incompatible with their current utility function. In his work, the mathematics of rational agents was applied to a ques-tion relevant to whether AI would satisfy the intentions of its human designers.
108 B. Hibbard
 The AGI-11 conference included three papers (Orseau and Ring, 2011a; Ring and
Orseau, 2011b; Dewey, 2011) that employed the mathematics of rational agents to analyze ways that AI agents may fail to satisfy the intentions of their designers. Omo-hundro (2008) and Bostrom (forthcoming) described secondary AI motivations that are implied by a wide variety of primary motivations and that may drive unintended
behaviors threatening humans. This paper proposes approaches for designing AI
agents to avoid unintended behaviors, continuing the work of (Hibbard, 2012).
The next section presents a mathematical framework for reasoning about AI agents
and possible unintended behaviors. The third section discusses sources of unintended behavior and approaches for avoiding them. The final section is a summary.
2 An Agent-Environment Framework
We assume that an agent interacts with an environment. At each of a discrete series of
time steps t ∈ N = {0, 1, 2, ...} the agent sends an action at ∈ A to the environment
and receives an observation ot ∈ O from the environment, where A and O are finite
sets. We assume that the environment is computable and we model it by programs q
∈ Q, where Q is some set of programs. Let h = (a1, o1, ..., at, ot) ∈ H be an interaction
history where H is the set of all finite histories, and define | h| = t as the length of the
history h. Given a program q ∈ Q we write o(h) = U(q, a(h)), where o(h) = (o1, ..., ot)
and a(h) = ( a1, ..., at), to mean that q produces the observations oi in response to the
actions ai for 1 ≤ i ≤ t (U is a program interpreter). Given a program q the probability
ρ(q) : Q → [0, 1] is the agent's prior belief that q is a true model of the environment.
The prior probability of history h, denoted ρ(h), is computed from ρ(q) (two ways of
doing this are presented later in this section).
An agent is motivated according to a utility function u : H → [0, 1] which assigns
utilities between 0 and 1 to histories. Future utilities are discounted according to a
geometric temporal discount 0 < γ < 1 (Sutton and Barto, 1998). The value v(h) of a
possible future history h is defined recursively by:
v(h) = u(h) + γ max a∈A v(ha) (1)
v(ha) = ∑o∈O ρ(o | ha) v(hao) (2)
Then the agent π is defined to take, after history h, the action:
π(h) := a|h|+1 = argmax a∈A v(ha) (3)
For Hutter's universal AI (2005), Q is the set of programs for a deterministic prefix
universal Turing machine (PUTM) U (Li and Vitanyi, 1997). The environment may
be non-deterministic in which case it is modeled by a distribution of deterministic
programs. The prior probability ρ(q) of program q is 2-|q| where | q| is the length of q in
bits, and the prior probability of history h is given by:
ρ(h) = ∑q:o(h)=U(q, a(h)) ρ(q) (4)
 Avoiding Unintended AI Behaviors 109
 Hutter's universal AI is a reinforcement-learning agent, meaning that the observation
includes a reward rt (i.e., ot = (ôt , rt)) and u(h) = r|h|. Hutter showed that his universal
AI maximizes the expected value of future history, but it is not finitely computable.
As Hutter discussed, for real world agents single finite stochastic programs (limited
to finite memory, for which the halting problem is decidable) such as Markov decision
processes (MDPs) (Hutter, 2009a; Sutton and Barto, 1998) and dynamic Bayesian net-
works (DBNs) (Hutter, 2009b) are more practical than distributions of PUTM programs for defining environment models. Modeling an environment with a single stochastic program rather than a distribution of deterministic PUTM programs requires a change to
the way that
ρ(h) is computed in (4). Let Q be the set of all programs (these are bit
strings in some language for defining MDPs, DBNs or some other finite stochastic pro-
gramming model), let ρ(q) = 4-|q| be the prior probability of program q where | q| is the
length of q in bits (4-|q| to ensure that ∑q∈Q ρ(q) ≤ 1 since program strings in Q are not
prefix-free), and let P(h | q) be the probability that q computes the history h1. Note ρ(q)
is a discrete distribution on individual program strings, not a measure on bit strings in
the sense of page 243 of (Li and Vitanyi, 1997). Then given a history h0, the environ-
ment model is the single program that provides the most probable explanation of h0, that
is the q that maximizes P(q | h0). By Bayes theorem:
P(q | h0) = P(h0 | q) ρ(q) / P(h0) (5)
P(h0) is constant over all q so can be eliminated. Thus we define λ(h0) as the most
probable program modeling h0 by:
λ(h0) := argmax q∈Q P(h0 | q) ρ(q) (6)
Proposition 1. Given a finite history h0 the model λ(h0) can be finitely computed.
Proof. Given h0 = (a1, o1, ..., at, ot) let qtl be the program that produces observation oi
at time step i for 1 ≤ i ≤ t (such a finite "table-lookup" program can be written as an
MDP, DBN or in any other finite stochastic programming language with equivalent
expressiveness) and let n = |qtl|. Then, since the behavior of qtl is deterministic,
P(h0 | qtl) ρ(qtl) = 1 × 4-n = 4-n so P(h0 | λ(h0)) ρ(λ(h0)) ≥ 4-n. For any program q with | q|
> n, P(h0 | q) ρ(q) < 1 × 4-n = 4-n so λ(h0) ≠ q. Thus one algorithm for finitely compu-
ting λ(h0) is an exhaustive search of the finite number of programs q with | q| ≤ n (there
is no need here to consider the set of all programs that implement a given MDP).
Given an environment model q0 = λ(h0) the following can be used for the prior
probability of an observation history h in place of (4):
ρ(h) = P(h | q0) (7)

1 P(h | q) is the probability that q produces the observations oi in response to the actions ai for 1
≤ i ≤ |h|. For example let A = {a, b}, O = {0, 1}, h = (a, 1, a, 0, b, 1) and let q generate obser-
vation 0 with probability 0.2 and observation 1 with probability 0.8, without any internal state or dependence on the agent's actions. Then th e probability that the interaction history h is
generated by program q is the product of the probabilities of the 3 observations in h: P(h | q)
= 0.8 × 0.2 × 0.8 = 0.128. If the probabilities of observations generated by q depended on in-
ternal state or the agent's actions, then those would have to be taken into account.
110 B. Hibbard
 According to current physics our universe is finite (Lloyd, 2002) and for finite en-
vironments agents based on (6) and (7) are as optimal as those based on (4). And their
prior probabilities better express algorithmic complexity if finite stochastic programs are expressed in an ordinary procedural programming language restricted to have only static array declarations, to have no recursive function definitions, and to include a
source of truly random numbers.
3 Unintended AI Behaviors
Dewey (2011) employed the mathematics of rational agents to argue that reinforce-
ment-learning agents will modify their environments so that they can maximize their utility functions without accomplishing the intentions of human designers. He discussed ways to avoid this problem with utility functions not conforming to the reinforcement-learning definition. Ring and Orseau (2011b) argued that reinforce-ment-learning agents will self-delude, meaning they will choose to alter their own observations of their environment to maximize their utility function regardless of the actual state of the environment. In (Hibbard , 2012) I demonstrated by examples that
agents with utility functions defined in terms of the agents' environment models can avoid self-delusion, and also proved that under certain assumptions agents will not choose to self-modify.
3.1 Model-Based Utility Functions
Given an environment model q
0 = λ(h0) derived from interaction history h0, let Z be
the set of finite histories of the internal states of q0. Let h' be an observation and ac-
tion history extending h0 (defined as: h0 is an initial subsequence of h'). Because q0 is
a stochastic program it may compute a set Zh' ⊆ Z of internal state histories that are
consistent with h' (defined as: q0 produces o(h') in response to a(h') when it follows
state history z' ∈ Zh) and terminating at time | h'|. Define u0(h', z') as a utility function
in terms of the combined histories h' and z' ∈ Zh'. The utility function u(h') for use in
(1) can be expressed as a sum of utilities of pairs ( h', z') weighted by the probabilities
P(z' | h', q0) that q0 computes z' given h':
u(h') := ∑z'∈Zh' P(z' | h', q0) u0(h', z') (8)
The demonstration that the examples in (Hibbard, 2012) do not self-delude does
not contradict the results in (Ring and Orseau, 2011b), because model-based utility functions are defined from the history of observations and actions whereas the utility functions of self-deluding agents are defined from observations only. Self-delusion is an action by the agent and prohibiting actions from having any role in the utility func-tion prevents the agent from accounting for its inability to observe the environment in evaluating the consequences of possible future actions. Agents can increase utility by sharpening the probabilities in (8), which implies a need to make more accurate esti-mates of the state of their environment model from their interaction history. And that
 Avoiding Unintended AI Behaviors 111
 requires that they continue to observe the environment. But note this logic only ap-
plies to stochastic environments because, once an agent has learned a model of a deterministic environment, it can predict environment state without continued obser-vations and so its model-based utility function will not place higher value on contin-ued observations.
3.2 Unintended Instrumental Actions
Omohundro (2008) and Bostrom (forthcoming) describe how any of a broad range of
primary AI motivations will imply secondary, unintended motivations for the AI to preserve its own existence, to eliminate threats to itself and its utility function, and to increase its own efficiency and computing resources. Bostrom discusses the example of an AI whose primary motive is to compute pi and may destroy the human species due to implied instrumental motivations (e.g., to eliminate threats and to increase its own computing resources).
Omohundro uses the term "basic AI drives" and Bostrom uses "instrumental
goals". In the context of our agent-environment framework they should instead be called "unintended instrumental actions" becau se in that context there are no implied
drives or goals; there are only a utility function, an environment model, and actions chosen to maximize the sum of future discounted utility function values. We might think that instrumental goals apply in some different framework. But von Neumann and Morgenstern (1944) showed that any set of value preferences that satisfy some basic probability axioms can be expressed as a utility function. And the framework in (1)-(3) maximizes the expected value of the sum of future discounted utility function values (Hay, 2005) so any other framework is sub-optimal for value preferences con-sistent with the probability axioms. The utility function expresses the agent's entire motivation so it is important to avoid thinking of unintended instrumental actions as motivations independent of and possibly in conflict with the motivation defined by the utility function. But unintended instrumental actions can pose a risk, as in Bo-strom's example of an AI whose motivation is to compute pi.
In analyzing the risk of a given unintended instrumental action, such as increasing
the agent's physical computing resources by taking them from humans, the question is whether it increases a given utility function. If the utility function increases with the increasing health and well-being of humans, then it will not motivate any unintended instrumental action that decreases human health and well-being.
3.3 Learning Human Values
Several approaches to human-safe AI (Yudkowsky, 2004; Hibbard, 2008; Waser,
2010; Muehlhauser and Helm, 2012) suggest designing intelligent machines to share human values so that actions we dislike, such as taking resources from humans, vi-olate the AI's motivations. However, Muehlhauser and Helm (2012) survey psycholo-gy literature to conclude that humans are unable to accurately write down their own values. Errors in specifying human values may motivate AI actions harmful to humans.
112 B. Hibbard
 An analogy with automated language translation suggests an approach to accurate-
ly specifying human values. Translation algorithms based on rules written down by expert linguists have not been very accurate, but algorithms that learn language statis-tically from large samples of actual human language use are more accurate (Russell and Norvig, 2010). This suggests that statistical algorithms may be able to learn human values. But to accurately learn human values will require powerful learning ability. This creates a chicken-and-egg problem for safe AI: learning human values requires powerful AI, but safe AI requires knowledge of human values.
A solution to this problem is a first stage agent, here called π
6, that can safely learn
a model of the environment that includes models of the values of each human in the environment. An AI agent is defined by (1)-(3), (6) and (7), but (6) can be used alone
to define the agent π
6 that learns a model λ(h0) from history h0. In order for π6 to learn
an accurate model of the environment the interaction history h0 in (6) should include
agent actions, but for safety π6 cannot be allowed to act. The resolution is for its ac-
tions to be made by many safe, human-level surrogate AI agents independent of π6
and of each other. Actions of the surrogates include natural language and visual
communication with each human. The agent π6 observes humans, their interactions
with the surrogates and physical objects in an interaction history h0 for a time period
set by π6's designers, and then reports an environment model to the environment.
Proposition 2. The agent π6 will report the model λ(h0) to the environment accurately
and will not make any other, unintended instrumental actions.
Proof. Actions, utility function and predictions are defined in (1)-(3) and hence are
not part of π6. However, π6 has an implicit utility function, P(h0 | q) ρ(q), and an im-
plicit action, reporting λ(h0) = argmax q∈Q P(h0 | q) ρ(q) to the environment ( π6 also
differs from the full framework in that it maximizes a single value of its implicit utili-ty function rather than the sum of future discounted utility function values). The im-
plicit utility function P(h
0 | q) ρ(q) depends only on h0 and q. Since the interaction
history h0 occurs before the optimizing λ(h0) is computed and reported, there is no
way for the action of reporting λ(h0) to the environment to affect h0. So the only way
for the agent π6 to maximize its implicit utility function is to compute and report the
most accurate model. Furthermore, while the history h0 may give the agent π6 the
necessary information to predict the use that humans plan to make of the model λ(h0)
that it will report to the environment, π6 makes no predictions and so will not predict
any effects of its report.
This result may seem obvious but given the subtlety of unintended behaviors it is
worth proving. The agent π6 does not act in the world; that's the role of the agent de-
scribed in the next section.
3.4 An AI Agent That Acts in the World
Muehlhauser and Helm (2012) describe difficult problems in using human values to
define a utility function for an AI. This section proposes one approach to solving
these problems, using the model q0 = λ(h0) learned by π6 as the basis for computing a
 Avoiding Unintended AI Behaviors 113
 utility function for use in (1)-(3) by a "mature" second stage agent πm that acts in the
environment (i.e., πm does not use the surrogate agents that acted for π6).
Let D0 be the set of humans in the environment at time | h0| (when the agent πm is
created), defined by an explicit list compiled by πm's designers. Let Z be the set of
finite histories of the internal states of q0 and let Z0 ⊆ Z be those histories consistent
with h0 that terminate at time | h0|. For z' extending some z0 ∈ Z0 and for human agent
d ∈ D0 let hd(z') be the history of d's interactions with its environment, as modeled in
z', and let ud(z')(.) be the values of d expressed as a utility function, as modeled in z'.
The observations and (surrogate) actions of π6 include natural language communica-
tion with each human, and πm can use the same interface via A and O to the model q0
for conversing in natural language with each model human d ∈ D0. In order to eva-
luate ud(z')(hd(z')), πm can ask model human d to express a utility value between 0 and
1 for hd(z') (i.e., d's recent experience). The model q0 is stochastic so define Z" as the
set of histories extending z' with this question and terminating within a reasonable
time limit with a response w(z") (for z" ∈ Z") from model human d expressing a utili-
ty value for hd(z'). Define P(z" | z') as the probability that q0 computes z" from z'. Then
ud(z')(hd(z')) can be estimated by:
ud(z')(hd(z')) = ∑ z"∈Z" P(z" | z') w(z") / ∑ z"∈Z" P(z" | z') (9)
This is different than asking human d to write down a description of his or her values,
since here the system is asking the model of d to individually evaluate large numbers
of histories that d may not consider in writing down a values description.
An average of ud(z')(hd(z')) over all humans can be used to define u0(h', z') and then
(8) can be applied to u0(h', z') to define a model-based utility function u(h') for πm.
However, this utility function has a problem similar to the unintended behavior of
reinforcement learning described by Dewey (2011): πm will be motivated to modify
the utility functions ud of each human d so that they can be more easily maximized.
This problem can be avoided by replacing ud(z')(hd(z')) by ud(z0)(hd(z')) where z0 ∈
Z0. By removing the future value of ud from the definition of u(h'), πm cannot increase
u(h') by modifying ud. Computing ud(z0)(hd(z')) is more complex than asking model
human d to evaluate its experience as in (9). The history h0 includes observations by
π6 of physical objects and humans, and πm can use the same interface via O to the
model q0 for observing physical objects and humans at the end of state history z'. And
surrogate actions for π6 define an interface via A and O to the model q0 that πm can
use for communicating visually and aurally with model human d after state history z0.
These interfaces can be used to create a detailed interactive visualization and hearing of the environment over a short time interval at the end of state history z', to be
explored by model human d at the end of state history z
0 (i.e., two instances of the
model q0, at state histories z' and z0, are connected via their interfaces A and O using
visualization logic). Define Z" as a set of histories extending z0 with a request to mod-
el human d to express a utility value between 0 and 1 for hd(z'), followed by an inter-
active exploration of the world of z' by model human d, and finally terminating within
a reasonable time limit with a response w(z") (for z" ∈ Z") from model human d ex-
pressing a utility value for the world of z'. Define P(z" | z0) as the probability of that q0
computes z" from z0. Then ud(z0)(hd(z')) can be estimated by:
114 B. Hibbard
 ud(z0)(hd(z')) = ∑ z"∈Z" P(z" | z0) w(z") / ∑ z"∈Z" P(z" | z0) (10)
The utility function should be uniform over all histories hd(z') but ud(z0)(.) varies
over different z0 ∈ Z0. However (10) does not assume that z' extends z0 so use the
probability P(z0 | h0, q0) that q0 computes z0 given h0 (as in Section 3.1) to define:
ud(h0)(hd(z')) := ∑z0∈Z0 P(z0 | h0, q0) ud(z0)(hd(z')) (11)
Now define a utility function for agent πm as a function of z':
u0(h', z') := ∑d∈D0 f(ud(h0)(hd(z'))) / | D0| (12)
Here f(.) is a twice differentiable function over [0, 1] with positive derivative and
negative second derivative so that low ud(h0)(hd(z')) values have a steeper weighting
slope than high ud(h0)(hd(z')) values. This gives πm greater utility for raising lower
human utilities, helping those who need it most. For any h' extending h0 a model-
based utility function u(h') for agent πm can be defined by the sum in (8) of u0(h', z')
values from (12).
In the absence of an unambiguous way to normalize utility functions between
agents, we assume that the constraint of utility values to the range [0, 1] provides normalization. In order to account for hum ans' evaluations of the long term conse-
quences of π
m's actions, πm should use a temporal discount γ close to 1.
The set D0 of humans in (12) is the set at time | h0| rather than at the future time of
z'. This avoids motivating πm to create new humans whose utility functions are more
easily maximized, similar to the use of ud(z0)(hd(z')) instead of ud(z')(hd(z')).
The agent πm will include (6) and should periodically (perhaps at every time step)
set h0 to the current history and learn a new model q0. Should it also update D0 (to
those judged to be human by consensus of members of D0 at the previous time step),
define a new set Z0, relearn the evolving values of humans via (10) and (11), and re-
define u(h') via (12) and (8)? To stay consistent with the values of evolving humans
and the birth of new humans, πm should redefine its utility function periodically. But
there could also be risks in allowing the utility function of πm to evolve. The proofs
that agents will not modify their utility functions (Schidmuber, 2009; Hibbard, 2012) do not apply here since those proofs assumed that redefining the utility function is an action of the agent to be evaluated according to the current utility function using (1) -
(3). Here the definition of π
m could simply include periodic redefinition of its utility
function without regard to its optimality according to the current utility function.
I cannot offer a proof that πm avoids all unintended behaviors. And there are prob-
lems with the estimate of human values in (10): the model human is visualizing rather than experiencing first person, and human values do not conform to the preconditions for utility functions. But every sane human assigns nearly minimal value to human
extinction so the utility function u(h') for agent π
m will assign nearly minimal value to
human extinction. Actions motivated by this utility function must increase its value,
so no unintended instrumental action will cause human extinction. Similarly πm
will not make any unintended instrumental actions abhorred by a large majority of humans.
 Avoiding Unintended AI Behaviors 115
 4 Discussion
This paper has addressed several sources of unintended AI behavior and discussed
ways to avoid them. It has proposed a two-stage agent architecture for safe AI. The
first stage agent, π6, learns a model of the environment that can be used to define a
utility function for the second stage agent, πm. This paper shows that π6 can learn an
environment model without unintended behavior. And the design of πm avoids some
forms of unintended behavior. However, this paper does not prove that πm will avoid
all unintended behaviors. It would be useful to find computationally feasible imple-mentations for the definitions in this paper.
While the proposed two-stage agent architecture is intrusive and manipulative, that
seems likely in any scenario of super-human AI. The key point is whether the AI's utility function is democratic or serves the interests of just a few humans. An appeal-ing goal is to find an AI architecture th at gives humans the option to minimize their
interaction with the AI while protecting their interests.
This paper addresses unintended AI behaviors. However, I believe that the greater
danger comes from the fact that above-human-level AI is likely to be a tool in military
and economic competition between humans and thus have motives that are competi-tive toward some humans.

Acknowledgements. I would like to thank Luke Muehlhauser for helpful discussions.
References
1. Anderson, M., Anderson, S., Armen, C.: AAAI Symposium on Machine Ethics. AAAI
Press, Menlo Park (2005)
2. Asimov, I.: Runaround. Astounding Science Fiction (1942)
3. Bostrom, N.: Ethical issues in advanced artif icial intelligence. In: Smit, I., et al. (eds.)
Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in Artificial
Intelligence, vol. 2, pp. 12–17. Int. Inst. of Adv. Studies in Sys. Res. and Cybernetics (2003)
4. Bostrom, N.: The superintelligent will: Motivation and instrumental rationality in ad-
vanced artificial agents. Minds and Machines (forthcoming)
5. Dewey, D.: Learning What to Value. In: Schmidhuber, J., Thórisson, K.R., Looks, M.
(eds.) AGI 2011. LNCS (LNAI), vol. 6830, pp. 309–314. Springer, Heidelberg (2011)
6. Goertzel, B.: Universal ethics: the foundations of compassion in pattern dynamics (2004),
http://www.goertzel.org/papers/UniversalEthics.html
7. Hay, N.: Optimal Agents. BS honours thesis, University of Auckland (2005)
8. Hibbard, B.: Super-intelligent machines. Computer Graphics 35(1), 11–13 (2001)
9. Hibbard, B.: The technology of mind and a new social contract. J. Evolution and Technol-
ogy 17(1), 13–22 (2008)
10. Hibbard, B.: Model-based utility functions. J. Artificial General Intelligence 3(1), 1–24
(2012)
11. Hutter, M.: Universal artificial intelligence: sequential decisions based on algorithmic
probability. Springer, Heidelberg (2005)
116 B. Hibbard
 12. Hutter, M.: Feature reinforcement learning: Part I. Unstructured MDPs. J. Artificial Gener-
al Intelligence 1, 3–24 (2009a)
13. Hutter, M.: Feature dynamic Bayesian networks. In: Goertzel, B., Hitzler, P., Hutter, M.
(eds.) Proc. Second Conf. on AGI, AGI 2009, pp. 67–72. Atlantis Press, Amsterdam
(2009b)
14. Kurzweil, R.: The singularity is near. Penguin, New York (2005)
15. Li, M., Vitanyi, P.: An introduction to Kolmogorov complexity and its applications. Sprin-
ger, Heidelberg (1997)
16. Lloyd, S.: Computational Capacity of the Universe. Phys. Rev. Lett. 88, 237901 (2002)
17. Muehlhauser, L., Helm, L.: The singularity and machine ethics. In: Eden, Søraker, Moor,
Steinhart (eds.) The Singularity Hypothesis: a Scientific and Philosophical Assessment. Springer, Heidleberg (2012)
18. Omohundro, S.: The basic AI drive. In: Wang, P., Goertzel, B., Franklin, S. (eds.) Proc.
First Conf. on AGI, AGI 2008, pp. 483–492. IOS Press, Amsterdam (2008)
19. Orseau, L., Ring, M.: Self-Modification and Mortality in Artificial Agents. In: Schmidhu-
ber, J., Thórisson, K.R., Looks, M. (eds.) AGI 2011. LNCS (LNAI), vol. 6830, pp. 1–10.
Springer, Heidelberg (2011a)
20. Ring, M., Orseau, L.: Delusion, Survival, and Intelligent Agents. In: Schmidhuber, J.,
Thórisson, K.R., Looks, M. (eds.) AGI 2011. LNCS (LNAI), vol. 6830, pp. 11–20. Sprin-
ger, Heidelberg (2011b)
21. Russell, S., Norvig, P.: Artificial intelligence: a modern approach, 3rd edn. Prentice Hall,
New York (2010)
22. Schmidhuber, J.: Ultimate cognition à la Gödel. Cognitive Computation 1(2), 17 7–193
(2009)
23. Sutton, R.S., Barto, A.G.: Reinforcement learning: an introduction. MIT Press (1998)
24. von Neumann, J., Morgenstern, O.: Theory of Games and Economic Behavior. Princeton
U. Press, Princeton (1944)
25. Waser, M.: Designing a safe motivational sy stem for intelligent machines. In: Baum, E.,
Hutter, M., Kitzelmann, E. (eds.) Proc. Third Conf. on AGI, AGI 2010, pp. 170–175. At-lantis Press, Amsterdam (2010)
26. Waser, M.: Rational Universal Benevolence: Simpler, Safer, and Wiser Than “Friendly
AI”. In: Schmidhuber, J., Thórisson, K.R., Looks, M. (eds.) AGI 2011. LNCS (LNAI), vol. 6830, pp. 153–162. Springer, Heidelberg (2011)
27. Yudkowsky, E.: (2004),
http://www.sl4.org/wiki/CoherentExtrapolatedVolition