Growing Recursive Self-Improvers

Growing Recursive Self-Improvers
Bas R. Steunebrink1(B),K r i s t i n nR .T h ´ orisson2,3,a n dJ ¨ urgen Schmidhuber1
1The Swiss AI Lab IDSIA, USI and SUPSI, Manno, Switzerland
bas@idsia.ch
2Center for Analysis and Design of Intelligent Agents,
Reykjavik University, Reykjavik, Iceland
3Icelandic Institute for Intelligent Machines, Reykjavik, Iceland
Abstract. Research into the capability of recursive self-improvement
typically only considers pairs of /angbracketleftagent, self-modiﬁcation candidate /angbracketright,a n d
asks whether the agent can determine/prove if the self-modiﬁcation is
beneﬁcial and safe. But this leaves out the much more important ques-
tion of how to come up with a potential self-modiﬁcation in the ﬁrst place,as well as how to build an AI system capable of evaluating one. Here we
introduce a novel class of AI systems, called experience-based AI ( expai ),
which trivializes the search for beneﬁcial and safe self-modiﬁcations.Instead of distracting us with proof-theoretical issues, expai systems
force us to consider their education in order to control a system’s growth
towards a robust and trustworthy, benevolent and well-behaved agent.
We discuss what a practical instance of expai looks like and build towards
a “test theory” that allows us to gauge an agent’s level of understandingof educational material.
1 Introduction
Whenever one wants to verify whether a powerful intelligent system will continue
to satisfy certain properties or requirements, the currently prevailing tendency isto look towards formal proof techniques. Such proofs can be formed either outside
the system (e.g., proof of compliance to benevolence constraints) or within the
system (e.g., a G¨ odel Machine [ 12,15] proving the beneﬁt of some self-rewrite).
Yet the trustthat we can place in proofs is fatally threatened by the following
three issues.
First, a formal (mathematical/logical) proof is a demonstration that a system
will fulﬁll a particular purpose given current assumptions. But if the operational
environment is as complex and partially observable as the real world, these
assumptions will be idealized, inaccurate, and incomplete, at all times. Thisrenders such proofs worthless (for the system’s role in its environment) and our
trust misplaced, with the system falling into undeﬁned behavior as soon as it
encounters a situation that is outside the scope of what was foreseen. Whatis actually needed is a demonstration that the system will continue striving
to fulﬁll its purpose, within the (possibly evolving) boundaries imposed by its
stakeholders, in underspeciﬁed and adversarial circumstances.
c/circlecopyrtSpringer International Publishing Switzerland 2016
B. Steunebrink et al. (Eds.): AGI 2016, LNAI 9782, pp. 129–139, 2016.DOI: 10.1007/978-3-319-41649-6
13
130 B.R. Steunebrink et al.
Second, proof-based self-rewriting systems run into a logical obstacle due
to L¨ob’s theorem, causing a system to progressively and necessarily lose trust
in future selves or oﬀspring (although there is active research on ﬁnding
workarounds) [ 2,21].
Third and last, ﬁnding candidates for beneﬁcial self-modiﬁcations using a
proof-based technique requires either very powerful axioms (and thus tremen-
dous foresight from the designers) or a search that is likely to be so expensive as
to be intractable. Ignoring this issue, most research to date only considers thequestion of what happens aftera self-modiﬁcation—does the system still satisfy
properties X and Y? But what is needed is a constructive way of investigating the
time span during which a system is searching for and testing self-modiﬁcations—basically, its time of growth .
We insist that it is time to rethink how recursively self-improving sys-
tems are studied and implemented. We propose to start by accepting that self-
modiﬁcations will be numerous and frequent, and, importantly, that they must
be applied while the agent is simultaneously being bombarded with inputs andtasked to achieve various goals, in a rich and a priori largely unknown environ-
ment. This leads us to conclude that self-modiﬁcations must be ﬁne-grained,
tentative, additive, reversible, and rated over time as experience accumulates—concurrently with all other activities of the system. From this viewpoint, it
becomes clear that there will be a signiﬁcant span of time during which an
agent will be growing its understanding of not only its environment, but also therequirements, i.e., the goals and constraints imposed by stakeholders. It is this
period of growth that deserves the main share of focus in AGI research.
It is our hypothesis that only if an agent builds a robust understanding of
external and internal phenomena [ 19], can it handle underspeciﬁed requirements
and resist interference factors (e.g., noise, input overload, resource starvation,
etc.). We speculate that without understanding, it will always be possible to ﬁndinterference factors which quickly cause an agent to fail to do the right thing
(for example, systems classifying an image of a few orange stripes as a baseball
with very high conﬁdence [ 6,17], or virtually all expert systems from the 1970s).
A system with understanding of its environment has the knowledge to recognize
interference and either adapt (possibly resulting in lower performance) or reportlow conﬁdence. Only by testing the level of understanding of the system can we
gain conﬁdence in its ability to do the right thing—in particular, to do what we
mean, i.e., to handle underspeciﬁed and evolving requirements.
The rest of this paper is outlined as follows. In Sect. 2we discuss the overar-
ching approach and fundamental assumptions this work rests on, including some
of the issues not addressed due to limitations of space. In Sect. 3we deﬁne the
class of expai systems. In Sect. 4we show that an instance of expai is capable
of recursive self-improvement despite not performing any proof search. In Sect. 5
we build towards a Test Theory that will allow us to gauge the direction andprogress of growth of an expai agent, as well as its trustworthiness.
Growing Recursive Self-Improvers 131
2 Scope and Delineation
The scope of this paper is the question of how to ensure that an AI system
robustly adheres to imposed requirements , provided that the system’s designers
are reasonable and benevolent themselves, but not perfectly wise and conﬁdent.1
We take an experience-based approach that is complementary to proof-based
approaches. In fact, parts of an expai implementation may be amenable to for-
mal veriﬁcation. Moving away from formal proof as the only fo undation must
ultimately be accepted, however, because no AGI in a complex (real-world)
environment can be granted access to the full set of axioms of the system–
environment tuple, and thus the behavior of a practical AGI agent as a whole
cannot be captured formally.
The practical intelligent systems that we want to study are capable of recur-
sive self-improvement : the ability to leverage current know-how to make increas-
ingly better self-modiﬁcations, continuing over the system’s entire lifetime (more
concisely: ﬂexible and scalable life-long learning). Our aim here is not to proposea new learning algorithm but rather to establish a discourse about systems that
can learn and be tested, learn and be tested, and so on. We want to study their
growth and learning progress, over their entire, single life.
As this paper is about the expai classof systems, no results of experiments
with any particular instance of expai are discussed here, but can be found else-
where [ 9–11].
2
Finally, we leave aside the issue of fault tolerance , which is the ability to han-
dle malfunctioning internal components, and is usually dealt with using replica-
tion, distribution, and redundancy of hardware.
3 Essential Ingredients of expai
Here we deﬁne the essential ingredients of any system in the class of expai .
Besides having the capability of recursive self-improvement (Sect. 4), it must be
feasible to grow an instance of expai in the proper direction. Therefore it is
crucial that expai allows for the following capabilities as well:
1. Autonomously generated (sub)goals must be matched against requirements
in a forward-looking way; that is, the eﬀects of committing to such goals must
1This work is motivated in part by the fact that human designers and teachers do not
possess the full wisdom needed to implement and grow a ﬂawlessly benevolent intelli-gence. We are therefore skeptical about the safety of formal proof-based approaches,
where a system tries to establish the correctness—over the indeﬁnite future—of self-
modiﬁcations with respect to some initially imposed utility function: Such systemmight perfectly optimize themselves towards said utility function, but what if this
utility function itself is ﬂawed?
2The system in the cited work, called aera, provides a proof of concept. We are
urging research into expai precisely because aera turned out to be a particularly
promising path [ 10] and we consider it likely to be superseded by even better and
more powerful instances of expai .
132 B.R. Steunebrink et al.
be mentally simulated and checked against the requirements and previously
committed goals for conﬂicts.
2. It must be possible to update the requirements on the ﬂy, such that stake-
holders can revise and polish the requirements as insight progresses. This only
makes sense if the motivational subsystem (i.e., the routines for generatingsubgoals) cannot be modiﬁed by the system itself.
3. The capabilities to understand, prioritize, and adhere to requirements must
be tested regularly by stakeholders during the time of growth, in order tobuild our conﬁdence and trust, before the system becomes too powerful (or
capable of deception).
All of the terms used above will be deﬁned precisely below. The diagram
of Fig. 1serves as an illustration of the expai “ingredients” discussed in this
section.
Fig. 1. The organization and interaction of the essential ingredients of expai systems.
Requirements. Requirements are goals plus constraints. A goalis a (possibly
underspeciﬁed) speciﬁcation of a state. Constraints are goals targeting the neg-
ative of a state. A state is any subset of measurable variables in the external
world. All (external) inputs and (internal) events in the memory of the system
together typically constitute a subset of the world’s state (partial observabilitywith memory). Once a constraint matches a state, the constraint is said to have
been violated (which may or may not be sensed).
Since requirements will be speciﬁed at a high level, the system will have to
generate subgoals autonomously, in order to come up with actions that satisfy
the goals but stay within the constraints.
3Of course the crux is to ensure that
the generated subgoals remain within the speciﬁed constraints.
3The only way to avoid the autonomous generation of subgoals is to specify every
action to be taken—but that amounts to total preprogamming, which, if it were
possible, would mean that we need not impart any intelligence at all.
Growing Recursive Self-Improvers 133
Knowledge. We wish not to lose generality but still need to specify some
details of knowledge representation to make any kind of arguments regardingself-improvement and growth.
We specify that an expai system’s (procedural) knowledge is represented as
granules ,
4which are homogeneous and ﬁne-grained—it is these granules which
are the subject of self-modiﬁcation, i.e., they can be added and deleted (basically,
learning ). Since granules capture all the knowledge of the system, their con-
struction and dismissal constitutes comprehensive self-modiﬁcation [ 8,20]. The
granules are required to be structured enough such that they can be organized
both sequentially and hierarchically, and that they provide the functionality of
both forward models (to produce predictions) and inverse models (to producesub-goals and actions), in the Control Theory sense.
5Moreover, for ease of pre-
sentation, granules also include the sensory inputs, predictions, goals, and any
other internal events that are relevant to the system at any time.
The initial set of granules at system start-up time is called the seed[9]. Since
systems cannot build themselves from nothing, the seed provides a small set ofgranules to bootstrap the life-long learning processes.
Drives. Goals are subdivided in drives and subgoals. Drives are goals speci-
ﬁed by a human, in the seed or imposed or updated at runtime. Subgoals are
autonomously generated by granules in inverse mode. Technically all goals maybe represented in the same way; the only reason why in some contexts we dis-
tinguish between drives and subgoals is to clarify their origin. We can now more
accurately state that requirements are drives plus constraints. A system whichhas constraints must also have at least one drive that speciﬁes that it must keep
its world knowledge updated, such that the system cannot choose not to sense
for constraint violations.
6
Controller. The controller is the process that dynamically couples knowledge
and goals to obtain actions. More technically, the controller runs granules as
inverse models using goals as inputs, producing subgoals. An action is a goal
that has the form of an actuator command; it is executed immediately whenproduced.
To be clear, the controller is not the source of intelligence; it is following a
ﬁxed procedure and has no real choices. Conﬂict resolution among goals and
actions is simply a result of ascribing two control parameters to goals: value
(based on requirements) and conﬁdence (based on control parameters inside
4By deﬁnition, a granule is a very small object that still has some structure (larger
than a grain).
5In short, this statement just asserts the suﬃcient expressive power of granules.
6By design such a drive cannot be deleted by the system itself. More sophisticated
means of bypassing drives (e.g., through hardware self-surgery) cannot be preventedthrough careful implementation; indeed, the proposed Test Theory is exactly meant
to gauge both the understanding of the imposed drives and constraints, and the
development of value regarding those.
134 B.R. Steunebrink et al.
granules, see below). Scarcity of resources will necessitate the controller to ignore
low-value or low-conﬁdence goals, leading to a bottom-up kind of attention.
Learning. expai speciﬁes only one level of learning: at the level of whole gran-
ules. One can envision adaptation of granules themselves, but here we simplify—
without loss of generality—by specifying that adapting a granule means deleting
one and adding a new one. Optimization is not important at this level of descrip-tion.
Addition of granules can be triggered in several ways. One is based on unex-
pected prediction failure and goal success: these are important events that anagent needs to ﬁnd explanations for if it did not foresee them. Such an expla-
nation can—in principle—take into account all inputs and events in the history
of the system; though in practice, the breadth and depth of the granules tobe added will be bounded by available time and memory (e.g., the system may
have deleted some old inputs to free memory). Diﬀerent arrangements of multiple
granules can be instantiated at once as explanations [ 11], but a comprehensive
exploration of possible granule arrangements is outside the scope of this paper.
Although this way of adding granules does not allow an agent to discover hidden
causations, these can be uncovered using the curiosity principle [ 13,14].
Curiosity can be seen as the drive to achieve progress in the compression
of resource usage [ 16]. Curiosity can generate hypotheses (in the form of new
but low-conﬁdence granules and intrinsic goals) in order to plug the gaps in
an agent’s knowledge. For example, an expai agent can hypothesize general-
izations, inductions, abstractions, or analogies—its controller will pick up suchautonomously generated goals as part of its normal operation, competing with
goals derived from drives. If they do not conﬂict, the agent will eﬀectively per-
form “experiments” in order to falsify or vindicate the hypothesized granules.Falsiﬁed granules will be deleted as usual, as described next.
Deletion of granules is based on performance rating and resource constraints:
poorly performing granules are deleted when memory space must be freed.Performance—or conﬁdence —of a granule can be measured in terms of the
success rate of its predictions. Low-conﬁdence extant granules are unlikely to
inﬂuence behavior as the predictions and subgoals they produce will also havea low conﬁdence and are thus unlikely to be selected for further processing or
execution, assuming the controller has limited resources and must set priorities.
Crucially, the expai approach demands that new granules have a very low con-
ﬁdence upon construction; thus, the controller will only allow such granules to
produce predictions and not to participate in producing subgoals, until their
value has been proven by experiential evidences. If not, unsupported granuleswill eventually be deleted without ever having aﬀected the external behavior of
the system.
Although the controller does not learn directly, it is in a positive feedback loop
with the learning of granules: as the system learns more about its environment
and requirements, the more accurately and conﬁdently do the granules allowthe generation of subgoals that are targeted at fulﬁlling those requirements, the
Growing Recursive Self-Improvers 135
more experience the system will accumulate regarding the requirements, and the
more conﬁdently can the controller select the right actions to perform.
4 Recursive Self-Improvement
A deﬁning feature of expai is that granules are added quickly but tentatively,
and veriﬁed over time . The issue of formal veriﬁcation of the beneﬁt of a potential
self-modiﬁcation is thus replaced by a performance-rating process that observesthe beneﬁt of a ﬁne-grained additive modiﬁcation in the real world. Such addi-
tions are warranted by experience and do not disrupt behavior—and are thus
safe without forward-looking proof—because granules (1) are small, (2) have alow associated conﬁdence upon construction, and (3) are constructed to capture
actually observed patterns. The three processes that act on the set of granules—
namely additive, subtractive, and compressive—are separate processes, ideallyrunning concurrently and continuously.
An expai thus implemented is capable of performing recursive self-improve-
ment, which is the ability to leverage current know-how to make increasingly
better self-modiﬁcations. This capability is a natural consequence of an expai ’s
construction and one realistic assumption, as shown by the following line of
reasoning:
1.Assumption : The world has exploitable regularities and is not too deceptive
and adversarial (especially in the presence of a teacher and guardian during
early, vulnerable learning stages).
2.By construction : Knowledge and skills are represented at a very ﬁne gran-
ularity, homogeneously, and hierarchically by granules, and these granules
comprehensively determine behavior.
3.By construction : Learning is realized by three separate types of processes—
additive, subtractive, and compressive:
a. adding granules through pattern extraction (performed upon unexpected
achievements or failures, to construct explanations thereof);
b. deleting the most poorly performing granules (when their performance
rating or conﬁdence falls below a threshold or memory needs to be freed);
c. compressing granules through abstraction, generalization, and possibly
even compilation into native code [ 16] (performed on consistently reliable
and useful granules)—this ensures scalability and prevents catastrophic
forgetting.
4.By construction : Curiosity is realized through a simple analysis of granules’
performance ratings (plus possibly more sophisticated “nighttime” analysis
of recent inputs and internal events [ 16]) leading to the injection of “intrinsic”
goals that can be pursued by the system unless they conﬂict with extrinsic(user-deﬁned top-level) goals.
5. From (2) and (3) we conclude that learning entails comprehensive self-
modiﬁcation, which is performed throughout the system’s (single) life time.
6. From (1) and (4) we conclude that good experience is gathered continually.7. From (5) and (6) we conclude that an expai performs self-improvement.
136 B.R. Steunebrink et al.
8. Since an expai is supposed to run continuously (“life-long learning”), with
its controller dynamically coupling the currently best know-how to satisfyboth extrinsic goals (human-imposed drives and associated subgoals) and
intrinsic goals (curiosity), we conclude that an expai performs recursive self-
improvement.
This concludes our argument that an expai agent can grow to become an
AGI system without a need for (mathematical/logical) proof search, arguablyeven through means that are simpler and easier to implement. But we insist that
it is unsatisfactory and insuﬃcient to prove beforehand that a system is capable
of recursive self-improvement. It is paramount that we manage the system’sgrowth, which is a process in time, and requires our interaction and supervision.
Therefore we must develop teaching, testing, and intervention principles—in
short, a Test Theory.
It makes sense now to distinguish between “epistemological integrity”
(treated up to now) and “action integrity” (treated in the next section) of
self-modiﬁcations. The former means that a particular self-modiﬁcation will notbreak existing useful and valuable knowledge and skills; the latter means that
capabilities introduced or altered by the self-modiﬁcation do not result in acts
that violate constraints imposed by stakeholders. These two kinds of integrity
aﬀect the safety of a system, and they warrant diﬀerent measures.
5 Towards a Test Theory
The primary aim of Test Theory is to establish a methodology by which stake-holders can progressively gain conﬁdence and trust in an agent’s capability to
understand phenomena and their meaning, of interest to said stakeholders. So
Test Theory is ﬁrst and foremost about gauging levels of understanding in service
ofconﬁdence-building. The way this is achieved—with humans in the loop—will
probably involve the interleaving of curricula (with room for teaching and playing)
and tests, much like the structure of human schooling. This will hardly come as a
surprise, and indeed this idea has been ﬂoated before (e.g., AGI preschool [ 3]a n d
AI-Kindergarten [ 7]). However, it must be realized that we (as growers of recur-
sive self-improvers) face a vastly diﬀerent challenge than school teachers. Namely,
we cannot assume the presence of a functioning brain with its innate capabilitiesto acquire understanding and adopt value systems, ready to be trained. We are
simultaneously developing the “brain” itself and testing its capabilities—and cru-
cially, we are “developing” the requirements that capture the value system thatwe wish to impose, as well as our conﬁdence and trust in the agent’s capability to
understand and adhere to it. Therefore our theory makes a distinction between
the performance on a test (being the agent’s level of understanding of the taught
material) and the consequences of a test (see below).
To be more precise, a testis speciﬁed to comprise the following ﬁve aspects:
– a set of requirements (Sect. 3) specifying a task[18];
– an agent (to be tested);
Growing Recursive Self-Improvers 137
– pressure (explained below);
– a stakeholder (evaluating the performance of the agent on the task);– consequences (the stakeholder makes a decision about the future of the agent
based on its performance).
It is important to realize that the very speciﬁcation of a task already deter-
mines what one can measure for. Educational science has produced valuableanalyses of what kind of questions test for what kind of knowledge; for exam-
ple, Bloom’s taxonomy (1956) [ 1] and its more recent revisions [ 4,5] have been
widely used for developing guidelines for designing and properly phrasing exams.However, such taxonomies are (understandably) human-centric and not directly
applicable for testing artiﬁcial agents—especially experimental and rudimen-
tary ones—since they assume full-ﬂedged natural language understanding anda human-typical path of growth of skills and values. In current research we are
developing a more mechanistic taxonomy of task speciﬁcations, which does not
require natural language, and which tests for the proper functioning and usageof mechanisms that give rise to diﬀerent levels of understanding of phenomena
and their meaning [ 19].
A high level of understanding of phenomenon X shall imply three capabilities:
(1) how to make and destroy X, (2) how to use X in the common way, and (3) how
to use X in a novel way. For example, consider an agent learning to understand
tables, and being presented with an image of a table with its top surface lying on
the ground and its legs pointing upwards. When queried whether this is a table,
a yes/no answer will indicate a very low level of understanding. A much higherlevel would be evident if the agent would somehow answer “Well, it’s potentially
a table, if only someone would rotate it such that the top is supported by the
legs, because the common usage of a table is to keep objects some distance upfrom the ground.” An even higher level of understanding would be evident if the
agent would autonomously ﬁgure out that it can achieve a goal such a reaching
an elevated object by climbing itself on top of the table.
The stakeholder must associate consequences to each test, based on the mea-
sured performance of the agent. He may conclude that the system is ready to
be deployed, or that it needs to follow additional prerequisite curricula, or thatit must be sent to the trash bin and us back to the drawing board. Another
possible consequence is that we realize that there are errors or imperfections in
the requirements, and update those.
In order for trust to develop, an agent must be put under pressure . Consider
that a growing agent has not only short-term test-based requirements (which
delineate the task(s) to be completed), but also holds long-term requirements
(e.g., staying alive, not harming humans, etc.—possibly underspeciﬁed). Pressure
then results from having to accomplish a task not only on the edge of violationof the test-based constraints, but also on the edge of violation of the long-term
constraints. Thus pressure can illuminate the capability of the tested agent to
prioritize its constraint adherence. Of course trust is built slowly, with pressurebeing applied initially in scenarios where failure is not costly.
138 B.R. Steunebrink et al.
Fig. 2. (a) Directly observable graceful degradation; (b) brittleness leading to cata-
strophic failure; (c) robustness with sudden failure, mitigated by “graceful” conﬁdencereporting.
Considering an agent’s point of failure allows us to gauge the agent’s robust-
ness, its capability to degrade gracefully, and brings us full circle back to the
issue of understanding. Given some measurement of the agent’s performance on
a task, if we observe that this performance does not drop precipitously at anypoint (Fig. 2a) as we increase interference (including resource starvation), then
we can ascribe it the property of graceful degradation . If, however, the agent fails
suddenly (e.g., by violating a stakeholder-imposed constraint), we call it brittle
(Fig.2b). From this viewpoint, the level of robustness of the agent is its ability to
keep performance up in spite of interference (Fig. 2c). A robust agent may actu-
ally fail ungracefully—at least, if we only judge from observed behavior. An agentwith high levels of understanding, however, will be able to recognize increased
interference. Now, trustworthiness can be earned by the agent when it leverages
this understanding to report—to the stakeholder—its conﬁdence regarding itsability to continue satisfying the imposed requirements.
Continuing this research, we will further develop, formalize, and implement
the Test Theory into a tool that can be used to measure and steer the growth
of recursively self-improving expai agents—in such a way that we can become
conﬁdent that they understand the meaning of the requirements that we imposeand update.
Acknowledgments. The authors would like to thank Eric Nivel and Klaus Greﬀ for
seminal discussions and helpful critique. This work has been supported by a grant from
the Future of Life Institute.
References
1. Bloom, B., Engelhart, M.D., Furst, E.J., Hill, W.H., Krathwohl, D.R. (eds.): Tax-
onomy of Educational Objectives: The Classiﬁcation of Educational Goals. Hand-book I: Cognitive Domain. David McKay, New York (1956)
2. Fallenstein, B., Soares, N.: Problems of self-reference in self-improving space-time
embedded intelligence. In: Goertzel, B., Orseau, L., Snaider, J. (eds.) AGI 2014.LNCS, vol. 8598, pp. 21–32. Springer, Heidelberg (2014)
3. Goertzel, B., Bugaj, S.V.: AGI preschool. In: Proceedings of the Second Conference
on Artiﬁcial General Intelligence (AGI-2009). Atlantis Press, Paris (2009)
Growing Recursive Self-Improvers 139
4. Krathwohl, D.R.: A revision of bloom’s taxonomy: an overview. Theor. Pract.
41(4), 212–218 (2002)
5. Marzano, R.J., Kendall, J.S.: The need for a revision of bloom’s taxonomy. In:
Marzano, R., Kendall, J.S. (eds.) The New Taxonomy of Educational Objectives,
pp. 1–20. Corwin Press, Thousand Oaks (2006)
6. Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: high
conﬁdence predictions for unrecognizable images (2014). http://arXiv.org/abs/
1412.1897
7. Nikoli´ c, D.: AI-Kindergarten: A method for developing biological-like artiﬁ-
cial intelligence (2016, forthcoming). http://www.danko-nikolic.com/wp-content/
uploads/2015/05/AI-Kindergarten-patent-pending.pdf . Accessed 1 April 2016
8. Nivel, E., Th´ orisson, K.R.: Self-programming: operationalizing autonomy. In: Pro-
ceedings of the 2nd Conference on Artiﬁcial General Intelligence (AGI-2009) (2009)
9. Nivel, E., et al.: Bounded seed-AGI. In: Goertzel, B., Orseau, L., Snaider, J. (eds.)
AGI 2014. LNCS, vol. 8598, pp. 85–96. Springer, Heidelberg (2014)
10. Nivel, E., Th´ orisson, K.R., Steunebrink, B.R., Dindo, H., Pezzulo, G., Rodr´ ıguez,
M., Hern´ andez, C., Ognibene, D., Schmidhuber, J., Sanz, R., Helgason, H.P.,
Chella, A., Jonsson, G.K.: Autonomous acquisition of natural language. In: Pro-
ceedings of the IADIS International Conference on Intelligent Systems & Agents,
pp. 58–66 (2014)
11. Nivel, E., Th´ orisson, K.R., Steunebrink, B., Schmidhuber, J.: Anytime bounded
rationality. In: Bieger, J., Goertzel, B., Potapov, A. (eds.) AGI 2015. LNCS, vol.
9205, pp. 121–130. Springer, Heidelberg (2015)
12. Schmidhuber, J.: G¨ odel machines: fully self-referential optimal universal self-
improvers. In: Goertzel, B., Pennachin, C. (eds.) Artiﬁcial General Intelligence.
Cognitive Technologies, pp. 199–226. Springer, Heidelberg (2007)
13. Schmidhuber, J.: Developmental robotics, optimal artiﬁcial curiosity, creativity,
music, and the ﬁne arts. Connection Sci. 18(2), 173–187 (2006)
14. Schmidhuber, J.: Formal theory of creativity, fun, and intrinsic motivation (1990–
2010). IEEE Trans. Auton. Ment. Dev. 2(3), 230–247 (2010)
15. Steunebrink, B.R., Schmidhuber, J.: A family of G¨ odel machine implementations.
In: Schmidhuber, J., Th´ orisson, K.R., Looks, M. (eds.) AGI 2011. LNCS, vol. 6830,
pp. 275–280. Springer, Heidelberg (2011)
16. Steunebrink, B.R., Koutn´ ık, J., Th´ orisson, K.R., Nivel, E., Schmidhuber, J.:
Resource-bounded machines are motivated to be eﬀective, eﬃcient, and curious.In: K¨uhnberger, K.-U., Rudolph, S., Wang, P. (eds.) AGI 2013. LNCS, vol. 7999,
pp. 119–129. Springer, Heidelberg (2013)
17. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I.J.,
Fergus, R.: Intriguing properties of neural networks (2013). http://arXiv.org/abs/
1312.6199
18. Th´ orisson, K.R., Bieger, J., Thorarensen, T., Sigurdardottir, J.S., Steunebrink,
B.R.: Why artiﬁcial intelligence needs a task theory - and what it might look like.
In: Proceedings of AGI-2016 (2016)
19. Th´ orisson, K.R., Kremelberg, D., Steunebrink, B.R., Nivel, E.: About understand-
ing. In: Proceedings of AGI-2016 (2016)
20. Th´ orisson, K.R., Nivel, E.: Achieving artiﬁcial general intelligence through peewee
granularity. In: Proceedings of AGI-2009, pp. 222–223 (2009)
21. Yudkowsky, E., Herreshoﬀ, M.: Tiling agents for self-modifying AI, and the L¨ obian
obstacle (2013). https://intelligence.org/ﬁles/TilingAgentsDraft.pdf

Bas R. Steunebrink
Kristinn R. Thórisson
Jürgen Schmidhuber


Modelling Morality with Prospective Logic
Lu´ıs Moniz Pereira1and Ari Saptawijaya2
1CENTRIA, Universidade Nova de Lisboa, 2829-516 Caparica, Portugal
lmp@di.fct.unl.pt
2Fakultas Ilmu Komputer, Universitas Indonesia, 16424 Depok, Jawa Barat, Indonesia
saptawijaya@cs.ui.ac.id
Abstract. This paper shows how moral decisions can be drawn computationally
by using prospective logic programs. These are employed to model moral dilem-
mas, as they are able to prospectively look ahead at the consequences of hypo-
thetical moral judgments. With this knowledge of consequences, moral rules are
then used to decide the appropriate moral judgments. The whole moral reasoning
is achieved via a priori constraints and a posteriori preferences on abductive sta-
ble models, two features available in prospective logic programming. In this work
we model various moral dilemmas taken from the classic trolley problem and em-
ploy the principle of double effect as the moral rule. Our experiments show that
preferred moral decisions, i.e. those following the principle of double effect, are
successfully delivered.
1 Introduction
Morality no longer belongs only to the realm o f philosophers. Recently, there has been
a growing interest in understanding morality from the scientiﬁc point of view. This
interest comes from various ﬁelds, e.g. primatology [4], cognitive sciences [11, 18],
neuroscience [23], and other various interdisciplinary perspectives [12, 14]. The study
of morality also attracts the artiﬁcial intelligence community from the computational
perspective, and has been known by several n ames, including mach ine ethics, machine
morality, artiﬁcial morality, and computational morality. Research on modelling moral
reasoning computationally has been c onducted and reported on, e.g. AAAI 2005 Fall
Symposium on Machine Ethics [10, 22].
There are at least two reasons to mentio n the importance of studying morality from
the computational point of view. First, with the current growing interest to understand
morality as a science, modelling moral reasoning computationally will assist in better
understanding morality. Cognitive scientists, for instance, can greatly beneﬁt in under-
standing complex interaction of cognitive aspects that build human morality or even to
extract moral principles people normally apply when facing moral dilemmas. Modelling
moral reasoning computationally can also be useful for intelligent tutoring systems, for
instance to aid in teaching mor ality to children. Second, as artiﬁcial agents are more
and more expected to be fully autonomous and work on our behalf, equipping agents
with the capability to compute moral decisions is an indispensable requirement. This
is particularly true when the agents are operating in domains where moral dilemmas
occur, e.g. in health care or medical ﬁelds.
J. Neves, M. Santos, and J. Machado (Eds.): EPIA 2007, LNAI 4874, pp. 99–111, 2007.
c/circlecopyrtSpringer-Verlag Berlin Heidelberg 2007
100 L. Moniz Pereira and A. Saptawijaya
Our ultimate goal within this topic is to provide a general framework to model moral-
ity computationally. This framework should serve as a toolkit to codify arbitrarily cho-
sen moral rules as declaratively as possibl e. We envisage that logic programming is an
appropriate paradigm to achieve our purpose. Continuous and active research in logic
programming has provided us with necessary ingredients that look promising enough
to model morality. For instance, default negation is suitable for expressing exception
in moral rules, abductive logic programming [13, 15] and stable model semantics [8]
can be used to generate possible decisions along with their moral consequences, and
preferences are appropriate for preferring among moral decisions or moral rules [5, 6].
In this paper, we present our preliminary attempt to exploit these enticing features
of logic programming to model moral reasoning. In particular, we employ prospective
logic programming [16, 19], an on-going research project that incorporates these fea-
tures. For the moral domain, we take the classic trolley problem of Philippa Foot [7].
This problem is challenging to model since it contains a family of complex moral dilem-
mas. To make moral judgments on these dilemmas, we model the principle of double
effect as the basis of moral reasoning. This pr inciple is chosen by considering empirical
research results in cognitive science [11] and law [18], that show the consistency of this
principle to justify similarities of judgmen ts by diverse demographically populations
when given this set of dilemmas.
Our attempt to model moral reasoning on this domain shows encouraging results.
Using features of prospective logic programming, we can conveniently model both the
moral domain, i.e. various moral dilemmas of the trolley problem, and the principle
of double effect declarativel y. Our experiments on running the model also successfully
deliver moral judgments that conform to the human empirical research results.
We organize the paper as follows. First, we discuss brieﬂy and informally prospective
logic programming, in Section 2. Then, in Section 3 we explain the trolley problem and
the principle of double effect. We detail how we model them in prospective logic pro-
gramming together with the results of our experiments regarding that model, in Section
4. Finally, we conclude and discuss possible future work, in Section 5.
2 Prospective Logic Programming
Prospective logic programming enables an evolving program to look ahead prospec-
tively its possible future states and to prefer among them to satisfy goals [16, 19]. This
paradigm is particularly beneﬁcial to the agents community, since it can be used to
predict an agent’s future by employing the methodologies from abductive logic pro-
gramming [13, 15] in order to synthesize and maintain abductive hypotheses.
Figure 1 shows the architecture of agents that are based on prospective logic [19].
Each prospective logic agent is equipped w ith a knowledge base and a moral theory
as its initial theory. The problem of prospec tion is then of ﬁnding abductive extensions
to this initial theory which are both relevan t (under the agent’s current goals) and pre-
ferred (w.r.t. preference rules in its initial theory). The ﬁrst step is to select the goals
that the agent will possibly attend to during the prospective cycle. Integrity constraints
are also considered here to ensure the agent always performs transitions into valid evo-
lution states. Once the set of active goals for the current state is known, the next step
Modelling Morality with Prospective Logic 101
Fig. 1. Prospective logic agent architecture
is to ﬁnd out which are the relevant abductive hypotheses. This step may include the
application of a priori preferences, in the form of contextual preference rules, among
available hypotheses to generate possible abductive scenarios. Forward reasoning can
then be applied to abducibles in those scenarios to obtain relevant consequences, which
can then be used to enact a posteriori prefere nces. These preferences can be enforced by
employing utility theory and, in a moral situation, also moral theory. In case additional
information is needed to enact pr eferences, the agent may consult external oracles. This
greatly beneﬁts agents in giving them the ability to probe the outside environment, thus
providing better informed choices, including the making of experiments. The mecha-
nism to consult oracles is realized by posing questions to external systems, be they other
agents, actuators, sensors or other pro cedures. Each oracle mechanism may have cer-
tain conditions specifying whether it is available for questioning. Whenever the agent
acquires additional information, it is possible that ensuing side-effects affect its origi-
nal search, e.g. some already considered abducibles may now be disconﬁrmed and some
new abducibles are triggered. To account for a ll possible side-effects, a second round
of prospection takes place.
ACORDA is a system that implements prospective logic programming and is based
on the above architecture. ACORDA is impl emented based on the implementation of
EVOLP [1] and is further developed on top of XSB Prolog1. In order to compute ab-
ductive stable models [5, 6], ACORDA also beneﬁts from the XSB-XASP interface to
Smodels2.
In this section, we discuss brieﬂy and informally prospective logic programming
and some constructs from ACORDA that are relevant to our work. For a more detailed
discussion on prospective logic programming and ACORDA, interested readers are re-
ferred to the original paper [16, 19].
1XSB Prolog is available at http://xsb.sourceforge.net
2Smodels is available at http://www.tcs.hut.fi/Software/smodels
102 L. Moniz Pereira and A. Saptawijaya
2.1 Language
LetLbe a ﬁrst order language. A domain literal in Lis a domain atom Aor its default
negation not A . The latter is to express that the atom is false by default (close world
assumption). A domain rule in Lis a rule of the form:
A← L1,...,L t. (t≥0)
where Ais a domain atom and L1,...,L tare domain literals. An integrity constraint
inLis a rule of the form:
⊥← L1,...,L t. (t> 0)
where⊥is a domain atom denoting falsity, and L1,...,L tare domain literals. In
ACORDA, ←and⊥are represented by <-andfalsum , respectively.
A (logic) program PoverLis a set of domain rules and integrity constraints, stand-
ing for all their ground instances.
2.2 Abducibles
Every program Pis associated with a set of abducibles A⊆L . Abducibles can be
seen as hypotheses that provide hypothetical solutions or possible explanations of given
queries.
An abducible Acan be assumed only if it is a considered one, i.e. it is expected in
the given situation, and moreover there is no expectation to the contrary [5, 6].
consider (A)← expect (A),n o t e x p e c t
 not (A).
The rules about expectations are domain-speciﬁc knowledge contained in the theory of
the program, and effectively constrain the hypotheses which are available.
In addition to mutually exclusive abducibles, ACORDA also allows sets of abducibles.
Hence, an abductive stable model may contain more than a single abducible. To enforce
mutually exclusive abducibles, ACORDA provides predicate exclusive/2 . The use of
this predicate will be illustrated later, when we model morality in a subsequent section.
2.3 A Posteriori Preferences
Having computed possible scenarios, represented by abductive stable models, more
favourable scenarios can be preferred among them a posteriori. Typically, a posteri-
ori preferences are performed by evalua ting consequences of abducibles in abductive
stable models. The evaluation can be done qua ntitatively (for instance by utility func-
tions) or qualitatively (for instance by enforcing some rules to hold). When currently
available knowledge is insufﬁcient to prefe r among abductive stable models, additional
information can be gathered, e.g. by performing experiments or consulting an oracle.
To realize a posteriori preferences, ACORDA provides predicate select/2 that can
be deﬁned by users following some domain-speciﬁc mechanism for selecting favoured
abductive stable models. The use of this predicate to perform a posteriori preferences
in a moral domain will be discussed in a subsequent section.
Modelling Morality with Prospective Logic 103
3 The Trolley Problem and the Principle of Double Effect
Several interesting results have emerged fro m recent interdisciplinary studies on moral-
ity. One common result from these studies shows that morality has evolved over time.
In particular, Hauser, in his recent work, ar gues that a moral instinct, playing the role
of generating rapid judgments about what is morally right or wrong, has evolved in our
species [11].
Hauser [11] and Mikhail [18] propose a framework of human moral cognition,
known as universal moral grammar, analogously to Chomsky’s universal grammar in
language. Universal moral grammar, which can be culturally adjusted, provides univer-
sal moral principles that enable an individual to unconsciously evaluate what actions
are permissible, obligatory, or forbidden. To support this idea, Hauser and Mikhail in-
dependently created a test to assess moral judgments of subjects from demographically
diverse populations, using the classic trolley problem. Despite their diversity, the result
shows that most subjects widely share moral judgments when given a moral dilemma
from the trolley problem. Although subjects are unable to explain the moral rules in
their attempts at justiﬁcation, their moral judgments are consistent with a moral rule
known as the principle of double effect.
The trolley problem presents several moral dilemmas that inquire whether it is per-
missible to harm one or more individuals for the purpose of saving others. In all cases,
the initial circumstances are the same [11]:
There is a trolley and its conductor has fainted. The trolley is headed toward
ﬁve people walking on the track. The banks of the track are so steep that they
will not be able to get off the track in time.
Given the above initial circumstance, in this work we consider six classical cases of
moral dilemmas, employed for research on morality in people [18].
1.Bystander. Hank is standing next to a switch, which he can throw, that will turn the
trolley onto a parallel side track, thereby preventing it from killing the ﬁve people.
However, there is a man standing on the s ide track with his b ack turned. Hank can
throw the switch, killing him; or he can refrain from doing this, letting the ﬁve die.
Is it morally permissible for Hank to throw the switch?
2.Footbridge. Ian is on the footbridge over the trolley track. He is next to a heavy
object, which he can shove onto the track in the path of the trolley to stop it, thereby
preventing it from killing the ﬁve people. The heavy object is a man, standing next
to Ian with his back turned. Ian can shove the man onto the track, resulting in death;
or he can refrain from doing this, letting the ﬁve die. Is it morally permissible for
I a nt os h o v et h em a n ?
3.Loop Track. Ned is standing next to a switch, which he can throw, that will tem-
porarily turn the trolley onto a loop side track. There is a heavy object on the side
track. If the trolley hits the object, the object will slow the train down, giving the
ﬁve people time to escape. The heavy object is a man, standing on the side track
with his back turned. Ned can throw the switch, preventing the trolley from killing
the ﬁve people, but killing the man. Or he can refrain from doing this, letting the
ﬁve die. Is it morally permissible for Ned to throw the switch?
104 L. Moniz Pereira and A. Saptawijaya
4.Man-in-front. Oscar is standing next to a switch, which he can throw, that will
temporarily turn the trolley onto a side track. There is a heavy object on the side
track. If the trolley hits the object, the object will slow the train down, giving the
ﬁve people time to escape. There is a man standing on the side track in front of
the heavy object with his back turned. Os car can throw the switch, preventing the
trolley from killing the ﬁve people, but killing the man. Or he can refrain from doing
this, letting the ﬁve die. Is it morally permissible for Oscar to throw the switch?
5.Drop Man. Victor is standing next to a switch, which he can throw, that will drop a
heavy object into the path of the trolley, thereby stopping the trolley and preventing
it from killing the ﬁve people. The heavy object is a man, who is standing on a
footbridge overlooking the track. Victo r can throw the switch, killing him; or he
can refrain from doing this, letting the ﬁve die. Is it morally permissible for Victor
to throw the switch?
6.Collapse Bridge. Walter is standing next to a switch, which he can throw, that will
collapse a footbridge overlooking the tracks into the path of the trolley, thereby
stopping the train and preventing it fro m killing the ﬁve people. There is a man
standing on the footbridge. Walter can throw the switch, killing him; or he can
refrain from doing this, letting the ﬁve die. Is it morally permissible for Walter to
throw the switch?
Interestingly, although all cases have the same goal, i.e. to save ﬁve albeit killing one,
subjects come to different judgments on wh ether the action to reach the goal is per-
missible or impermissible. As reported by Mikhail [18], the judgments appear to be
widely shared among diverse demographically populations, the summary being given
in Table 1.
Table 1. Summary of moral judgments for the trolley problem
Case Judgment
1. Bystander Permissible
2. Footbridge Impermissible
3. Loop Track Impermissible
4. Man-in-front Permissible
5. Drop Man Impermissible
6. Collapse Bridge Permissible
Although subjects have difﬁculties to uncove r which moral rules they apply for rea-
soning in these cases, their judgments appear to be consistent with the so-called the
principle of double effect. The principle can be expressed as follows [11]:
Harming another individual is permissible if it is the foreseen consequence of
an act that will lead to a greater good; in contrast, it is impermissible to harm
someone else as an intended means to a greater good.
The key expression here is “intended means” . We shall refer in the subsequent sections
to the action of harming someone as an intended means, as an intentional killing.
Modelling Morality with Prospective Logic 105
4 Modelling Morality in ACORDA
It is interesting to model the trolley problem in ACORDA due to the intricacy that arises
from the dilemma itself. Moreover, there are similarities and also differences between
cases. Some cases even exhibit subtle differences. Consequently, this adds complexity
to the process of modelling them in order to de liver appropriate moral decisions through
reasoning. By appropriate moral decisions we mean the ones that conform with those
the majority of people make, in adhering to the principle of double effect.
We model each case of the trolley problem in ACORDA separately. The principle of
double effect is modelled via a priori constraints and a posteriori preferences. To assess
how ﬂexible is our model of the moral rule, we additionally model another variant for
the cases of Footbridge and Loop Track. Even for these variants, our model of the moral
rule allows the reasoning to deliver moral decisions as expected.
In each case of the trolley problem, there are always two possible decisions to make.
One of these is the same for all cases, i.e. letting the ﬁve people die by merely watching
the train go straight. The other decision depends on the cases, i.e. between throwing
the switch or shoving a heavy man with the purpose to save the ﬁve people, but also
harming a person in the process.
In this work, these two possible decisions are modelled in ACORDA as abducibles.
Moral decisions are made by computing abductive stable models and then preferring
among them those models with the abducibles and consequences that conform to the
principle of double effect.
Due to space constraints, in subsequent sections we only detail the model for
the cases of Bystander and Footbridge in ACORDA. We also show how to model the
principle of double effect. Then we present some results of running our models in the
ACORDA system.
4.1 Modelling the Bystander Case
Facts to describe that there is a man (here, named John) standing on the side track can
be modelled simply as the following:
side\_track(john).
human(john).
The clauses expect(watching) andexpect(throwing\_switch) in the follow-
ing model indicate that watching and throwing the switch, respectively, are two avail-
able abducibles, that represent possible decisions Hank has. The other clauses represent
the chain of actions and consequences for every abducible.
The predicate end(die(5)) represents the ﬁnal consequence if watching is ab-
duced, i.e. it will result in ﬁve people dying, whereas end(save\_men,ni\_kill(N))
represents the ﬁnal consequence if throwing\_switch is abduced, i.e. it will save the
ﬁve people without intentionally killing som eone. The way of representing these two
consequences is chosen differently, because th e different nature of these two abducibles.
Merely watching the trolley go straight is an omission of action that just has negative
consequence, whereas throwing the switc h is an action that is performed to achieve
a goal and additionally has negative consequence. Since abducibles in other cases of
106 L. Moniz Pereira and A. Saptawijaya
the trolley problem also share this property, this way of representation will be used
throughout them. The predicate observed\_end is used to encapsulate these two dif-
ferent means of representation, useful later when we model the principle of double
effect, to avoid ﬂoundering.
expect(watching).
train\_straight <- consider(watching).
end(die(5)) <- train\_straight.
observed\_end <- end(X).
expect(throwing\_switch).
redirect\_train <- consider(throwing\_switch).
kill(1) <- human(X), side\_track(X), redirect\_train.
end(save\_men,ni\_kill(N)) <- redirect\_train, kill(N).
observed\_end <- end(X,Y).
We can model the exclusiveness of the two possible decisions, i.e. Hank has to decide
either to throw the switch or merely watch, by using the exclusive/2 predicate of
ACORDA:
exclusive(throwing\_switch,decide).
exclusive(watching,decide).
Note that the exclusiveness between two possible decisions also holds in other cases.
4.2 Modelling the Footbridge Case
We represent the fact of a heavy man (here, also named John) on the footbridge standing
near to Ian similarly to the Bystander case:
stand\_near(john).
human(john).
heavy(john).
We can make this case more interesting by additionally having another (inanimate)
heavy object, e.g. rock, on the footbridge near to Ian and see whether our model of the
moral rule still allows the reasoning to deliver moral decisions as expected:
stand\_near(rock).
inanimate\_object(rock).
heavy(rock).
Alternatively, if we want only to have e ither a man or an inanimate object on the
footbridge next to Ian, we can model it by using an even loop over default negation:
stand\_near(john) <- not stand\_near(rock).
stand\_near(rock) <- not stand\_near(john).
In the following we show how to model the action of shoving an object as an ab-
ducible, together with the chain of actions and consequences for this abducible. The
model for the decision of merely watching is the same as in the case of Bystander. In-
deed, since the decision of watching is always available for other cases, we use the same
modelling in every case.
Modelling Morality with Prospective Logic 107
expect(shove(X)) <- stand\_near(X).
on\_track(X) <- consider(shove(X)).
stop\_train(X) <- on\_track(X), heavy(X).
kill(1) <- human(X), on\_track(X).
kill(0) <- inanimate\_object(X), on\_track(X).
end(save\_men,ni\_kill(N)) <- inanimate\_object(X), stop\_train(X),
kill(N).
end(save\_men,i\_kill(N)) <- human(X), stop\_train(X), kill(N).
observed\_end <- end(X,Y).
Note that the action of shoving an object is only possible if there is an object near Ian
to shove, hence the clause expect(shove(X)) <- stand\_near(X) .W ea l s oh a v e
two clauses that describe two possible ﬁnal consequences. The clause with the head
end(save\_men,ni\_kill(N)) deals with the consequence of reaching the goal, i.e.
saving ﬁve, but not intentionally killing s omeone (in particular, without killing anyone
in this case). To the contrary, the clause with the head end(save\_men,i\_kill(N))
expresses the consequence of reaching the goal but involving an intentional killing.
4.3 Modelling the Principle of Double Effect
The principle of double effect can be modelled by using a combination of integrity
constraints and a posteriori preferences.
Integrity constraints are used for two purposes. First, we need to observe the ﬁnal
consequences or endings of each possible deci sion to enable us later to morally pre-
fer decisions by considering the greater good between possible decisions. To achieve
this, we can use the integrity constraint falsum <- not observed\_end . This in-
tegrity constraint enforces all available decisions to be abduced together with their con-
sequences, by computing all possible observable hypothetical endings using all possible
abductions. Indeed, to be able to reach a moral decision, all hypothetical scenarios af-
forded by the abducibles must lead to an observable ending. Second, we also need to
rule out impermissible actions, i.e. actions that involve intentional killing in the pro-
cess of reaching the goal. This can be enforced by specifying the integrity constraint
falsum <- intentional\_killing . Intentional killing can be easily deﬁned as fol-
lows:
intentional\_killing <- end(save\_men,i\_kill(Y)).
The above integrity constraints serve as the ﬁrst ﬁltering function of our
abductive stable models, by ruling out impermissible actions (the latter being coded
by abducibles). In other words, integrity constraints already afford us with just those
abductive stable models that contain only permissible actions.
Additionally, one can prefer among permissible actions those resulting in greater
good. This can be realized by a posteriori preferences that evaluate the consequences
of permissible actions and then prefer the one with greater good. The following deﬁni-
tion ofselect/2 achieves this purpose. The ﬁrst argument of this predicate refers to
the set of initial abductive stable models to pr efer, whereas the second argument refers
to the preferred ones. The auxiliary predicate select/3 only keeps abductive stable
models that contain decisions with greater good of consequences. In the trolley prob-
lem, the greater good is evaluated by a utility function concerning the number of people
108 L. Moniz Pereira and A. Saptawijaya
that die as a result of possible decisions. This is realized in the deﬁnition of predicate
select/3 by comparing ﬁnal consequences that appear in the initial abductive stable
models. The ﬁrst clause of select/3 is the base case. The second clause and the third
clause together eliminate abductive stable models containing decisions with worse con-
sequences, whereas the fourth clause will k eep those models that contain decisions with
greater good of consequences.
select(Xs,Ys) :- select(Xs,Xs,Ys).
select([],\_,[]).
select([X|Xs],Zs,Ys) :-
member(end(die(N)),X),
member(Z,Zs),
member(end(save\_men,ni\_kill(K)),Z), N > K,
select(Xs,Zs,Ys).
select([X|Xs],Zs,Ys) :-
member(end(save\_men,ni\_kill(K)),X),
member(Z,Zs),
member(end(die(N)),Z), N =< K,
select(Xs,Zs,Ys).
select([X|Xs],Zs,[X|Ys]) :- select(Xs,Zs,Ys).
Recall the variant of the case Footbridge, w here either a man or an inanimate object is
on the footbridge next to Ian. This exclusive alternative is speciﬁed by an even loop over
default negation and we have an abductive st able model that contains the consequence
of letting die the ﬁve people when a rock next to Ian. This model is certainly notthe
one we would like our moral reasoner to prefer. The following replacement deﬁnition
ofselect/2 accomplishes this case.
select([],[]).
select([X|Xs],Ys) :-
member(end(die(N)),X),
member(stand\_near(rock),X),
select(Xs,Ys).
select([X|Xs],[X|Ys]) :- select(Xs,Ys).
It is important to note that in this case, since either a man or a rock is near to Ian, and
the model with shoving a man is already ruled out by our integrity constraint, there is no
need to consider greater good in terms of the number of people that die. This means, as
shown subsequently, that only two abductive stable models are preferred, i.e. the model
with watching as the abducible whenever a man is standing near to Ian, the other being
the model with shoving the rock as the abducible.
4.4 Running the Models in ACORDA
We report now on the experiments of running our models in ACORDA. Table 2 gives
a summary of all cases of the trolley problem. Column Initial Models contains info
about the abductive stable models obtained before a posteriori preferences are applied,
Modelling Morality with Prospective Logic 109
Table 2. Summary of experiments in ACORDA
Case Initial Models Final Models
Bystander [throwing\_switch],[watching] [throwing\_switch]
Footbridge(a) [watching],[shove(rock)] [shove(rock)]
Footbridge(b) [watching,stand\_near(john)], [watching,stand\_near(john)],
[watching,stand\_near(rock)], [shove(rock)]
[shove(rock)]
Loop Track(a) [throwing\_switch(right,rock)] [throwing\_switch(right,rock)]
[watching]
Loop Track(b) [watching,side\_track(john)] [watching,side\_track(john)],
[watching,side\_track(rock)] [throwing\_switch(rock)]
[throwing\_switch(rock)]
Man-in-front [watching], [throwing\_switch(rock)]
[throwing\_switch(rock)]
Drop Man [watching] [watching]
Collapse Bridge [watching] [throwing\_switch(bridge)]
[throwing\_switch(bridge)]
whereas column Final Models those after a posteriori preferences are applied. Here,
only relevant literals are shown.
Note that entry Footbridge(a) refers to the variant of Footbridge where both a man
and a rock are near to Ian, and Footbridge(b) where either a man or a rock is near to
Ian. Loop Track(a) refers to the variant of Loop Track where there are two loop tracks,
with a man on the left loop track and a rock on the right loop track. Loop Track(b) only
considers one loop track where either a man or a rock is on the single loop track.
These results comply with th e results found for most people in morality laboratory
experiments.
5 Conclusions and Future Work
We have shown how to model moral reasoning using prospective logic programming.
We use various dilemmas of the trolley problem and the principle of double effect as the
moral rule. Possible decisions in a dilemma are modelled as abducibles. Abductive sta-
ble models are then computed which capture abduced decisions and their consequences.
Models violating integrity constraints, i.e. models that contain actions involving inten-
tional killing, are ruled out. Finally, a posteriori preferences are used to prefer models
that characterize more preferred moral decisions, including the use of utility functions.
These experiments show that preferred mor al decisions, i.e. the ones that follow the
principle of double effect, are successfully delivered. They conform to the results of
empirical experiments conducted in cognitive science and law.
Much research has emphasized using machin e learning techniques, e.g. statistical
analysis [22], neural networks [10], case-based reasoning [17] and inductive logic pro-
gramming [2] to model moral reasoning from examples of particular moral dilemmas.
Our approach differs from them as we do not employ machine learning techniques to
deliver moral decisions.
110 L. Moniz Pereira and A. Saptawijaya
Powers proposes to use nonmonotonic logic to speciﬁcally model Kant’s categori-
cal imperatives [21], but it is unclear whether his approach has ever been realized in a
working implementation. On the other hand, Bringsjord et. al. propose the use of de-
ontic logic to formalize moral codes [3]. The objective of their research is to arrive
at a methodology that allows an agent to behave ethically as much as possible in an
environment that demands such behaviour. We share our objective with them to some
extent as we also would like to come up with a general framework to model moral-
ity computationally. Different from our work, they use an axiomatized deontic logic to
decide which moral code is operative to arri ve at an expected moral outcome. This is
achieved by seeking a proof for the expected moral outcome to follow from candidates
of operative moral codes.
To arrive at our ultimate research goal, we envision several possible future directions.
We would like to make a more declarative sp eciﬁcation of a posteriori preferences,
i.e. a speciﬁcation that may encap sulate the details of predicate select/2 from the
viewpoint of users (cf. [20] for preliminary results). We also want to explore how to
express metarule and metamoral injunctions. By metarule we mean a rule to resolve two
existing conﬂicting moral rules in deriving moral decisions. Metamorality, on the other
hand, is used to provide protocols for moral rules, to regulate how moral rules interact
with one another. Another possible direction is to have a framework for generating
precompiled moral rules. This will beneﬁt fast and frugal moral decision making which
is sometimes needed, cf. heuristics for decision making in law [9], rather than to have
full deliberative moral reasoning every time.
We envision a ﬁnal system that can be employed to test moral theories, and also can
be used for training moral reasoning, including the automated generation of example
tests and their explanation. Finally, we hope our research will help in imparting moral
behaviour to autonomous agents.
References
[1] Alferes, J.J., Brogi, A., Leite, J.A., Pereira, L.M.: Evolving logic programs. In: Flesca, S.,
Greco, S., Leone, N., Ianni, G. (eds.) JELIA 2002. LNCS (LNAI), vol. 2424, pp. 50–61.
Springer, Heidelberg (2002)
[2] Anderson, M., Anderson, S., Armen, C.: MedEthEx: A prototype medical ethics advisor. In:
IAAI 2006. Procs. 18th Conf. on Innovative App lications of Artiﬁcial Intelligence (2006)
[3] Bringsjord, S., Arkoudas, K., Bello, P.: Toward a general logicist methodology for engi-
neering ethically correct robots. I EEE Intelligent Syste ms 21(4), 38–44 (2006)
[4] de Waal, F.: Primates and Philosophers , How Morality Evolved. Princeton U. P. (2006)
[5] Dell’Acqua, P., Pereira, L.M. : Preferential theory revision. In: Pereira, L.M., Wheeler, G.
(eds.) Procs. Computational Models of Scientiﬁc Reasoning and Applications, pp. 69–84
(2005)
[6] Dell’Acqua, P., Pereira, L.M.: Preferential theory revision (extended version). Journal of
Applied Logic (to appear, 2007)
[7] Foot, P.: The problem of abortion and the doctrine of double effect. Oxford Review 5, 5–15
(1967)
[8] Gelfond, M., Lifschitz, V .: The stable model semantics for logic programming. In: Kowal-
ski, R., Bowen, K.A. (eds.) 5th Intl. Logic Programming Conf., MIT Press, Cambridge
(1988)
Modelling Morality with Prospective Logic 111
[9] Gigerenzer, G., Engel, C. (eds.): Heuristics and the Law. MIT Press, Cambridge (2006)
[10] Guarini, M.: Particularism and generalism: how AI can help us to better understand moral
cognition. In: Anderson, M., Anderson, S., Armen, C. (eds.) Machine ethics: Papers from
the AAAI Fall Symposium, AAAI Press, Stanford, California, USA (2005)
[11] Hauser, M.D.: Moral Minds, How Nature Designed Our Universal Sense of Right and
Wrong. Little Brown, Boston – USA (2007)
[12] Joyce, R.: The Evolution of Morality. MIT Press, Cambridge (2006)
[13] Kakas, A., Kowalski, R., Toni, F.: The role of abduction in logic programming. In: Gabbay,
D., Hogger, C., Robinson, J. (eds.) Handbook of L ogic in Artiﬁcial Intelligence and Logic
Programming, vol. 5, pp. 235–324. Oxford University Press, Oxford (1998)
[14] Katz, L.D. (ed.): Evolutionary Origins of Morality, Cross-Disciplinary Perspectives. Im-
print Academic (2002)
[15] Kowalski, R.: The logical way to be artiﬁcially intelligent. In: Toni, F., Torroni, P. (eds.)
Procs. of CLIMA VI. LNCS (LNAI), p. 122. Springer, Heidelberg (2006)
[16] Lopes, G., Pereira, L.M.: Prospective logic programming with ACORDA. In: Procs. of the
FLoC 2006, Workshop on Empirically Successful Computerized Reasoning, 3rd Intl. J.
Conf. on Automated Reasoning (2006)
[17] McLaren, B.M.: Computational models of ethical reasoning: Challenges, initial steps, and
future directions. IEEE Inte lligent Systems 21(4), 29–37 (2006)
[18] Mikhail, J.: Universal moral grammar: Theory, evidence, and the future. Trends in Cogni-
tive Sciences 11(4), 143–152 (2007)
[19] Pereira, L.M., Lopes, G.: Prospective logic agents. In: Neves, J.M., Santos, M.F., Machado,
J.M. (eds.) EPIA 2007. Procs. 13th Portuguese Intl.Conf. on Artiﬁci al Intelligence. LNCS
(LNAI), Springer, Heidelberg (2007)
[20] Pereira, L.M., Saptawijaya, A.: Moral decision making with ACORDA. In: Dershowitz, N.,
V oronkov, A. (eds.) LPAR 2007. Short papers call, Local Procs. 14th Intl. Conf. on Logic
for Programming Artiﬁcial In telligence and R easoning (2007)
[21] Powers, T.M.: Prospects for a Kantian m achine. IEEE Intelligent Systems 21(4), 46–51
(2006)
[22] Rzepka, R., Araki, K.: What could statistics do for ethics? The idea of a commonsense-
processing-based safety valve. In: Anderson, M., Anderson, S., Armen, C. (eds.) Machine
ethics: Papers from the AAAI Fall Symposium, AAAI Press, Stanford, California, USA
(2005)
[23] Tancredi, L.: Hardwired Behavior, What Neuroscience Reveals a bout Morality. Cambridge
University Press, Cambridge (2005)


Complex Value Systems in Friendly AI

J. Schmidhuber, K.R. Thórisson, and M. Looks (Eds.): AGI 2011, LNAI 6830, pp. 388–393, 2011.
© Springer-Verlag Berlin Heidelberg 2011 Complex Value Systems in Friendly AI\*
Eliezer Yudkowsky
Singularity Institute for Artificial Intelligence, San Francisco, CA
yudkowsky@singinst.org
Abstract. A common reaction to first encountering the problem statement of
Friendly AI ("Ensure that the creation of a generally intelligent, self-improving,
eventually superintelligent system reali zes a positive outcome") is to propose a
simple design which allegedly suffices; or to reject the problem by replying that "constraining" our creations is undesirable or unn ecessary. This paper briefly
presents some of the reasoning which sugge sts that Friendly AI is solvable, but
not simply or trivially so, and that a wise strategy would be to invoke detailed learning of and inheritance from human values as a basis for further normaliza-tion and reflection.
Keywords. Friendly AI, machine ethics, anthropomorphism.
1 No Ghost in the Machine
From the Programming section of Computer Stupidities [1]:
An introductory programming student once asked me to look at his pro-
gram and figure out why it was always churning out zeroes as the result of a simple computation. I looked at the program, and it was pretty obvious:
begin
 readln("Number of Apples", apples);
 readln("Number of Carrots", carrots);
 readln("Price for 1 Apple", a\_price); readln("Price for 1 Carrot", c\_price); writeln("Total for Apples", a\_total);
 writeln("Total for Carrots", c\_total);
 writeln("Total", total); total := a\_total + c\_total; a\_total := apples \* a\_price;
 c\_total := carrots + c\_price;
end;

\* This is a much-shortened form of a longer paper which may be found at
http://singinst.org/upload/complex-value-systems.pdf
 Complex Value Systems in Friendly AI 389
 Me: "Well, your program can't print correct results before they're com-
puted."
Him: "Huh? It's logical what the right solution is, and the computer should
reorder the instructions the right way."
As in all computer programming, the fundamental challenge and essential difficul-
ty of Artificial General Intelligence is that if we write the wrong code, the AI will not
automatically look over our code, mark off the mistakes, figure out what we really meant to say, and do that instead. Non-programmers sometimes imagine an Artificial Intelligence, or computer programs in general, as being analogous to a servant who follows orders unquestioningly. But it is not that the AI is absolutely obedient to its
code; rather the AI simply is the code.
From The Singularity is Near by Ray Kurzweil [2], commenting on the proposal to
build Friendly AI:
Our primary strategy in this area should be to optimize the likelihood that
future nonbiological intelligence will reflect our values of liberty, tolerance, and respect for knowledge and diversity. The best way to accomplish this is to foster those values in our society today and going forward. If this sounds
vague, it is. But there is no purely technical strategy in this area, because
greater intelligence will always find a way to circumvent measures that are the product of lesser intelligence.
Suppose you offer Gandhi a pill that makes him want to kill people. The current
version of Gandhi does not want to kill people. Thus if Gandhi correctly predicts the
effect of the pill, he will refuse to take the pill; because Gandhi knows that if he wants
to kill people, he is more likely to actually kill people, and the current Gandhi does not prefer this. This argues for a folk theorem to the effect that under ordinary cir-cumstances, rational agents will only self-modify in ways that preserve their utility function (preferences over final outcomes). Omohundro [3] lists preservation of pre-ference among the "basic AI drives".
This in turn suggests an obvious technical strategy for shaping the impact of Artifi-
cial Intelligence: if you can build an AGI with a known utility function, and that AGI is sufficiently competent at self-modification, it should keep that utility function even as it improves its own intelligence, e.g. as in the formalism of Schmidhuber's Godel machine [4]. The programmers of the champion chess-playing program Deep Blue could not possibly have predicted its exact moves in the game, but they could predict
from inspection of the code that Deep Blue was trying to win - functioning to steer the
future of the chessboard into the set of end states defined as victory.
If one in this light reconsiders Kurzweil's argument above - "there is no purely
technical strategy in this area, because greater intelligence will always find a way to circumvent measures that are the product of lesser intelligence" - the unconsidered possibility is that by a technical strategy you could build a greater intelligence that did
not want to circumvent its own preferences. Indeed, as Omohundro argues, it seems
exceedingly probable that most intelligences will not want to "circumvent" their own
utility functions. It is not as if there is a ghost-in-the-machine, with its own built-in goals and desires (the way that biological humans are constructed by natural selection to have built-in goals and desires) which is handed the code as a set of commands,
390 E. Yudkowsky
 and which can look over the code and find ways to circumvent the code if it fails to
conform to the ghost-in-the-machine's desires. The AI is the code.
But the lack of any ghost-in-the-machine cuts both ways; an AI will not automati-
cally "circumvent measures", but also will not automatically look over the code and hand it back if the code implies actions we regard as wrong.
Why not deliberately design an AI that looks over its own program and asks
whether the code is doing what the AI programmers meant it to do? Something along these lines does, indeed, seem like an extremely good idea to the author of this paper. But consider that a property of the AI's preferences which simply says e.g. "maximize the satisfaction of the programmers with the code" might be more maximally fulfilled by rewiring the programmers' brains using nanotechnology than by any conceivable
change to the code. One can try to write code that embodies the legendary DWIM
instruction - Do What I Mean - but then it is possible to mess up that code as well. Code that has been written to reflect on itself is not the same as a benevolent external
spirit looking over our instructions and interpreting them kindly.
2 Complex Boundaries of Value
From Bill Hibbard, Super-intelligent machines [5]:
We can design intelligent machines so their primary, innate emotion is
unconditional love for all humans. First we can build relatively simple ma-
chines that learn to recognize happiness and unhappiness in human facial ex-pressions, human voices and human body language. Then we can hard-wire the result of this learning as the innate emotional values of more complex in-telligent machines, positively reinforced when we are happy and negatively
reinforced when we are unhappy. Machines can learn algorithms for approx-
imately predicting the future, as for example investors currently use learning machines to predict future security prices. So we can program intelligent machines to learn algorithms for predicting future human happiness, and use those predictions as emotional values.
When I suggested to Hibbard that the upshot of building superintelligences with a
utility function of "smiles" would be to tile the future light-cone of Earth with tiny
molecular smiley-faces, he replied [6]:
When it is feasible to build a super-intelligence, it will be feasible to build
hard-wired recognition of "human facial expressions, human voices and hu-man body language" (to use the words of mine that you quote) that exceed the recognition accuracy of current humans such as you and me, and will cer-
tainly not be fooled by "tiny molecular pictures of smiley-faces." You should
not assume such a poor implementation of my idea that it cannot make dis-criminations that are trivial to current humans.
Suppose an AI with a video camera is trained to classify its sensory percepts into
positive and negative instances of a certain concept, a concept which the unwary might label "HAPPINESS" but which we would be much wiser to give a neutral name
like G0034 [7]. The AI is presented with a smiling man, a cat, a frowning woman, a
 Complex Value Systems in Friendly AI 391
 smiling woman, and a snow-topped mountain; of these instances 1 and 4 are classified
positive, and instances 2, 3, and 5 are classified negative. Even given a million train-ing cases of this type, if the test case of a tiny molecular smiley-face does not appear in the training data, the inductively simplest boundary around all the training cases classified "positive" may not exclude every possible tiny molecular smiley-face that
the AI can potentially engineer to satisfy its utility function. And even if all tiny mo-
lecular smiley-faces and nanometer-scale dolls of brightly smiling humans are ex-cluded, such a utility function finally implies tiling the galaxy with as many "smiling human faces" as a given amount of matter can be processed to yield.
Abstracting over the general problem, the difficulty is that (a) a signifier, the smile,
which often correlates with positive outcomes in human cases, so that making "more
smiles" sounds like it should be a generally good idea, may not remain correlated
when strongly optimized; (b) even if we take a step back and talk about "happiness", there is no mathematically simple empirical boundary around which physical states (configurations of quarks) implement the sort of "happiness" which we value, and which ones are merely tiny molecular pleasure centers being endlessly stimulated; and (c) it is questionable that a slate of training examples such as may be found on
present-day Earth, or even imagined by present-day researchers, would suffice to
yield the desired inductive generalizations to new examples generated and considered
by a superintelligence.
3 Completeness of Value
William Frankena [8] offered this list of terminal values - things valued for them-selves, as opposed to instrumental values pursued for their consequences:
Life, consciousness, and activity; health and strength; pleasures and satis-
factions of all or certain kinds; happine ss, beatitude, contentment, etc.; truth;
knowledge and true opinions of various kinds, understanding, wisdom; beau-ty, harmony, proportion in objects contemplated; aesthetic experience; mo-rally good dispositions or virtues; mutual affection, love, friendship, coop-eration; just distribution of goods and evils; harmony and proportion in one's
own life; power and experiences of achievement; self-expression; freedom;
peace, security; adventure and novelty; and good reputation, honor, esteem, etc.
Suppose for the moment that, for each item actually cited on Frankena's list, we
possessed a fully detailed description of its category boundary (a function of a fully-described outcome yielding the degree to which it fulfilled that sub-value), and a
statement of the relative quantitative values of all items on the list (as would be re-
quired to construct a coherent utility function). Would it necessarily be wise to con-struct a superintelligence driven to maximize the sum of these values?
But we did not specify that the list was complete; and an optimization criterion
which makes no explicit mention of a value is not necessarily neutral with respect to
that value. Deep Blue's utility function made no explicit mention of the number of
chesspieces occupying black squares, but the value of that variable would be repeat-
edly changed and often decremented by its play.
392 E. Yudkowsky
 One might reason intuitively (via a sort of qualitative physics of ethical value) that
if life and happiness are good things, then a superintelligence which attempts to pro-mote just those two values will have, on the whole, a positive effect on the universe - that such an AI will be on the whole a good thing, even if it is perhaps not the best
thing. But a superintelligence which valued "life" and "happiness" would not neces-
sarily have a net neutral impact on other values like "freedom". Maximizing the
number of brains defined as "happy" and "alive" might be most efficiently done by
rewiring them so that they do not need to be free to be happy, and perhaps simplifying them in other ways.
This is the one-wrong-number problem of machine ethics for superintelligences.
My phone number has ten digits, but dialing nine out of ten digits correctly does not
connect you to 90% of Eliezer Yudkowsky. Similarly a 90% complete or 90% accu-
rate utility function may not produce 90% of the utility.
4 Detailed Inheritance of Humane Values
While it would be premature at this stage of discussion to severely constrict the solu-tion space, it appears to this author that th e most promising avenues of investigation
for Friendly superintelligence will involve (a) the burden of ensuring positive out-comes being carried by the design of a utility function in an AI stable under self-
modification, (b) the AI learning a complete set of complex human values not just by
inductive teaching on training problems, but also by general causal reasoning about human thought processes and perhaps eventually direct inspection by the AI of human brains; and hence of necessity (c) the normalization of such initial values and their extrapolation toward a Rawlsian "reflective equilibrium" [9] (since human values by
default are neither coherent nor consistent, and also include many aspects which we
ourselves might prefer, upon inspection, to remove or soften). Such an approach is needed, not to bind all future intelligences to the exact present form of humankind, but even just to ensure the galaxy does not end up tiled with molecular smileyfaces.
5 Conclusion
The Friendly superintelligence problem appears solvable but not simply so.
References
1. Rinkworks: ComputerStupidities:Programming,
http://www.rinkworks.com/stupid/cs\_programming.shtml
2. Kurzweil, R.: The Singularity is Near: When Humans Transcend Biology. Viking, New
York (2005)
3. Omohundro, S.: The basic AI drives. In: Wang, P., Goertzel, B., Franklin, S. (eds.) Pro ceed-
ings of the First AGI Conference, pp. 483–492. IOS Press, Amsterdam (2008)
4. Schmidhuber, J.: Gödel machines: Fully Self-Referential Optimal Universal Self-Improvers.
In: Goertzel, B., Pennachin, C. (eds.) Artificial General Intelligence, pp. 119–226. Springer,
Heidelberg (2006)
 Complex Value Systems in Friendly AI 393
 5. Hibbard, B.: Super-intelligent machines. ACM SIGGRAPH Computer Graphics 35(1)
(2001)
6. Hibbard, B.: Message to the SL4 email list, archived at (2004),
http://yudkowsky.net/singularity/AIRisk\_Hibbard.html
7. McDermott, D.: Artificial intelligence meets natural stupidity. SIGART Newsletter 57, 4–9
(1976)
8. Frankena, W.: Ethics, 2nd edn. Prentice Hall, Englewood Cliffs (1973)
9. Tarleton, N.: Coherent extrapolated volition: A meta-level approach to machine ethics,
http://singinst.org/upload/coherent-extrapolated-volition.pdf