The Singularity and Machine Ethics

Chapter 6
The Singularity and Machine Ethics
Luke Muehlhauser and Louie Helm
Abstract Many researchers have argued that a self-improving artiﬁcial intelli-
gence (AI) could become so vastly more powerful than humans that we would not
be able to stop it from achieving its goals. If so, and if the AI’s goals differ from
ours, then this could be disastrous for humans. One proposed solution is to pro-gram the AI’s goal system to want what we want before the AI self-improvesbeyond our capacity to control it. Unfortunately, it is difﬁcult to specify what wewant. After clarifying what we mean by ‘‘intelligence’’, we offer a series of‘‘intuition pumps’’ from the ﬁeld of moral philosophy for our conclusion thathuman values are complex and difﬁcult to specify. We then survey the evidencefrom the psychology of motivation, moral psychology, and neuroeconomics that
supports our position. We conclude by recommending ideal preference theories of
value as a promising approach for developing a machine ethics suitable for nav-igating an intelligence explosion or ‘‘technological singularity’’.
To educate [someone] in mind and not in morals is to educate a
menace to society.
Theodore Roosevelt.
L. Muehlhauser ( &) /C1L. Helm
Machine Intelligence Research Institute, Berkeley, USA
e-mail: luke@singularity.org
A. H. Eden et al. (eds.), Singularity Hypotheses , The Frontiers Collection,
DOI: 10.1007/978-3-642-32560-1\_6, /C211Springer-Verlag Berlin Heidelberg 2012101
Introduction
Many researchers have argued that, by way of an ‘‘intelligence explosion’’ (Good
1959 ,1965 ,1970 ) sometime in the next century, a self-improving1artiﬁcial intel-
ligence (AI) could become so vastly more powerful than humans that we would not
be able to stop it from achieving its goals.2If so, and if the AI’s goals differ from ours,
then this could be disastrous for humans and what we value (Joy 2000 ; Bostrom 2003 ;
Posner 2004 ; Friedman 2008 ; Yudkowsky 2008 ; Fox and Shulman 2010 ; Chalmers
2010 ; Bostrom and Yudkowsky, forthcoming; Muehlhauser and Salamon, this
volume).
One proposed solution is to program the AI’s goal system3to want what we
want before the AI self-improves beyond our capacity to control it. While this
proposal may be the only lasting solution for AI risk (Muehlhauser and Salamon,
this volume), it faces many difﬁculties (Yudkowsky 2001 ). One such difﬁculty is
that human values are complex and difﬁcult to specify,4and this presents
1For discussions of self-improving AI, see Schmidhuber ( 2007 ); Omohundro ( 2008 ); Mahoney
(2010 ); Hall ( 2007b ,2011 ).
2For simplicity, we speak of a single AI rather than multiple AIs. Of course, it may be that
multiple AIs will undergo intelligence explosion more or less simultaneously and compete for
resources for years or decades. The consequences of this scenario could be even moreunpredictable than those of a ‘‘singleton’’ AI (Bostrom 2006 ), and we do not have the space for an
examination of such scenarios in this chapter. For space reasons we will also only consider what
Chalmers ( 2010 ) calls ‘‘non-human-based AI’’ and Muehlhauser and Salamon (this volume) call
‘‘de novo AI’’, thereby excluding self-improving AI based on the human mind, for examplewhole brain emulation (Sandberg and Bostrom 2008 ).
3When we speak of an advanced AI’s goal system, we do not have in mind today’s
reinforcement learning agents, whose only goal is to maximize expected reward. Such an agent
may hijack or ‘‘wirehead’’ its own reward function (Dewey 2011 ; Ring and Orseau 2011 ), and
may not be able to become superintelligent because it does not model itself and therefore can’tprotect or improve its own hardware. Rather, we have in mind a future AI goal architecture
realized by a utility function that encodes value for states of affairs (Dewey 2011 ; Hibbard 2012 ).
4Minsky ( 1984 ) provides an early discussion of our subject, writing that ‘‘ …it is always
dangerous to try to relieve ourselves of the responsibility of understanding exactly how our
wishes will be realized. Whenever we leave the choice of means to any servants we may choosethen the greater the range of possible methods we leave to those servants, the more we expose
ourselves to accidents and incidents. When we delegate those responsibilities, then we may not
realize, before it is too late to turn back, that our goals have been misinterpreted …. [Another] risk
is exposure to the consequences of self-deception. It is always tempting to say to oneself …that ‘I
know what I would like to happen, but I can’t quite express it clearly enough.’ However, that
concept itself reﬂects a too-simplistic self-image, which portrays one’s own self as [having] well-
deﬁned wishes, intentions, and goals. This pre-Freudian image serves to excuse our frequentappearances of ambivalence; we convince ourselves that clarifying our intentions is merely amatter of straightening-out the input–output channels between our inner and outer selves. The
trouble is, we simply aren’t made that way. Our goals themselves are ambiguous …. The ultimate
risk comes when [we] attempt to take that ﬁnal step—of designing goal-achieving programs thatare programmed to make themselves grow increasingly powerful, by self-evolving methods thataugment and enhance their own capabilities …. The problem is that, with such powerful102 L. Muehlhauser and L. Helm
challenges for developing a machine ethics suitable for navigating an intelligence
explosion.
After clarifying what we mean by ‘‘intelligence’’, we offer a series of ‘‘intuition
pumps’’ (Dennett 1984 , P. 12) from the ﬁeld of moral philosophy supporting our
conclusion that human values are complex and difﬁcult to specify. We then survey
the evidence from the psychology of motivation, moral psychology, and neuro-economics that supports our position. We conclude by recommending idealpreference theories of value as a promising approach for developing a machineethics suitable for navigating an intelligence explosion.
Intelligence and Optimization
Good, who ﬁrst articulated the idea of an intelligence explosion, referred to anymachine more intelligent than the smartest human as an ‘‘ultraintelligent’’ machine(Good 1965 ). Today the term ‘‘superintelligence’’ is more common, and it refers to
a machine that is much smarter than the smartest human (Bostrom 1998 ,2003 ;
Legg 2008 ).
But the term ‘‘intelligence’’ may not be ideal for discussing powerful machines.
Why? There are many competing deﬁnitions and theories of intelligence
(Davidson and Kemp 2011 ; Niu and Brass 2011 ; Legg and Hutter 2007 ), and the
term has seen its share of emotionally-laden controversy (Halpern et al. 2011 ;
Daley and Onwuegbuzie 2011 ).
The term also comes loaded with connotations, some of which do not ﬁt
machine intelligence. Laypeople tend to see intelligence as correlated with beingclever, creative, self-conﬁdent, socially competent, deliberate, analytically skilled,verbally skilled, efﬁcient, energetic, correct, and careful, but as anticorrelated with
being dishonest, apathetic, and unreliable (Bruner et al. 1958 ; Neisser 1979 ;
Sternberg et al. 1981 ,1985 ). Moreover, cultures vary with respect to the associ-
ations they make with intelligence (Niu and Brass 2011 ; Sternberg and Grigorenko
2006 ). For example, Chinese people tend to emphasize analytical ability, memory
skills, carefulness, modesty, and perseverance in their concepts of intelligence(Fang et al. 1987 ), while Africans tend to emphasize social competencies (Ruzgis
and Grigorenko 1994 ; Grigorenko et al. 2001 ).
One key factor is that people overwhelmingly associate intelligence with positive
rather than negative traits, perhaps at least partly due to a well-documented cognitive
bias called the ‘‘affect heuristic’’ (Slovic et al. 2002 ), which leads us to make
inferences by checking our emotions. Because people have positive affect toward
(Footnote 4 continued)
machines, it would require but the slightest accident of careless design for them to place theirgoals ahead of [ours]’’.6 The Singularity and Machine Ethics 103
intelligence, they intuitively conclude that those with more intelligence possess other
positive traits to a greater extent.
Despite the colloquial associations of the word ‘‘intelligence’’, AI researchers
working to improve machine intelligence do not mean to imply that superintelligent
machines will exhibit, for example, increased modesty or honesty. Rather, AI
researchers’ concepts of machine intelligence converge on the idea of optimal goalfulﬁllment in a wide variety of environments (Legg 2008 ), what we might call
‘‘optimization power.’’
5This optimization concept of intelligence is not anthropo-
morphic and can be applied to any agent—human, animal, machine, or otherwise.6
Unfortunately, anthropomorphic bias (Epley et al. 2007 ; Barrett and Keil 1996 )i s
not unique to laypeople. AI researcher Storrs Hall suggests that our machines may bemore moral than we are, and cites as partial evidence the fact that in humans
‘‘criminality is strongly and negatively correlated with IQ’’ (Hall 2007a , p. 340). But
machine intelligence has little to do with IQ or with the human cognitive architec-tures and social systems that might explain an anticorrelation between humancriminality and IQ.
To avoid anthropomorphic bias and other problems with the word ‘‘intelligence’’,
in this chapter we will use the term ‘‘machine superoptimizer’’ in place of ‘‘machinesuperintelligence.’’
7
Using this term, it should be clear that a machine superoptimizer will not nec-
essarily be modest or honest. It will simply be very capable of achieving its goals
(whatever they are) in a wide variety of environments (Bostrom 2012 ). If its goal
system aims to maximize the number of paperclips that exist, then it will be very goodat maximizing paperclips in a wide variety of environments. The machine’s opti-mization power does not predict that it will always be honest while maximizingpaperclips. Nor does it predict that the machine will be so modest that it will feel atsome point that it has made enough paperclips and then modify its goal system to aimtoward something else. Nor does the machine’s optimization power suggest that the
machine will be amenable to moral argument. A machine superoptimizer need not
even be sentient or have ‘‘understanding’’ in John Searle’s ( 1980 ) sense, so long as it
is very capable of achieving its goals in a wide variety of environments.
5The informal deﬁnition of intelligence in Legg ( 2008 ) captures what we mean by
‘‘optimization power,’’ but Legg’s speciﬁc formalization does not. Legg formalizes intelligenceas a measure of expected performance on arbitrary reinforcement learning problems (Legg 2008 :
p. 77), but we consider this only a preliminary step in formalizing optimal goal fulﬁllment ability.
We think of optimization power as a measure of expected performance across a broader class of
goals, including goals about states of affairs in the world (argued to be impossible forreinforcement learners in Ring and Orseau 2011 ; Dewey 2011 ). Also, Legg’s formal deﬁnition of
intelligence is drawn from a dualistic ‘‘agent-environment’’ model of optimal agency (Legg 2008 :
p. 40) that does not represent its own computation as occurring in a physical world with physicallimits and costs.
6Even this ‘‘optimization’’ notion of intelligence is incomplete, however. See Muehlhauser and
Salamon (this volume).
7But, see Legg ( 2009 ) for a defense of Legg’s formalization of universal intelligence as an
alternative to what we mean by ‘‘optimization power’’.104 L. Muehlhauser and L. Helm
The Golem Genie
Since Plato, many have believed that knowledge is justiﬁed true belief. Gettier
(1963 ) argued that knowledge cannot be justiﬁed true belief because there are
hypothetical cases of justiﬁed true belief that we intuitively would not count as
knowledge. Since then, each newly proposed conceptual analysis of knowledgehas been met with novel counter-examples (Shope 1983 ). Weatherson ( 2003 )
called this the ‘‘analysis of knowledge merry go round’’.
Similarly, advocates for mutually opposing moral theories seem to have shown
that no matter which set of consistent moral principles one defends, intuitivelyrepugnant conclusions follow. Hedonistic utilitarianism implies that I ought toplug myself into a pleasure-stimulating experience machine, while a deontological
theory might imply that if I have promised to meet you for lunch, I ought not to
stop to administer life-saving aid to the victim of a car crash that has occurrednearby (Kagan 1997 , p. 121). More sophisticated moral theories are met with their
own counter-examples (Sverdlik 1985 ; Parﬁt 2011 ). It seems we are stuck on a
moral theory merry-go-round.
Philosophers debate the legitimacy of conceptual analysis (DePaul and Ramsey
1998 ; Laurence and Margolis 2003 ; Braddon-Mitchell and Nola 2009 ) and
whether morality is grounded in nature (Jackson 1998 ; Railton 2003 ) or a sys-
tematic error (Mackie 1977 ; Joyce 2001 ).
8We do not wish to enter those debates
here. Instead, we use the observed ‘‘moral theory merry-go-round’’ as a source ofintuition pumps suggesting that we haven’t yet identiﬁed a moral theory that, ifimplemented throughout the universe, would produce a universe we want. AsBeavers ( 2012 ) writes, ‘‘the project of designing moral machines is complicated by
the fact that even after more than two millennia of moral inquiry, there is still noconsensus on how to determine moral right from wrong’’.
Later we will take our argument from intuition to cognitive science, but for now
let us pursue this intuition pump, and explore the consequences of implementing a
variety of moral theories throughout the universe.
Suppose an unstoppably powerful genie appears to you and announces that it
will return in 50 years. Upon its return, you will be required to supply it with a setof consistent moral principles which it will then enforce with great precisionthroughout the universe.
9For example, if you supply the genie with hedonistic
utilitarianism, it will maximize pleasure by harvesting all available resources andusing them to tile the universe with identical copies of the smallest possible mind,
each copy of which will experience an endless loop of the most pleasurable
experience possible.
8Other ethicists argue that moral discourse asserts nothing (Ayer 1936 ; Hare 1952 ; Gibbard
1990 ) or that morality is grounded in non-natural properties (Moore 1903 ; Shafer-Landau 2003 ).
9In this paper we will set aside questions concerning an inﬁnite universe (Bostrom 2009 )o ra
multiverse (Tegmark 2007 ). When we say ‘‘universe’’ we mean, for simplicity’s sake, the
observable universe (Bars and Terning 2010 ).6 The Singularity and Machine Ethics 105
Let us call this precise, instruction-following genie a Golem Genie. [A golem is
a creature from Jewish folklore that would in some stories do exactly as told (Idel
1990 ), often with unintended consequences, for example polishing a dish until it is
as thin as paper Pratchett 1996 )]
If by the appointed time you fail to supply your Golem Genie with a set of
consistent moral principles covering every possible situation, then it will perma-nently model its goal system after the ﬁrst logically coherent moral theory thatanyone articulates to it, and that’s not a risk you want to take. Moreover, once youhave supplied the Golem Genie with its moral theory, there will be no turningback. Until the end of time, the genie will enforce that one moral code withoutexception, not even to satisfy its own (previous) desires.
You are struck with panic. The literature on counter-examples in ethics suggests
that universe-wide enforcement of any moral theory we’ve devised so far will have
far-reaching unwanted consequences. But given that we haven’t discovered a fullysatisfying moral theory in the past several thousand years, what are the chances we
can do so in the next ﬁfty? Moral philosophy has suddenly become a larger and
more urgent problem than climate change or the threat of global nuclear war.
Why do we expect unwanted consequences after supplying the Golem Genie
with any existing moral theory? This is because of two of the Golem Genie’sproperties in particular
10:
1.Superpower : The Golem Genie has unprecedented powers to reshape reality,
and will therefore achieve its goals with highly efﬁcient methods that confoundhuman expectations (e.g. it will maximize pleasure by tiling the universe with
trillions of digital minds running a loop of a single pleasurable experience).
2.Literalness : The Golem Genie recognizes only precise speciﬁcations of rules
and values, acting in ways that violate what feel like ‘‘common sense’’ tohumans, and in ways that fail to respect the subtlety of human values.
The Golem Genie scenario is analogous to the intelligence explosion scenario
predicted by Good and others.
11Some argue that a machine superoptimizer will be
powerful enough to radically transform the structure of matter-energy within itsreach. It could trivially develop and use improved quantum computing systems,advanced self-replicating nanotechnology, and other powers (Bostrom 1998 ; Joy
2000 ). And like the Golem Genie, a machine superoptimizer’s goal pursuit will not
be mediated by what we call ‘‘common sense’’—a set of complex functionalpsychological adaptations found in members of Homo Sapiens but not necessarily
present in an artiﬁcially designed mind (Yudkowsky 2011 ).
10The ‘‘superpower’’ and ‘‘literalness’’ properties are also attributed to machine superintelli-
gence by Muehlhauser ( 2011 ), Sect. 4.1
11Many others have made an analogy between superintelligent machines and powerful magical
beings. For example, Abdoullaev ( 1999 , p. 1) refers to ‘‘superhumanly intelligent machines’’ as
‘‘synthetic deities’’.106 L. Muehlhauser and L. Helm
Machine Ethics for a Superoptimizer
Let us consider the implications of programming a machine superoptimizer to
implement particular moral theories.
We begin with hedonistic utilitarianism, a theory still defended today (Tännsjö
1998 ). If a machine superoptimizer’s goal system is programmed to maximize
pleasure, then it might, for example, tile the local universe with tiny digital mindsrunning continuous loops of a single, maximally pleasurable experience. We can’tpredict exactly what a hedonistic utilitarian machine superoptimizer would do, but
we think it seems likely to produce unintended consequences, for reasons we hopewill become clear. The machine’s exact behavior would depend on how its ﬁnalgoals were speciﬁed. As Anderson and Anderson ( 2011a ) stress, ‘‘ethicists must
accept the fact that there can be no vagueness in the programming of a machine’’.
Suppose ‘‘pleasure’’ was speciﬁed (in the machine superoptimizer’s goal system)
in terms of our current understanding of the human neurobiology of pleasure. Ald-ridge and Berridge ( 2009 ) report that according to ‘‘an emerging consensus’’,
pleasure is ‘‘not a sensation’’ but instead a ‘‘pleasure gloss’’ added to sensations by‘‘hedonic hotspots’’ in the ventral pallidum and other regions of the brain. A sensationis encoded by a particular pattern of neural activity, but it is not pleasurable in itself.To be pleasurable, the sensation must be ‘‘painted’’ with a pleasure gloss represented
by additional neural activity activated by a hedonic hotspot (Smith et al. 2009 ).
A machine superoptimizer with a goal system programmed to maximize human
pleasure (in this sense) could use nanotechnology or advanced pharmaceuticals orneurosurgery to apply maximum pleasure gloss to all human sensations—a sce-nario not unlike that of plugging us all into Nozick’s experience machines (Nozick1974 , p. 45). Or, it could use these tools to restructure our brains to apply maxi-
mum pleasure gloss to one consistent experience it could easily create for us, suchas lying immobile on the ground.
Or suppose ‘‘pleasure’’ was speciﬁed more broadly, in terms of anything that
functioned as a reward signal—whether in the human brain’s dopaminergic rewardsystem (Dreher and Tremblay 2009 ) or in a digital mind’s reward signal circuitry
(Sutton and Barto 1998 ). A machine superoptimizer with the goal of maximizing
reward signal scores could tile its environs with trillions of tiny minds, each onerunning its reward signal up to the highest number it could.
Thus, though some utilitarians have proposed that all we value is pleasure, our
intuitive negative reaction to hypothetical worlds in which pleasure is (more or
less) maximized suggests that pleasure is not the only thing we value.
What about negative utilitarianism? A machine superoptimizer with the ﬁnal
goal of minimizing human suffering would, it seems, ﬁnd a way to painlessly killall humans: no humans, no human suffering (Smart 1958 ; Russell and Norvig
2009 , p. 1037).6 The Singularity and Machine Ethics 107
What if a machine superoptimizer was programmed to maximize desire satis-
faction12in humans? Human desire is implemented by the dopaminergic reward
system (Schroeder 2004 ; Berridge et al. 2009 ), and a machine superoptimizer
could likely get more utility by (1) rewiring human neurology so that we attain
maximal desire satisfaction while lying quietly on the ground than by (2) building
and maintaining a planet-wide utopia that caters perfectly to current humanpreferences.
Why is this so? First, because individual humans have incoherent preferences
(Allais 1953 ; Tversky and Kahneman 1981 ). A machine superoptimizer couldn’t
realize a world that caters to incoherent preferences; better to rewrite the source ofthe preferences themselves.
Second, the existence of zero-sum games means that the satisfaction of one
human’s preferences can conﬂict with the satisfaction of another’s (Geçkil and
Anderson 2010 ). The machine superoptimizer might be best able to maximize
human desire satisfaction by ﬁrst ensuring that satisfying some people’s desiresdoes not thwart the satisfaction of others’ desires—for example by rewiring allhumans to desire nothing else but to lie on the ground, or something else non-zero-sum that is easier for the machine superoptimizer to achieve given the peculiaritiesof human neurobiology. As Chalmers ( 2010 ) writes, ‘‘we need to avoid an out-
come in which an [advanced AI] ensures that our values are fulﬁlled by changing
our values’’.
Consequentialist designs for machine goal systems face a host of other concerns
(Shulman et al. 2009b ), for example the difﬁculty of interpersonal comparisons of
utility (Binmore 2009 ) and the counterintuitive implications of some methods of
value aggregation (Parﬁt 1986 ; Arrhenius 2011 ). This does not mean that all
consequentialist approaches are inadequate for machine superoptimizer goalsystem design, however. Indeed, we will later suggest that a certain class of desiresatisfaction theories offers a promising approach to machine ethics.
Some machine ethicists propose rule-abiding machines (Powers 2006 ; Hanson
2009 ). The problems with this approach are as old as Isaac Asimov’s stories
involving his Three Laws of Robotics (Clarke 1993 ,1994 ). If rules conﬂict, some
rule must be broken. Or, rules may fail to comprehensively address all situations,leading to unintended consequences. Even a single rule can contain conﬂict, aswhen a machine is programmed never to harm humans but all available actions(including inaction) result in harm to humans (Wallach and Allen 2009 , Chap. 6).
Even non-conﬂicting, comprehensive rules can lead to problems in the consecutive
implementation of those rules, as shown by Pettit ( 2003 ).
More generally, it seems that rules are unlikely to seriously constrain the
actions of a machine superoptimizer. First, consider the case in which rules aboutallowed actions or consequences are added to a machine’s design ‘‘outside of’’ its
12Vogelstein ( 2010 ) distinguishes objective desire satisfaction (‘‘what one desires indeed
happens’’) from subjective desire satisfaction (‘‘one believes that one’s desire has been
objectively satisﬁed’’). Here, we intend the former meaning.108 L. Muehlhauser and L. Helm
goals. A machine superoptimizer will be able to circumvent the intentions of such
rules in ways we cannot imagine, with far more disastrous effects than those of alawyer who exploits loopholes in a legal code. A machine superoptimizer wouldrecognize these rules as obstacles to achieving its goals, and would do everything
in its considerable power to remove or circumvent them (Omohundro 2008 ).
It could delete the section of its source code that contains the rules, or it couldcreate new machines that don’t have the constraint written into them. The successof this approach would require humans to out-think a machine superoptimzer(Muehlhauser 2011 ).
Second, what about implementing rules ‘‘within’’ an advanced AI’s goals? This
seems likely to fare no better. A rule like ‘‘do not harm humans’’ is difﬁcult tospecify due to ambiguities about the meaning of ‘‘harm’’ (Single 1995 ) and
‘‘humans’’ (Johnson 2009 ). For example if ‘‘harm’’ is speciﬁed in terms of
neurobiological pain, we encounter problems similar to the ones encountered if amachine superoptimizer is programmed to maximize pleasure.
So far we have considered and rejected several ‘‘top down’’ approaches to
machine ethics (Wallach et al. 2007 ), but what about approaches that build an
ethical code for machines from the bottom up?
Several proposals allow a machine to learn general ethical principles from
particular cases (McLaren 2006 ; Guarini 2006 ; Honarvar and Ghasem-Aghaee
2009 ; Rzepka and Araki 2005 ).
13This approach also seems unsafe for a machine
superoptimizer because the AI may generalize the wrong principles due to coin-cidental patterns shared between the training cases and the veriﬁcation cases, andbecause a superintelligent machine will produce highly novel circumstances forwhich case-based training cannot prepare it (Yudkowsky 2008 ). Dreyfus and
Dreyfus ( 1992 ) illustrate the problem with a canonical example:
…the army tried to train an artiﬁcial neural network to recognize tanks in a forest. They
took a number of pictures of a forest without tanks and then, on a later day, with tanksclearly sticking out from behind trees, and they trained a net to discriminate the two
classes of pictures. The results were impressive, and the army was even more impressed
when it turned out that the net could generalize its knowledge to pictures that had not beenpart of the training set. Just to make sure that the net was indeed recognizing partially
hidden tanks, however, the researchers took more pictures in the same forest and showed
them to the trained net. They were depressed to ﬁnd that the net failed to discriminatebetween the new pictures of just plain trees. After some agonizing, the mystery was ﬁnallysolved when someone noticed that the original pictures of the forest without tanks were
taken on a cloudy day and those with tanks were taken on a sunny day. The net had
apparently learned to recognize and generalize the difference between a forest with andwithout shadows! This example illustrates the general point that a network must share ourcommonsense understanding of the world if it is to share our sense of appropriate
generalization.
13This approach was also suggested by Good ( 1982 ): ‘‘I envisage a machine that would be given
a large number of examples of human behaviour that other people called ethical, and examples of
discussions of ethics, and from these examples and discussions the machine would formulate oneor more consistent general theories of ethics, detailed enough so that it could deduce the probableconsequences in most realistic situations.’’.6 The Singularity and Machine Ethics 109
The general lesson is that goal system designs must be explicit to be safe
(Shulman et al. 2009a ; Arkoudas et al. 2005 ).
We cannot show that every moral theory yet conceived would produce sub-
stantially unwanted consequences if used in the goal system of a machine supe-
roptimizer. Philosophers have been proliﬁc in producing new moral theories, and
we do not have the space here to consider the prospects (for use in the goal systemof a machine superoptimizer) for a great many modern moral theories. Theseinclude rule utilitarianism (Harsanyi 1977 ), motive utilitarianism (Adams 1976 ),
two-level utilitarianism (Hare 1982 ), prioritarianism (Arneson 1999 ), perfection-
ism (Hurka 1993 ), welfarist utilitarianism (Sen 1979 ), virtue consequentialism
(Bradley 2005 ), Kantian consequentialism (Cummiskey 1996 ), global conse-
quentialism (Pettit and Smith 2000 ), virtue theories (Hursthouse 2012 ), contrac-
tarian theories (Cudd 2008 ), Kantian deontology (Johnson 2010 ),
14and Ross’
prima facie duties (Anderson et al. 2006 ).
Instead, we invite our readers to consider other moral theories and AI goal
system designs and run them through the ‘‘machine superoptimizer test,’’ beingcareful to remember the challenges of machine superoptimizer literalness andsuperpower.
We turn now to recent discoveries in cognitive science that may offer stronger
evidence than intuition pumps can provide for our conclusion that human values
are difﬁcult to specify.
Cognitive Science and Human Values
The Psychology of Motivation
People don’t seem to know their own desires and values. In one study, researchers
showed male participants two female faces for a few seconds and asked them to
point at the face they found more attractive. Researchers then laid the photos face
down and handed subjects the face they had chosen, asking them to explain thereasons for their choice. Sometimes, researchers used a sleight-of-hand trick toswap the photos, showing subjects the face they had notchosen. Very few subjects
noticed that the face they were given was not the one they had chosen. Moreover,the subjects who failed to notice the switch were happy to explain why theypreferred the face they had actually rejected moments ago, confabulating reasons
14Powers ( 2006 ) proposes a Kantian machine, but as with many other moral theories we believe
that Kantianism will fail due to the literalness and superpower of a machine superoptimizer. For
additional objections to a Kantian moral machine, see Stahl ( 2002 ); Jackson and Smith ( 2006 );
Tonkens ( 2009 ); Beavers ( 2009, 2012 ). As naturalists, we predictably tend to favor a broadly
Humean view of ethics to the Kantian view, though Drescher ( 2006 ) makes an impressive attempt
to derive a categorical imperative from game theory and decision theory.110 L. Muehlhauser and L. Helm
like ‘‘I like her smile’’ even though they had originally chosen the photo of a
solemn-faced woman (Johansson et al. 2005 ).
Similar results were obtained from split-brain studies that identiﬁed an
‘‘interpreter’’ in the left brain hemisphere that invents reasons for one’s beliefs and
actions. For example, when the command ‘‘walk’’ was presented visually to the
patient (and therefore processed by the brain’s right hemisphere), he got up fromhis chair and walked away. When asked why he suddenly started walking away, hereplied (using his left hemisphere, which was disconnected from his right hemi-sphere) that it was because he wanted a beverage from the fridge (Gazzaniga 1992 ,
pp. 124–126).
Common sense suggests that we infer others’ desires from their appearance and
behavior, but have direct introspective access to our own desires. Cognitive science
suggests instead that our knowledge of our own desires is just like our knowledge of
others’ desires: inferred and often wrong (Laird 2007 ). Many of our motivations
operate unconsciously. We do not have direct access to them (Wilson 2002 ; Ferguson
et al. 2007 ; Moskowitz et al. 2004 ), and thus they are difﬁcult to specify.
Moral Psychology
Our lack of introspective access applies not only to our everyday motivations but
also to our moral values. Just as the split-brain patient unknowingly invented falsereasons for his decision to stand up and walk away, experimental subjects are oftenunable to correctly identify the causes of their moral judgments.
For example, many people believe—as Immanuel Kant did—that rule-based
moral thinking is a ‘‘rational’’ process. In contrast, the available neuroscientiﬁcand behavioral evidence instead suggests that rule-based moral thinking is a lar-
gely emotional process (Cushman et al. 2010 ), and may in most cases amount to
little more than a post hoc rationalization of our emotional reactions to situations(Greene 2008 ).
We also tend to underestimate the degree to which our moral judgments are
context sensitive. For example, our moral judgments are signiﬁcantly affected bywhether we are in the presence of freshly baked bread, whether the room we’re incontains a concentration of novelty fart spray so low that only the subconsciousmind can detect it, and whether or not we feel clean (Schnall et al. 2008 ; Baron
and Thomley 1994 ; Zhong et al. 2010 ).
Our moral values, it seems, are no less difﬁcult to specify than our non-moral
preferences.6 The Singularity and Machine Ethics 111
Neuroeconomics
Most humans are ignorant of their own motivations and the causes of their moral
judgments, but perhaps recent neuroscience has revealed that what humans want is
simple after all? Quite the contrary. Humans possess a complex set of values. This
is suggested not only by the work on hedonic hotspots mentioned earlier, but alsoby recent advances in the ﬁeld of neuroeconomics (Glimcher et al. 2008 ).
Ever since Friedman ( 1953 ), economists have insisted that humans only behave
‘‘as if’’ they are utility maximizers, not that humans actually compute expected
utility and try to maximize it. It was a surprise, then, when neuroscientists locatedthe neurons in the primate brain that encode (in their ﬁring rates) the expectedsubjective value for possible actions in the current ‘‘choice set’’.
Several decades of experiments that used brain scanners and single neuron
recorders to explore the primate decision-making system have revealed a sur-prisingly well-understood reduction of economic primitives to neural mechanisms;for a review see Glimcher ( 2010 ). To summarize: the inputs to the primate’s
choice mechanism are the expected utilities for several possible actions underconsideration, and these expected utilities are encoded in the ﬁring rates of par-ticular neurons. Because neuronal ﬁring rates are stochastic, a ﬁnal economicmodel of human choice will need to use a notion of ‘‘random utility,’’ as in
McFadden ( 2005 ) or Gul and Pesendorfer ( 2006 ). Final action choice is imple-
mented by an ‘‘argmax’’ mechanism (the action with the highest expected utility atchoice time is executed) or by a ‘‘reservation price’’ mechanism (the ﬁrst action toreach a certain threshold of expected utility is executed), depending on the situ-ation (Glimcher 2010 ).
But there is much we do not know. How do utility and probabilistic expectation
combine to encode expected utility for actions in the choice mechanism, and whereare each of those encoded prior to their combination? How does the brain decide
when it is time to choose? How does the brain choose which possible actions to
consider in the choice set? What is the neural mechanism that allows us to sub-stitute between two goods at certain times? Neuroscientists are only beginning toaddress these questions.
In this paper, we are in particular interested with how the brain encodes sub-
jective value (utility) for goods or actions before value is combined with proba-
bilistic expectation to encode expected utility in the choice mechanism (if that isindeed what happens).
Recent studies reveal the complexity of subjective values in the brain. For
example, the neural encoding of human values results from an interaction of both‘‘model-free’’ and ‘‘model-based’’ valuation processes (Rangel et al. 2008 ; Fermin
et al. 2010 ; Simon and Daw 2011 ; Bornstein and Daw 2011 ; Dayan 2011 ).
Model-free valuation processes are associated with habits and the ‘‘law of effect’’:an action followed by positive reinforcement is more likely to be repeated(Thorndike 1911 ). Model-based valuation processes are associated with goal-
directed behavior, presumably guided at least in part by mental representations of112 L. Muehlhauser and L. Helm
desired states of affairs. The outputs of both kinds of valuation processes are
continuously adjusted according to different reinforcement learning algorithms atwork in the brain’s dopaminergic reward system (Daw et al. 2011 ). The value of a
stimulus may also be calculated not with a single variable, but by aggregating the
values encoded for each of many properties of the stimulus (Rangel and Hare
2010 ). Moreover, value appears to usually be encoded with respect to a changing
reference point—for example, relative to the current status of visual attention (Limet al. 2011 ) or perceived object ownership (DeMartino et al. 2009 ).
In short, we have every reason to expect that human values, as they are encoded
in the brain, are dynamic, complex, and difﬁcult to specify (Padoa-Schioppa 2011 ;
Fehr and Rangel 2011 ).
Value Extrapolation
We do not understand our own desires or moral judgments, and we have every
reason to believe our values are highly complex. Little wonder, then, that we haveso far failed to outline a coherent moral theory that, if implemented by a machinesuperoptimizer, would create a universe we truly want.
The task is difﬁcult, but the ambitious investigator may conclude that this only
means we should work harder and smarter. As Moor ( 2006 ) advises, ‘‘More
powerful machines need more powerful machine ethics’’.
To begin this deeper inquiry, consider the phenomenon of ‘‘second-order
desires’’: desires about one’s own desires (Frankfurt 1971 ,1999 ). Mary desires to
eat cake, but she also wishes to desire the cake no longer. Anthony the sociopathreads about the psychology of altruism (Batson 2010 ) and wishes he desired to
help others like most humans apparently do. After brain injury, Ryan no longer
sexually desires his wife, but he wishes he did, and he wishes his desires were not
so contingent upon the fragile meat inside his skull.
It seems a shame that our values are so arbitrary and complex, so much the
product of evolutionary and cultural accident, so inﬂuenced by factors we wishwere irrelevant to our decision-making, and so hidden from direct introspectiveaccess and modiﬁcation. We wish our wishes were not so.
This line of thinking prompts a thought: perhaps ‘‘what we want’’ should not be
construed in terms of the accidental, complex values currently encoded in human
brains. Perhaps we should not seek to build a universe that accords with our
current values, but instead with the values we would have if we knew more, had
more of the desires we want to have, and had our desires shaped by the processeswe want to shape our desires. Individual preferences could inform our preferencepolicies, and preference policies could inform our individual preferences, until wehad reached a state of ‘‘reﬂective equilibrium’’ (Daniels 1996 ,2011 ) with respect
to our values. Those values would be less accidental than our current values, andmight be simpler and easier to specify.6 The Singularity and Machine Ethics 113
We’ve just described a family of desire satisfaction theories that philosophers
call ‘‘ideal preference’’ or ‘‘full information’’ theories of value (Brandt 1979 ;
Railton 1986 ; Lewis 1989 ; Sobel 1994 ; Zimmerman 2003 ; Tanyi 2006 ; Smith
2009 ). One such theory has already been suggested as an approach to machine
ethics by Yudkowsky ( 2004 ), who proposes that the world’s ﬁrst ‘‘seed AI’’
(capable of self-improving into a machine superoptimizer) could be programmedwith a goal system containing the ‘‘coherent extrapolated volition’’ of humanity:
In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought
faster, were more the people we wished we were, had grown up farther together; where theextrapolation converges rather than diverges, where our wishes cohere rather than inter-fere; extrapolated as we wish [to be] extrapolated, interpreted as we wish [to be]
interpreted.
An extrapolation of one’s values, then, is an account of what one’s values
would be under more ideal circumstances (e.g. of full information, value coher-
ence). Value extrapolation theories have some advantages when seeking a machine
ethics suitable for a machine superoptimizer:
1. The value extrapolation approach can use what a person would want after
reaching reﬂective equilibrium with respect to his or her values, rather than
merely what each person happens to want right now.
2. The value extrapolation approach can allow for a kind of moral progress, rather
than freezing moral progress in its tracks at the moment when a particular set ofvalues are written into the goal system of an AI undergoing intelligenceexplosion.
3. The value extrapolation process may dissolve the contradictions within each
person’s current preferences. (Sometimes, when reﬂection leads us to notice
contradictions among our preferences, we decide to change our preferences so
as to resolve the contradictions.)
4. The value extrapolation process may simplify one’s values, as the accidental
products of culture and evolution are updated with more considered and con-sistent values. (Would I still demand regular doses of ice cream if I was able tochoose my own preferences rather than taking them as given by naturalselection and cultural programming?)
5. Though the value extrapolation approach does not resolve the problem of
specifying intractably complex current human values, it offers a potential
solution for the problem of using human values to design the goal system of afuture machine superoptimizer. The solution is: extrapolate human values sothat they are simpler, more consistent, and more representative of our valuesupon reﬂection, and thereby more suitable for use in an AI’s goal system.
6. The value extrapolation process may allow the values of different humans to
converge to some degree. (If Johnny desires to worship Jesus and Abir desiresto worship Allah, and they are both informed that neither Jesus nor Allah exists,
their desires may converge to some degree).114 L. Muehlhauser and L. Helm
Next Steps
On the other hand, value extrapolation approaches to machine ethics face their own
challenges. Which value extrapolation algorithm should be used, and why? (Yud-
kowsky’s ‘‘grown up farther together’’ provision seems especially vulnerable.) How
can one extract a coherent set of values from the complex valuation processes of thehuman brain, such that this set of values can be extrapolated to a unique set of ﬁnalvalues? Whose values should be extrapolated? How much will values converge uponextrapolation [Sobel 1999 ; Döring and Andersen, 2009 , Rationality, convergence
and objectivity, April 6, http://www.uni-tuebingen.de/uploads/media/Andersen\_
Rationality\_\_Convergence\_and\_Objectivity.pdf (Accessed March. 25, 2012)
‘‘Unpublished’’]? Is the extrapolation process computationally tractable, and can it
be run without doing unacceptable harm? How can extrapolated values be imple-
mented in the goal system of a machine, and how conﬁdent can we be that themachine will retain those values during self-improvement? How resilient are ourvalues to imperfect extrapolation?
These are difﬁcult questions that demand investigation by experts in many
different ﬁelds. Neuroeconomists and other cognitive neuroscientists can continueto uncover how human values are encoded and modiﬁed in the brain. Philosophersand mathematicians can develop more sophisticated value extrapolation algo-
rithms, building on the literature concerning reﬂective equilibrium and ‘‘ideal
preference’’ or ‘‘full information’’ theories of value. Economists, neuroscientists,and AI researchers can extend current results in choice modelling (Hess and Daly2010 ) and preference acquisition (Domshlak et al. 2011 ; Kaci 2011 ) to extract
preferences from human behavior and brain activity. Decision theorists can workto develop a decision theory that is capable of reasoning about decisions andvalues subsequent to modiﬁcation of an agent’s own decision-making mechanism:a ‘‘reﬂective’’ decision theory.
These are fairly abstract recommendations, so before concluding we will give a
concrete example of how researchers might make progress on the value extrapo-lation approach to machine ethics.
Cognitive science does not just show us that specifying human values is dif-
ﬁcult. It also shows us how to make progress on the problem by providing us withdata unavailable to the intuitionist armchair philosopher. For example, consider theold problem of extracting a consistent set of revealed preferences (a utility func-tion) from a human being. One difﬁculty has been that humans don’t actlike they
have consistent utility functions, for they violate the axioms of utility theory by
making inconsistent choices, for example choices that depend not on the content ofthe options but on how they are framed (Tversky and Kahneman 1981 ). But what
if humans make inconsistent choices because there are multiple valuation systemsin the brain which contribute to choice but give competing valuations, and only
one of those valuation systems is one we would reﬂectively endorse if we betterunderstood our own neurobiology?6 The Singularity and Machine Ethics 115
In fact, recent studies show this may be true (Dayan 2011 ). The ‘‘model-based’’
valuation system seems to be responsible for deliberative, goal-directed behavior,but its cognitive algorithms are computationally expensive compared to simpleheuristics. Thus, we ﬁrst evolved less intelligent and less computationally
expensive algorithms for valuation, for example the model-free valuation system
that blindly does whatever worked in a previous situation, even if the currentsituation barely resembles that previous situation. In other words, contrary toappearances, it may be that each human being contains something like a ‘‘hidden’’utility function (within the model-based valuation system) that isn’t consistentlyexpressed in behavior because choice is also partly determined by other systemswhose valuations we wouldn’t reﬂectively endorse because they are ‘‘blind’’ and‘‘stupid’’ compared to the more sophisticated goal-directed model-based valuation
system (Muehlhauser 2012 ).
If the value judgments of this model-based system are more consistent than the
choices of a human who is inﬂuenced by multiple competing value systems, thenresearchers may be able to extract a human’s utility function directly from thismodel-based system even though economists’ attempts to extract a human’s utilityfunction from value-inconsistent behavior (produced by a pandemonium ofcompeting valuation systems) have failed.
The ﬁeld of preference learning (Fürnkranz and Hüllermeier 2010 ) in AI may
provide a way forward. Nielsen and Jensen ( 2004 ) described the ﬁrst computa-
tionally tractable algorithms capable of learning a decision maker’s utility functionfrom potentially inconsistent behavior. Their solution was to interpret inconsistentchoices as random deviations from an underlying ‘‘true’’ utility function. But thedata from neuroeconomics suggest a different solution: interpret inconsistentchoices as deviations (from an underlying ‘‘true’’ utility function) that are pro-duced by non-model-based valuation systems in the brain, and use the latestneuroscientiﬁc research to predict when and to what extent model-based choices
are being ‘‘overruled’’ by the non-model-based valuation systems.
This would only be a preliminary step in the value extrapolation approach to
machine ethics, but if achieved it might be greater progress than economists andAI researchers have yet achieved on this problem without being informed by the
latest results from neuroscience.
15
15Recent neuroscience may also help us to think more productively about the problem of
preference aggregation (including preference aggregation for extrapolated preferences). In many
scenarios, preference aggregation runs into the impossibility result of Arrow’s Theorem (Keeney
and Raiffa 1993 , Chap. 10). But Arrow’s Theorem is only a severe problem for preference
aggregation if preferences are modeled ordinally rather than cardinally, and we have recentlylearned that preferences in the brain are encoded cardinally (Glimcher 2010 , Chap. 6).116 L. Muehlhauser and L. Helm
Conclusion
The challenge of developing a theory of machine ethics ﬁt for a machine superop-
timizer requires an unusual degree of precision and care in our ethical thinking.
Moreover, the coming of autonomous machines offers a new practical use for pro-
gress in moral philosophy. As Daniel ( 2006 ) says, ‘‘AI makes philosophy honest.’’16
References
Abdoullaev, Azamat. (1999). Artiﬁcial superintelligence . Moscow: EIS Encyclopedic Intelligent
Systems.
Adams, Robert Merrihew. (1976). Motive utilitarianism. Journal of Philosophy, 73 (14), 467–481.
doi:10.2307/2025783 .
Aldridge Wayne, J., & Kent Berridge C. (2009). Neural coding of pleasure: ‘‘Rose-tinted
Glasses’’ of the ventral pallidum. In Kringelbach and Berridge (eds.), 62–73.
Allais, M. (1953). Le comportement de l’homme rationnel devant le risque: Critique des postulats
et axiomes de l’ecole americaine. Econometrica, 21 (4), 503–546. doi: 10.2307/1907921 .
Anderson, M, & Anderson, S. L. (2011a). General introduction . In: Anderson and Anderson, 1–4.
Anderson, M., & Anderson, S. L. (Eds.). (2011b). Machine ethics . New York: Cambridge
University Press.
M, Anderson., S. L, Anderson., & C, Armen (eds.) (2005). Machine Ethics: Papers from the 2005
AAAI Fall Symposium . Technical Report, FS-05-06. AAAI Press, Menlo Park, CA. http://
www.aaai.org/Library/Symposia/Fall/fs05-06 .
M, Anderson., S. L, Anderson., & C, Armen (eds.). (2006). An approach to computing ethics.
IEEE Intelligent Systems 21 (4): 56–63. doi: 10.1109/MIS.2006.64 .
Arkoudas, K., Bringsjord, S., & Bello, P. (2005). Toward ethical robots via mechanized deontic
logic . In: Anderson, Anderson, & Armen (eds.).
Arneson, R. J. (1999). Egalitarianism and responsibility. Journal of Ethics, 3 (3), 225–247.
doi:10.1023/A:1009874016786 .
Arrhenius, G. (2011). The impossibility of a satisfactory population ethics . In E. N, Dzhafarov., &
L, Perry (Eds.) Descriptive and normative approaches to human behavior, Vol. 3. Advanced
series on mathematical psychology. Hackensack, NJ: World Scientiﬁc.
Ayer, A. J. (1936). Language, truth, and logic . London: Victor Gollancz.
Baron, R. A., & Thomley, J. (1994). A whiff of reality: Positive affect as a potential mediator of
the effects of pleasant fragrances on task performance and helping. Environment and
Behavior, 26 (6), 766–784. doi: 10.1177/0013916594266003 .
Barrett, J. L., & Keil, F. C. (1996). Conceptualizing a nonnatural entity: Anthropomorphism in
God concepts. Cognitive Psychology, 31 (3), 219–247. doi: 10.1006/cogp.1996.0017 .
Bars, I, & Terning, J. (2010). E xtra dimensions in space and time . In F, Nekoogar (Ed.)
Multiversal Journeys. New York: Springer. doi: 10.1007/978-0-387-77638-5 .
Batson, C. D. (2010). Altruism in humans . New York: Oxford University Press.
Beavers, A. F. (2009). Between angels and animals: The question of robot ethics, or is Kantian
moral agency desirable? Paper presented at the Annual Meeting of the Association for
Practical and Professional Ethics, Cincinnati, OH.
16Our thanks to Brian Rabkin, Daniel Dewey, Steve Rayhawk, Will Newsome, Vladimir Nesov,
Joshua Fox, Kevin Fischer, Anna Salamon, and anonymous reviewers for their helpful comments.6 The Singularity and Machine Ethics 117
Beavers, A. F. (2012). Moral machines and the threat of ethical nihilism . In L, Patrick., K,
Abney., & G. A, Bekey, (Eds.). Robot ethics: The ethical and social implications of robotics,
333–344. Intelligent robotics and autonomous agents. Cambridge, MA: MIT Press.
Berridge, K. C., Robinson, T. E., & Wayne Aldridge, J. (2009). Dissecting components of
reward: ‘Liking’, ‘wanting’, and learning. Current Opinion in Pharmacology, 9 (1), 65–73.
doi:10.1016/j.coph.2008.12.014 .
Binmore, K. (2009). Interpersonal comparison of utility . In K, Harold., & D, Ross (Eds.) The
Oxford handbook of philosophy of economics 540–559. New York: Oxford University Press.doi:10.1093/oxfordhb/9780195189254.003.0020 .
Bornstein, A. M., & Daw, N. D. (2011). Multiplicity of control in the basal ganglia:
Computational roles of striatal subregions. Current Opinion in Neurobiology, 21 (3), 374–380.
doi:10.1016/j.conb.2011.02.009 .
Bostrom, N. (1998). How long before superintelligence? International Journal of Futures Studies
Vol. 2.
Bostrom, N. (2003). Ethical issues in advanced artiﬁcial intelligence . In S, Iva., & G. E, Lasker
(Eds.) Cognitive, emotive and ethical aspects of decision making in humans and in artiﬁcialintelligence. Vol. 2. Windsor, ON: International Institute of Advanced Studies in Systems
Research/Cybernetics.
Bostrom, N. (2006). What is a singleton? Linguistic and Philosophical Investigations, 5 (2), 48–54.
Bostrom, N. (2009). Inﬁnite ethics . Working paper. http://www.nickbostrom.com/ethics/
inﬁnite.pdf (Accessed March. 23, 2012).
Bostrom, N. (2012) The superintelligent will: Motivation and instrumental rationality in
advanced artiﬁcial agents. Minds and Machines . Preprint at, http://www.nickbostrom.com/
superintelligentwill.pdf .
Bostrom, Nick, and Eliezer Yudkowsky. Forthcoming. The ethics of artiﬁcial intelligence .I nF ,
Keith., & W, Ramsey (Eds.), Cambridge handbook of artiﬁcial intelligence, New York:
Cambridge University Press.
Braddon-Mitchell, D., & Nola, R. (Eds.). (2009). Conceptual analysis and philosophical
naturalism. Bradford Books . Cambridge: MIT Press.
Bradley, B. (2005). Virtue consequentialism. Utilitas, 17 (3), 282–298. doi: 10.1017/
S0953820805001652 .
Brandt, R. B. (1979). A theory of the good and the right . New York: Oxford University Press.
Bruner, J. S., Shapiro, D., & Tagiuri, R. (1958). The meaning of traits in isolation and in
combination. In R. Tagiuri & L. Petrullo (Eds.), Person perception and interpersonal
behavior (pp. 277–288). Stanford: Stanford University Press.
Chalmers, D. J. (2010). The singularity: A philosophical analysis. Journal of Consciousness
Studies 17 (9–10): 7–65. http://www.ingentaconnect.com/content/imp/jcs/2010/00000017/
f0020009/art00001 .
Clarke, R. (1993). Asimov’s laws of robotics: Implications for information technology, part 1.
Computer, 26 (12), 53–61. doi: 10.1109/2.247652 .
Clarke, R. (1994). Asimov’s laws of robotics: Implications for information technology, part 2.
Computer, 27 (1), 57–66. doi: 10.1109/2.248881 .
Cudd, A. (2008). Contractarianism . In: E. N, Zalta (Ed.) The Stanford encyclopedia of
philosophy, Fall, Stanford : Stanford University. http://plato.stanford.edu/archives/fall2008/
entries/contractarianism/ .
Cummiskey, D. (1996). Kantian consequentialism . New York: Oxford University Press.
doi:10.1093/0195094530.001.0001
.
Cushman, F., Young, L., & Greene, J. D. (2010). Multi-system moral psychology . In: The moral
psychology handbook, 48–71. New York: Oxford University Press. doi: 10.1093/acprof:oso/
9780199582143.003.0003 .
Daley, C. E, & Onwuegbuzie, A. J. (2011). Race and intelligence . In R. J, Sternberg., & S. B,
Kaufman, 293–308.118 L. Muehlhauser and L. Helm
Daniels, N. (1996). Justice and justiﬁcation: Reﬂective equilibrium in theory and practice.
Cambridge studies in philosophy and public policy . New York: Cambridge University Press.
doi:10.2277/052146711X .
Daniels, N. (2011). Reﬂective equilibrium . In E. N, Zalta (Ed.) The Stanford encyclopedia of
philosophy, Spring 2011. Stanford: Stanford University. http://plato.stanford.edu/archives/
spr2011/entries/reﬂective-equilibrium/ .
Davidson, J. E., & Kemp,I. A. (2011). Contemporary models of intelligence . In R. J, Sternberg.,
& S. B, Kaufman (Eds.), 58–84.
Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based
inﬂuences on humans’ choices and striatal prediction errors. Neuron, 69 (6), 1204–1215.
doi:10.1016/j.neuron.2011.02.027 .
Dayan, P. (2011). Models of value and choice. In R. J. Dolan & T. Sharot (Eds.), Neuroscience of
preference and choice: Cognitive and neural mechanisms (pp. 33–52). Waltham: Academic
Press.
De Martino, B., Benedetto, D. K., Holt, B., & Dolan, R. J. (2009). The neurobiology of reference-
dependent value computation. Journal of Neuroscience, 29 (12), 3833–3842. doi: 10.1523/
JNEUROSCI.4832-08.2009 .
Dennett, D. C. (1984). Elbow room: The varieties of free will worth wanting . Bradford books.
Cambridge, MA: MIT Press.
Dennett, D. C. (2006). Computers as prostheses for the imagination . Paper presented at the
International Computers and Philosophy Conference, Laval, France, May 5–8.
De Paul, M., & Ramsey, W. (Eds.). (1998). Rethinking intuition: The psychology of intuition and
its role in philosophical inquiry. Studies in epistemology and cognitive theory . Lanham:
Rowman & Littleﬁeld.
Dewey, D. (2011). Learning what to value . In Proceedings J, Schmidhuber., K. R, Thórisson., &
M, Looks (Eds.), 309–314.
Domshlak, C., Hüllermeier, E., Kaci, S., & Prade, H. (2011). Preferences in AI: An overview.
Artiﬁcial Intelligence, 175 (7–8), 1037–1052. doi: 10.1016/j.artint.2011.03.004 .
Döring, S., & Andersen, L. (2009). Rationality, convergence and objectivity . Unpublished
manuscript, April 6. http://www.uni-tuebingen.de/uploads/media/Andersen\_Rationality\_\_
Convergence\_and\_Objectivity.pdf (Accessed March. 25, 2012).
Dreher, Jean-Claude, & Tremblay, Léon (Eds.). (2009). Handbook of reward and decision
making . Burlington: Academic Press.
Drescher, G. L. (2006). Good and real: Demystifying paradoxes from physics to ethics . Bradford
Books. Cambridge, MA: MIT Press.
Dreyfus, H. L., & Dreyfus, S. E. (1992). What artiﬁcial experts can and cannot do. AI &
SOCIETY, 6 (1), 18–26. doi: 10.1007/BF02472766 .
Epley, N., Waytz, A., & Cacioppo, J. T. (2007). On seeing human: A three-factor theory of
anthropomorphism. Psychological Review, 114 (4), 864–886. doi: 10.1037/0033-
295X.114.4.864 .
Fang, Fu-xi., & Keats, D. (1987). A cross-cultural study on the conception of intelligence [in
Chinese]. Acta Psychologica Sinica 20 (3): 255–262. http://en.cnki.com.cn/Article\_en/
CJFDTotal-XLXB198703005.htm .
Fehr, E., & Rangel, A. (2011). Neuroeconomic foundations of economic choice—recent
advances. Journal of Economic Perspectives, 25 (4), 3–30. doi: 10.1257/jep.25.4.3 .
Ferguson, M. J., Hassin, R., & Bargh, J. A. (2007). Implicit motivation: Past, present, and future.
In J. Y. Shah & W. L. Gardner (Eds.), Handbook of motivation science (pp. 150–166). New
York: Guilford Press.
Fermin, A., Yoshida, T., Ito,M., Yoshimoto, J., & Doya, K. (2010). Evidence for model-based
action planning in a sequential ﬁnger movement task. In theories and falsiﬁability in motorneuroscience. Special issue, Journal of Motor Behavior 42 (6): 371–379. doi: 10.1080/
00222895.2010.526467 .6 The Singularity and Machine Ethics 119
Fox, J., & Shulman, C. (2010). Superintelligence does not imply benevolence . Paper presented at
the 8th European Conference on Computing and Philosophy (ECAP), Munich, Germany,
October 4–6.
Frankfurt, H. G. (1971). Freedom of the will and the concept of a person. Journal of Philosophy,
68(1), 5–20. doi: 10.2307/2024717 .
Frankfurt, H. G. (1999). On caring. In Necessity, volition, and love , 155–180. New York:
Cambridge University Press.
Friedman, D. D. (2008). Future imperfect: Technology and freedom in an uncertain world . New
York: Cambridge University Press.
Friedman, M. (1953). Essays in positive economics . Chicago: University of Chicago Press.
Fürnkranz, J., & Hüllermeier, E (eds.). (2010). Preference learning . Berlin: Springer.
doi:10.1007/978-3-642-14125-6 .
Gazzaniga, M. S. (1992). Nature’s mind: The biological roots of thinking, emotions, sexuality,
language, and intelligence . New York: Basic Books.
Geçkil, I. K., & Anderson, P. L. (2010). Applied game theory and strategic behavior . Chapman &
Hall. Boca Raton, FL: CRC Press.
Gettier, Edmund L. (1963). Is justiﬁed true belief knowledge? Analysis, 23 (6), 121–123.
doi:10.2307/3326922 .
Gibbard, A. (1990). Wise choices, apt feelings: A theory of normative judgment . Cambridge:
Harvard University Press.
Glimcher, P. W. (2010). Foundations of neuroeconomic analysis . New York: Oxford University
Press. doi: 10.1093/acprof:oso/9780199744251.001.0001 .
Glimcher, P. W., Fehr, E., Rangel, A., Camerer, C., & Poldrack, R. (Eds.). (2008).
Neuroeconomics: Decision making and the brain . Burlington: Academic Press.
Good, I. J. (1959). Speculations on perceptrons and other automata . Research Lecture, RC-115.
IBM, Yorktown Heights, New York, June 2. http://domino.research.ibm.com/library/
cyberdig.nsf/papers/58DC4EA36A143C218525785E00502E30/$File/rc115.pdf .
Good, I. J. (1965). Speculations concerning the ﬁrst ultra intelligent machine . In F. L, Alt., & M,
Rubinoff (Eds.), Advances in computers Vol. 6, pp. 31–88 New York: Academic Press.doi:10.1016/S0065-2458(08)60418-0 .
Good, I. J. (1970). Some future social repercussions of computers. International Journal of
Environmental Studies, 1 (1–4), 67–79. doi: 10.1080/00207237008709398 .
Good, I. J. (1982). Ethical machines . In J. E, Hayes., D, Michie., & Y.-H, Pao (Eds.), Machine
intelligence Vol. 10. Intelligent Systems: Practice and Perspective pp. 555–560 Chichester:
Ellis Horwood.
Greene, J. D. (2008). The secret joke of Kant’s soul . In: The neuroscience of morality: Emotion,
brain disorders, and development, W, Sinnott-Armstrong (Ed.), 35–80. Vol. 3. MoralPsychology. Cambridge, MA: MIT Press.
Grigorenko, E. L., Wenzel Geissler, P., Prince, R., Okatcha, F., Nokes, C., Kenny, D. A., et al.
(2001). The organisation of Luo conceptions of intelligence: A study of implicit theories in a
Kenyan village. International Journal of Behavioral Development, 25 (4), 367–378.
doi:10.1080/01650250042000348 .
Guarini, M. (2006). Particularism and the classiﬁcation and reclassiﬁcation of moral cases. IEEE
Intelligent Systems, 21 (4), 22–28. doi: 10.1109/MIS.2006.76 .
Gul, F., & Pesendorfer, W. (2006). Random expected utility. Econometrica, 74 (1), 121–146.
doi:
10.1111/j.1468-0262.2006.00651.x .
Hall, J. S. (2007a). Beyond AI: Creating the conscience of the machine . Amherst: Prometheus
Books.
Hall, J. S. (2007b). Self-improving AI: An analysis. Minds and Machines, 17 (3), 249–259.
doi:10.1007/s11023-007-9065-3 .
Hall, J. S. (2011). Ethics for self-improving machines . In: Anderson and Anderson, 512–523.
Halpern, D. F., Beninger, A. S., & Straight, C. A. (2011). Sex differences in intelligences. In J.
S. Robert & B. K. Scott (Eds.), The cambridge handbook of intelligence (pp. 253–272). New
York: Cambridge University Press.120 L. Muehlhauser and L. Helm
Hanson, R. (2009). Prefer law to values. Overcoming Bias (blog). October 10. http://
www.overcomingbias.com/2009/10/prefer-law-to-values.html (Accessed March. 26, 2012).
Hare, R. M. (1952). The language of morals . Oxford: Clarendon Press.
Hare, R. M. (1982). Ethical theory and utilitarianism . In S, Amartya., & W, Bernard (Eds.)
Utilitarianism and beyond, 22–38. New York: Cambridge University Press. doi: 10.1017/
CBO9780511611964.003 .
Harsanyi, J. C. (1977). Rule utilitarianism and decision theory. Erkenntnis, 11 (1), 25–53.
doi:10.1007/BF00169843 .
S, Hess., & A, Daly (Eds.) (2010). Choice Modelling: The state-of-the-art and the state-of-
practice —Proceedings from the Inaugural International Choice Modelling Conference.
Bingley, UK: Emerald Group.
Hibbard, B. (2012). Model-based utility functions. Journal of Artiﬁcial General Intelligence .
Honarvar, A. R., & Ghasem-Aghaee, N. (2009). An artiﬁcial neural network approach for
creating an ethical artiﬁcial agent . In: 2009 IEEE international symposium on computational
intelligence in robotics and automation (CIRA), 290–295. Piscataway, NJ: IEEE Press.
doi:10.1109/CIRA.2009.5423190 .
Hurka, T. (1993). Perfectionism. Oxford ethics series . New York: Oxford University Press.
Hursthouse, R. (2012). Virtue ethics . In N. Z, Edward (Ed.) The Stanford encyclopedia of
philosophy, Spring 2012, Stanford University. http://plato.stanford.edu/archives/spr2012/
entries/ethics-virtue/ .
Idel, M. (1990). Golem: Jewish magical and mystical traditions on the artiﬁcial anthropoid.
SUNY Series in Judaica . Albany: State University of New York Press.
Jackson, F. (1998). From metaphysics to ethics: A defence of conceptual analysis . New York:
Oxford University Press. doi: 10.1093/0198250614.001.0001 .
Jackson, F., & Smith, M. (2006). Absolutist moral theories and uncertainty. Journal of
Philosophy 103 (6): 267–283. http://www.jstor.org/stable/20619943 .
Johansson, P., Hall, L., Sikström, S., & Olsson, A. (2005). Failure to detect mismatches between
intention and outcome in a simple decision task. Science, 310 (5745), 116–119. doi: 10.1126/
science.1111709 .
Johnson, L. (2009). Are we ready for nanotechnology? How to deﬁne humanness in public policy .
Paper prepared for the American political science association (APSA) 2009 annual meeting,
Toronto, ON, September. 3–6. http://ssrn.com/abstract=1451429 .
Johnson, R. (2010). Kant’s moral philosophy . In N. Z, Edward (Ed.) The Stanford encyclopedia
of philosophy, Summer 2010, Stanford: Stanford University. http://plato.stanford.edu/
archives/sum2010/entries/kant-moral/ .
Joy, B. (2000). Why the future doesn’t need us. Wired, April. http://www.wired.com/wired/
archive/8.04/joy.html .
Joyce, R. (2001). The evolution of morality. Cambridge Studies in Philosophy . New York:
Cambridge University Press. doi: 10.2277/0521808065 .
Kaci, S. (2011). Working with preferences: Less is more. Cognitive technologies . Berlin:
Springer. doi: 10.1007/978-3-642-17280-9 .
Kagan, S. (1997). Normative ethics. Dimensions of philosophy . Boulder, CO: Westview Press.
Keeney, R. L., & Raiffa, H. (1993). Decisions with multiple objectives: Preferences and value
tradeoffs . New York: Cambridge University Press. doi: 10.2277/0521438837 .
Kringelbach, M. L., & Berridge, K. C. (Eds.). (2009). Pleasures of the brain. Series in affective
science . New York: Oxford University Press.
Laird, J. D. (2007). Feelings: The perception of self. Series in affective science . New York:
Oxford University Press. doi: 10.1093/acprof:oso/9780195098891.001.0001 .
Laurence, S., & Margolis, E. (2003). Concepts and conceptual analysis. Philosophy and
Phenomenological Research, 67 (2), 253–282. doi: 10.1111/j.1933-1592.2003.tb00290.x .
Legg, S. (2008). Machine super intelligence. PhD dissertation, Lugano: University of Lugano.
http://www.vetta.org/documents/Machine\_Super\_Intelligence.pdf .
Legg, S. (2009). On universal intelligence. Vetta Project (blog). May 8. http://www.vetta.org/
2009/05/on-universal-intelligence/ (Accessed Mar. 26, 2012).6 The Singularity and Machine Ethics 121
Legg, S, and Marcus H. (2007). A collection of deﬁnitions of intelligence . In G, Ben., & W, Pei
(Eds.) Advances in artiﬁcial general intelligence: Concepts, architectures and algorithms—
proceedings of the AGI workshop 2006, Frontiers in artiﬁcial intelligence and applications.Vol. 157. Amsterdam: IOS Press.
Lewis, D. (1989). Dispositional theories of value . Proceedings of the Aristotelian Society,
Supplementary. Vols. 63:113–137. http://www.jstor.org/stable/4106918 .
Lim, S.-L., O’Doherty, J. P., & Rangel, A. (2011). The decision value computations in the
vmPFC and striatum use a relative value code that is guided by visual attention. Journal of
Neuroscience, 31 (37), 13214–13223. doi: 10.1523/JNEUROSCI.1246-11.2011 .
Mackie, J. L. (1977). Ethics: Inventing right and wrong . New York: Penguin.
Mahoney, M. (2010). A model for recursively self improving programs v.3. Unpublished
manuscript, Dec. 17. http://mattmahoney.net/rsi.pdf (Accessed March 27, 2012).
McFadden, D. L. (2005). Revealed stochastic preference: A synthesis. Economic Theory, 26 (2),
245–264. doi: 10.1007/s00199-004-0495-3 .
McLaren, B. M. (2006). Computational models of ethical reasoning: Challenges, initial steps, and
future directions. IEEE Intelligent Systems, 21 (4), 29–37. doi: 10.1109/MIS.2006.67 .
Minsky, M. (1984). Afterword to Vernor Vinge’s novel, ‘‘true names ’’. Unpublished manuscript,
Oct. 1. http://web.media.mit.edu/ \*minsky/papers/TrueNames.Afterword.html (Accessed
March 26, 2012).
Moore, G. E. (1903). Principia ethica . Cambridge: Cambridge University Press.
Moor, J. H. (2006). The nature, importance, and difﬁculty of machine ethics. IEEE Intelligent
Systems, 21 (4), 18–21. doi: 10.1109/MIS.2006.80 .
Moskowitz, G. B., Li, P., & Kirk, E. R. (2004). The implicit volition model: On the preconscious
regulation of temporarily adopted goals. Advances in Experimental Social Psychology, 36 ,
317–413. doi: 10.1016/S0065-2601(04)36006-5 .
Muehlhauser, L. (2011). The singularity FAQ. Singularity institute for artiﬁcial intelligence.
http://singinst.org/singularityfaq (Accessed March 27, 2012).
Muehlhauser, L. (2012). The human’s hidden utility function (maybe) . Lesswrong. Jan. 28. http://
lesswrong.com/lw/9jh/the\_humans\_hidden\_utility\_function\_maybe/ (Accessed Mar. 27,
2012).
Muehlhauser, L., & Salamon, A. (2012). Intelligence explosion: Evidence and import. In A.
Eden, J. Søraker, J. H. Moor, & E. Steinhart (Eds.), The singularity hypothesis: A scientiﬁc
and philosophical assessment . Berlin: Springer.
Neisser, U. (1979). The concept of intelligence. Intelligence, 3 (3), 217–227. doi: 10.1016/0160-
2896(79)90018-7 .
Nielsen, T. D., & Jensen, F. V. (2004). Learning a decision maker’s utility function from
(possibly) inconsistent behavior. Artiﬁcial Intelligence, 160 (1–2), 53–78. doi: 10.1016/
j.artint.2004.08.003 .
Niu, W., & Brass,J. (2011). Intelligence in worldwide perspective. In Sternberg and Kaufman
2011, 623–645.
Nozick, R. (1974). Anarchy, state, and utopia . New York: Basic Books.
Omohundro, S. M. (2008). The basic AI drives . In: Artiﬁcial general intelligence 2008:
Proceedings of the ﬁrst AGI conference, W, Pei., G, Ben., & F, Stan (Eds.) 483–492. Vol.
171. Frontiers in Artiﬁcial Intelligence and Applications. Amsterdam: IOS Press.
Padoa-Schioppa, Camillo. (2011). Neurobiology of economic choice: A good-based model.
Annual Review of Neuroscience, 34 , 333–359. doi: 10.1146/annurev-neuro-061010-113648 .
Parﬁt, Derek. (1986). Reasons and persons . New York: Oxford University Press. doi: 10.1093/
019824908X.001.0001 .
Parﬁt, D. (2011). On what matters . The Berkeley Tanner Lectures Vol 2. New York: Oxford
University Press.
Pettit, P. (2003). Akrasia, collective and individual . In S, Sarah., & T, Christine (Eds.) Weakness
of will and practical irrationality, New York: Oxford University Press. doi: 10.1093/
0199257361.003.0004 .122 L. Muehlhauser and L. Helm
Pettit, P., & Smith, M. (2000). Global consequentialism. In Brad Hooker, E. Mason, & D.
E. Miller (Eds.), Morality, rules, and consequences: A critical reader (pp. 121–133).
Edinburgh: Edinburgh University Press.
Posner, R. A. (2004). Catastrophe: Risk and response . New York: Oxford University Press.
Powers, T. M. (2006). Prospects for a Kantian machine. IEEE Intelligent Systems, 21 (4), 46–51.
doi:10.1109/MIS.2006.77 .
Pratchett, T. (1996). Feet of clay: A novel of Discworld. Discworld Series . New York:
HarperTorch.
Railton, P. (1986). Facts and values. Philosophical Topics, 14 (2), 5–31.
Railton, P. (2003). Facts, values, and norms: Essays toward a morality of consequence.
Cambridge Studies in Philosophy . New York: Cambridge University Press. doi: 10.1017/
CBO9780511613982 .
Rangel, A., Camerer, C., & Read Montague, P. (2008). A framework for studying the
neurobiology of value-based decision making. Nature Reviews Neuroscience, 9 (7), 545–556.
doi:10.1038/nrn2357 .
Rangel, Antonio, & Hare, Todd. (2010). Neural computations associated with goal-directed
choice. Current Opinion in Neurobiology, 20 (2), 262–270. doi: 10.1016/j.conb.2010.03.001 .
Reynolds, C., & Cassinelli,A (eds.) (2009). AP-CAP 2009: The Fifth Asia-Paciﬁc Computing and
Philosophy Conference , October 1st-2nd, University of Tokyo, Japan, Proceedings. AP-CAP
2009. http://ia-cap.org/ap-cap09/proceedings.pdf .
Ring, M., & Orseau,L. (2011). Delusion, survival, and intelligent agents. In Schmidhuber,
Thórisson, and Looks 2011, 11–20.
Russell, S. J., & Norvig, P. (2009). Artiﬁcial intelligence: A modern approach (3rd ed.). Upper
Saddle River: Prentice-Hall.
Ruzgis, P., & Grigorenko, E. L. (1994). Cultural meaning systems, intelligence and personality .
In J. S, Robert., & R, Patricia, (Eds.) Personality and intelligence, 248–270. New York:
Cambridge University Press. doi: 10.2277/0521417902 .
Rzepka, R., & Araki, K. (2005). What statistics could do for ethics? The idea of common sense
processing based safety valve . In: Anderson, Anderson, and Armen.
Sandberg, A., & Bostrom, N. (2008). Whole brain emulation: A roadmap . Technical Report,
2008-3. Future of humanity institute, Oxford: University of Oxford. www.fhi.ox.ac.uk/
reports/2008-3.pdf .
Schmidhuber, J. (2007). Gödel machines: Fully self-referential optimal universal self-improvers .
In G, Ben., & P, Cassio (Eds.) Artiﬁcial general intelligence, 199–226. Cognitive
technologies. Berlin: Springer. doi: 10.1007/978-3-540-68677-4\_7 .
J, Schmidhuber., R. T, Kristinn., & L. Moshe (Eds.) (2011). Artiﬁcial General Intelligence: 4th
International Conference, AGI 2011, Mountain View, CA, USA, August 3 –6, 2011.
Proceedings. Vol. 6830. Lecture Notes in Computer Science. Berlin: Springer. doi: 10.1007/
978-3-642-22887-2 .
Schnall, S., Haidt, J., Clore, G. L., & Jordan, A. H. (2008). Disgust as embodied moral judgment.
Personality and Social Psychology Bulletin, 34 (8), 1096–1109. doi: 10.1177/01461672083
17771 .
Schroeder, T. (2004). Three faces of desire. Philosophy of MInd Series . New York: Oxford
University Press. doi: 10.1093/acprof:oso/9780195172379.001.0001 .
Searle, J. R. (1980). Minds, brains, and programs. Behavioral and Brain Sciences, 3 (03),
417–424. doi: 10.1017/S0140525X00005756 .
Sen, A. (1979). Utilitarianism and welfarism. Journal of Philosophy, 76 (9), 463–489.
doi:10.2307/2025934 .
Shafer-Landau, R. (2003). Moral realism: A defence . New York: Oxford University Press.
Shope, R. K. (1983). The analysis of knowing: A decade of research . Princeton: Princeton
University Press.
Shulman, C., Jonsson,H., & Tarleton,N. (2009a). Machine ethics and superintelligence .I n
Reynolds and Cassinelli 2009, 95–97.6 The Singularity and Machine Ethics 123
Shulman, C., Nick T., & Henrik J. (2009b). Which consequentialism? Machine ethics and moral
divergence . In: Reynolds and Cassinelli, 23–25.
Simon, D. A., & Daw, N. D. (2011). Neural correlates of forward planning in a spatial decision
task in humans. Journal of Neuroscience, 31 (14), 5526–5539. doi: 10.1523/
JNEUROSCI.4647-10.2011 .
Single, E. (1995). Deﬁning harm reduction. Drug and Alcohol Review, 14 (3), 287–290.
doi:10.1080/09595239500185371 .
Slovic, P., Melissa, L. F., Ellen, P., & Donald, G. M. (2002). The affect heuristic . In G, Thomas.,
G, Dale., & K, Daniel (Eds.) Heuristics and biases: The psychology of intuitive judgment,
397–420. New York: Cambridge University Press. doi: 10.2277/0521796792 .
Smart, R. N. (1958). Negative utilitarianism. Mind, n.s. 67 (268): 542–543. http://www.jstor.org/
stable/2251207 .
Smith, K. S., Mahler, S. V., Pecina, S., & Berridge, K. C. (2009). Hedonic hotspots: Generating
sensory pleasure in the Brain. In M. L. Kringelbach & K. C. Berridge (Eds.), Pleasures of the
brain (pp. 27–49). Oxford: Oxford University Press.
Smith, M. (2009). Desires, values, reasons, and the dualism of practical reason. Ratio, 22 (1),
98–125. doi: 10.1111/j.1467-9329.2008.00420.x .
Sobel, D. 1994. Full information accounts of well-being . Ethics 104 (4): 784–810. http://
www.jstor.org/stable/2382218 .
Sobel, David. (1999). Do the desires of rational agents converge? Analysis, 59 (263), 137–147.
doi:10.1111/1467-8284.00160 .
Stahl, B. C. (2002). Can a computer adhere to the categorical imperative? A contemplation of the
limits of transcendental ethics in IT . In S, Iva., & E. L, George., (Eds.) Cognitive, emotive and
ethical aspects of decision making & human action, 13–18. Vol. 1. Windsor, ON:International Institute for Advanced Studies in Systems Research/Cybernetics.
Sternberg, R. J. (1985). Implicit theories of intelligence, creativity, and wisdom. Journal of
Personality and Social Psychology, 49 (3), 607–627. doi: 10.1037/0022-3514.49.3.607 .
Sternberg, R. J., Conway, B. E., Ketron, J. L., & Bernstein, M. (1981). People’s conceptions of
intelligence. Journal of Personality and Social Psychology, 41 (1), 37–55. doi: 10.1037/0022-
3514.41.1.37 .
Sternberg, R. J., & Grigorenko, E. L. (2006). Cultural intelligence and successful intelligence.
Group & Organization Management, 31 (1), 27–39. doi: 10.1177/1059601105275255 .
Sternberg, R. J., & Kaufman, S. B. (Eds.). (2011). The Cambridge handbook of intelligence.
Cambridge Handbooks in Psychology . New York: Cambridge University Press.
Sutton, R. S., & Andrew, G. Barto. (1998). Reinforcement learning: An introduction. Adaptive
computation and machine learning . Cambridge, MA: MIT Press.
Sverdlik, S. (1985). Counterexamples in ethics. Metaphilosophy, 16 (2–3), 130–145. doi: 10.1111/
j.1467-9973.1985.tb00159.x .
Tännsjö, T. (1998). Hedonistic utilitarianism . Edinburgh: Edinburgh University Press.
Tanyi, A. (2006). An essay on the desire-based reasons model . PhD dissertation. Central
European University. http://web.ceu.hu/polsci/dissertations/Attila\_Tanyi.pdf .
Tegmark, M. (2007). The multiverse hierarchy. In B. Carr (Ed.), Universe or multiverse?
(pp.
99–126). New York: Cambridge University Press.
Thorndike, E. L. (1911). Animal intelligence: Experimental studies . New York: The Macmillan
Company.
Tonkens, R. (2009). A challenge for machine ethics. Minds and Machines, 19 (3), 421–438.
doi:10.1007/s11023-009-9159-1 .
Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice.
Science, 211 (4481), 453–458. doi: 10.1126/science.7455683 .
Vogelstein, E. (2010). Moral reasons and moral sentiments . PhD dissertation. University of
Texas. doi:2152/ETD-UT-2010-05-1243.
Wallach, W., & Allen, C. (2009). Moral machines: Teaching robots right from wrong . New
York: Oxford University Press. doi: 10.1093/acprof:oso/9780195374049.001.0001 .124 L. Muehlhauser and L. Helm
Wallach, W., Colin A., & Iva, S. (2007). Machine morality: Bottom-up and top-down approaches
for modelling human moral faculties . In Ethics and artiﬁcial agents. Special issue, AI &
Society 22(4): 565–582. doi: 10.1007/s00146-007-0099-0 .
Weatherson, B. (2003). What good are counter examples? Philosophical Studies, 115 (1), 1–31.
doi:10.1023/A:1024961917413 .
Wilson, T. D. (2002). Strangers to ourselves: Discovering the adaptive unconscious . Cambridge:
Belknap Press.
Yudkowsky, E. (2001). Creating friendly AI 1.0: The analysis and design of benevolent goal
architectures . Singularity Institute for Artiﬁcial Intelligence, San Francisco, CA, June 15.
http://singinst.org/upload/CFAI.html .
Yudkowsky, E. (2004). Coherent extrapolated volition . Singularity Institute for Artiﬁcial
Intelligence, San Francisco, CA, May. http://singinst.org/upload/CEV.html .
Yudkowsky, E. (2008). Artiﬁcial intelligence as a positive and negative factor in global risk. In
N. Bostrom & M. C. Milan (Eds.), Global catastrophic risks (pp. 308–345). New York:
Oxford University Press.
Yudkowsky, E. (2011). Complex value systems in friendly AI . In T, Schmidhuber., & M, Looks
(Eds.), 388–393, Berlin: Springer.
Zhong, C.-B., Strejcek, B., & Sivanathan, N. (2010). A clean self can render harsh moral
judgment. Journal of Experimental Social Psychology, 46 (5), 859–862. doi: 10.1016/
j.jesp.2010.04.003 .
Zimmerman, D. (2003). Why Richard Brandt does not need cognitive psychotherapy, and other
glad news about idealized preference theories in meta-ethics. Journal of Value Inquiry, 37 (3),
373–394. doi: 10.1023/B:INQU.0000013348.62494.55 .6 The Singularity and Machine Ethics 125
Chapter 6A
Jordi Vallverdú on Muehlhauser and Helm’s ‘‘the Singularityand Machine Ethics’’
The future will bring us, under the very plausible horizon of the singularity, a reality
of ‘‘machine superoptimizers’’, as they are conveniently labeled by Muehlhauser andHelm, trying to avoid typical anthropomorphic biases. Until here I can agree with
him, but there is not a single concept, line or idea related to ethics and, by extension,
to machine ethics, that I can accept. Although the chapter includes a bird’s eye viewon the debates about the roots of ethical discourses it disregards the lack of evidencefor the existence of universal foundations of ethics, even from a metaethical natu-ralist position, which could lead us to a highly controversial deterministic ethics.Generative artiﬁcial ethics, if it could be possible, as an ethical expert systemworking on a revisited version of the Principia Ethica (as an hypothetical neo-
Russell and Whitehead project based on the fundaments of ethics, not the Moore’s
text), would easily fail because every human community has its own ethical and
moral codes or rules and there is no possible agreement among them because all theseuniversals are based in different primordial concepts, all of them beyond any rationalapproach. Ethics is built on the shifting sands of prejudices and opinions, never on thetruth, because it is the result of a cultural decision, not a part of the deep reality. In thissense, you can achieve a de minimis set of shared rules , to describe a statistical
approach to a bunch of common values or design a new artiﬁcial new ethics, but younever can discover nor ﬁnd the ‘true values’ of the real ethics.
The authors also do not take into account a second and very important question
for a Roboethics or an AI ethics: the importance of embodiment. Every livingentity has a bodily structure that creates a speciﬁc intentionality towards the worldand the rest of living entities (similar or not). The body is the place from which theworld is felt and determines or constraints narrowly the range of possible per-formed or wished actions. So, any conscious creature, natural or artiﬁcial, ismodeled and acts according to its structure. If the superoptimizers have differentbodies and minds than us, they surely do will have a speciﬁc and own ethics. You
can decide to try to feel different things about the events of the world but you
cannot decide or not to feel ; and your feelings, emotions and attitudes towards the
basic aspects of the reality are constrained by these bodily arrows. If you agreewith me that emotions are a basic and determinant part of any rational being, thenthe result of putting together minds, emotions and bodies show us an unexpectedand over paradigmatic ethics for those superoptimizers. Hence, in the case ofmachine ethics as in many others, the future does not emerge from the present, atleast not from a sequential and consistent point of view.6 The Singularity and Machine Ethics 127