Chapter 3
Responses to the Journey
to the Singularity
Kaj Sotala and Roman Yampolskiy
3.1 Introduction
The notion of catastrophic AGI risk is not new, and this concern was expressed by
early thinkers in the ﬁeld. Hence, there have also been many proposals concerning
what to do about it. The proposals we survey are neither exhaustive nor mutuallyexclusive: the best way of achieving a desirable outcome may involve pursuingseveral proposals simultaneously.
Section 3.2brieﬂy discusses some of the most recent developments in the ﬁeld.
Sections 3.3–3.5survey three categories of proposals for dealing with AGI risk:
societal proposals, proposals for external constraints on AGI behaviors, and
proposals for creating AGIs that are safe due to their internal design. Although themain purpose of this paper is to provide a summary of existing work, we brie ﬂy
provide commentary on the proposals in each major subsection of Sects. 3.3–3.5and
highlight some of the proposals we consider the most promising in Sect. 3.6. which
are, regulation (Sect. 3.3.3 ), merging with machines (Sect. 3.3.4 ), AGI con ﬁnement
(Sect. 3.4.1 ), Oracle AI (Sect. 3.5.1 ), and motivational weaknesses (Sect. 3.5.7 ).
In the long term, the most promising approaches seem to be value learning
(Sect. 3.5.2.5 ) and human-like architectures (Sect. 3.5.3.4 ). Section 3.6provides an
extended discussion of the various merits and problems of these proposals.
Foundational Research Institute (this work was written while the author was at the Machine
Intelligence Research Institute)
K. Sotala
Foundational Research Institute, Basel, Switzerland
e-mail: kaj.sotala@foundational-research.org
R. Yampolskiy ( ✉)
University of Louisville, 222 Eastern Parkway, Louisville, KY 40292, USA
e-mail: roman.yampolskiy@louisville.edu
© Springer-Verlag GmbH Germany 2017
V. Callaghan et al. (eds.), The Technological Singularity ,
The Frontiers Collection, DOI 10.1007/978-3-662-54033-6\_325
3.2 Post-Superintelligence Responses
This chapter is based on an earlier paper (Sotala and Yampolskiy 2015 ), which was
the formally published version of a previous technical report (Sotala and Yam-
polskiy 2013 ). The tech report, in turn, was a greatly expanded version of an earlier
conference paper (Yampolskiy 2013 ). Since the writing of the original papers, the
topic of catastrophic AGI risk has attracted considerable attention both in academiaand the popular press, much of it due to the publication of the book Superintelli-
gence (Bostrom 2014 ).
We feel that it would not be appropriate to simply lump in all the new responses
together with the old sections, as the debate has now become considerably moreactive and high-pro ﬁle. In particular, numerous AI researchers have signed an open
letter calling for more research into making sure that AI systems will be robust and
beneﬁcial rather than just capable (Future of Life Institute 2015 ). The open letter
included a list of suggested research directions (Russell et al. 2015 ), including ones
speciﬁcally aimed at dealing with the risks from AGI. The research directions
document draws on a number of sources, including an ambitious research agendarecently published by the Machine Intelligence Research Institute (see Chap. 5).
Soon after the publication of the open letter, Elon Musk donated 10 million dollarsfor the purpose of furthering research into safe and bene ﬁcial AI and AGI.
At the same time, several prominent researchers have also expressed the feeling
that the risks from AGI are overhyped, and that there is a danger of the generalpublic taking them too seriously at this stage. This position has been expressed ininterviews of researchers such as Professor Andrew Ng (Madrigal 2015 ) and
Facebook AI director Yann LeCun (Gomes 2015 ), who emphasize that current-day
technology is still a long way from AGI. Even the more skeptical researchers tendto agree that the issue will eventually require some consideration, however(Alexander 2015 ).
3.3 Societal Proposals
Proposals can be divided into three general categories proposals for societal action,
design proposals for external constraints on AGI behavior, and design recom-mendations for internal constraints on AGI behavior. In this section we brie ﬂy
survey societal proposals. These include doing nothing, integrating AGIs with
society, regulating research, merging with machines, and relinquishing researchinto AGI.26 K. Sotala and R. Yampolskiy
3.3.1 Do Nothing
3.3.1.1 AI Is Too Distant to Be Worth Our Attention
One response is that, although AGI is possible in principle, there is no reason to
expect it in the near future. Typically, this response arises from the belief that,although there have been great strides in narrow AI, researchers are still very farfrom understanding how to build AGI. Distinguished computer scientists such asGordon Bell and Gordon Moore, as well as cognitive scientists such as DouglasHofstadter and Steven Pinker, have expressed the opinion that the advent of AGI isremote (IEEE Spectrum 2008 ). Davis ( 2012 ) reviews some of the ways in which
computers are still far from human capabilities. Bringsjord and Bringsjord ( 2012 )
even claim that a belief in AGI this century is ﬁdeistic, appropriate within the realm
of religion but not within science or engineering.
Some writers also actively criticize any discussion of AGI risk in the ﬁrst place.
The philosopher Alfred Nordmann ( 2007 ,2008 ) holds the view that ethical concern
is a scarce resource, not to be wasted on unlikely future scenarios such as AGI.Likewise, Dennett ( 2012 ) considers AGI risk an “imprudent pastime ”because it
distracts our attention from more immediate threats.
Others think that AGI is far off and not yet a major concern, but admit that it
might be valuable to give the issue some attention. A presidential panel of the
Association for the Advancement of Arti ﬁcial Intelligence considering the
long-term future of AI concluded that there was overall skepticism about AGI risk,but that additional research into the topic and related subjects would be valuable(Horvitz and Selman 2009 ). Posner ( 2004 ) writes that dedicated efforts for
addressing the problem can wait, but that we should gather more information aboutthe problem in the meanwhile.
Potential negative consequences of AGI are enormous, ranging from economic
instability to human extinction. “Do nothing ”could be a reasonable course of action
if near-term AGI seemed extremely unlikely, if it seemed too early for any pro-posals to be effective in reducing risk, or if those proposals seemed too expensive toimplement.
As a comparison, asteroid impact prevention is generally considered a topic
worth studying, even though the probability of a civilization-threatening asteroidimpact in the near future is not considered high. Napier ( 2008 ) discusses several
ways of estimating the frequency of such impacts. Many models produce a rate of
one civilization-threatening impact per ﬁve hundred thousand or more years, though
some models suggest that rates of one such impact per hundred thousand yearscannot be excluded.
An estimate of one impact per hundred thousand years would suggest less than a
0.1% chance of a civilization-threatening impact within the next hundred years. Theprobability of AGI being developed within the same period seems considerablyhigher (Müller and Bostrom 2014 ), and there is likewise a reasonable chance of a
hard takeoff after it has been developed (Yudkowsky 2008 ,2008b ), suggesting that3 Responses to the Journey to the Singularity 27
the topic is at the very least worth studying. Even without a hard takeoff, society is
becoming increasingly automated, and even narrow AI is starting to require ethicalguidelines (Wallach and Allen 2009 ).
We know neither which ﬁelds of science will be needed nor how much progress
in them will be necessary for safe AGI. If much progress is needed and we believe
effective progress to be possible this early on, it becomes reasonable to startstudying the topic even before AGI is near. Muehlhauser and Helm ( 2012 ) suggest
that, for one safe AGI approach alone (value learning, discussed further inSect. 3.5.2.5 ), efforts by AGI researchers, economists, mathematicians, and
philosophers may be needed. Safe AI may require the solutions for some of theseproblems to come well before AGI is developed.
3.3.1.2 Little Risk, no Action Needed
Some authors accept that a form of AGI will probably be developed but do not
consider autonomous AGI to be a risk, or consider the possible negative conse-quences acceptable. Bryson and Kime ( 1998 ) argue that, although AGI will require
us to consider ethical and social dangers, the dangers will be no worse than those ofother technologies. Whitby ( 1996 ) writes that there has historically been no con-
sistent trend of the most intelligent people acquiring the most authority, and that
computers will augment humans rather than replace them. Whitby and Oliver(2000 ) further argue that AGIs will not have any particular motivation to act against
us. Jenkins ( 2003 ) agrees with these points to the extent of saying that a machine
will only act against humans if it is programmed to value itself over humans,although she does ﬁnd AGI to be a real concern.
Another kind of “no action needed ”response argues that AGI development will
take a long time (Brooks 2008 ), implying that there will be plenty of time to deal
with the issue later on. This can also be taken as an argument for later efforts being
more effective, as they will be better tuned to AGI as it develops.
Others argue that superintelligence will not be possible at all.
1McDermott
(2012 ) points out that there are no good examples of algorithms which could be
improved upon indeﬁnitely . Deutsch ( 2011 ) argues that there will never be super-
intelligent AGIs, because human minds are already universal reasoners, and com-puters can at best speed up the experimental work that is required for testing andﬁne-tuning theories. He also suggests that even as the speed of technological
development increases, so will our ability to deal with change. Anderson ( 2010 )
likewise suggests that the inherent unpredictability of the world will place upperlimits on an entity ’s effective intelligence.
1The opposite argument is that superior intelligence will inevitably lead to more moral behavior.
Some of the arguments related to this position are discussed in the context of evolutionaryinvariants (Sect. 3.5.3.1 ), although the authors advocating the use of evolutionary invariants do
believe AGI risk to be worth our concern.28 K. Sotala and R. Yampolskiy
Heylighen ( 2012 ) argues that a single, stand-alone computer is exceedingly
unlikely to become superintelligent, and that individual intelligences are alwaysoutmatched by the distributed intelligence found in social systems of many minds.Superintelligence will be achieved by building systems that integrate and improve
the“Global Brain, ”the collective intelligence of everyone on Earth. Heylighen
does acknowledge that this kind of a transition will pose its own challenges, but notof the kind usually evoked in discussions of AGI risk.
The idea of AGIs not having a motivation to act against humans is intuitively
appealing, but there seem to be strong theoretical arguments against it. As men-tioned earlier, Omohundro ( 2007 ,2008 ) and Bostrom ( 2012 ) argue that
self-replication and the acquisition of resources are useful in the pursuit of manydifferent kinds of goals, and that many types of AI systems will therefore exhibit
tendencies toward behaviors such as breaking into other machines, self-replicating,
and acquiring resources without regard for anyone else ’s safety. The right design
might make it possible to partially work around these behaviors (Shulman 2010a ;
Wang 2012 ), but they still need to be taken into account. Furthermore, we might not
foresee all the complex interactions of different AGI mechanisms in the systemsthat we build, and they may end up with very different goals than the ones weintended (Yudkowsky 2008 ,2011 ).
Can AGIs become superintelligent? First, we note that AGIs do not necessarily
need to be much more intelligent than humans in order to be dangerous. AGIs
already enjoy advantages such as the ability to rapidly expand their population byhaving themselves copied (Hanson 1994 ,2008 ; Sotala 2012 a), which may confer
on them considerable economic and political in ﬂuence even if they were not
superintelligent. A better-than-human ability to coordinate their actions, whichAGIs of a similar design could plausibly have (Sotala 2012 ), might then be enough
to tilt the odds in their favor.
Another consideration is that AGIs do not necessarily need to be qualitatively
more intelligent than humans in order to outperform humans. An AGI that merely
thought twice as fast as any single human could still defeat him at intellectual tasksthat had a time constraint, all else equal. Here an “intellectual ”task should be
interpreted broadly to refer not only to “book smarts ”but to any task that animals
cannot perform due to their mental limitations —including tasks involving social
skills (Yudkowsky 2008 ). Straightforward improvements in computing power
could provide AGIs with a considerable advantage in speed, which the AGI couldthen use to study and accumulate experiences that improved its skills.
As for Heylighen ’s(2012 ) Global Brain argument, there does not seem to be a
reason to presume that powerful AGIs could not be geographically distributed, orthat they couldn ’t seize control of much of the Internet. Even if individual minds
were not very smart and needed a society to make progress, for minds that arecapable of copying themselves and communicating perfectly with each other,individual instances of the mind might be better understood as parts of a whole thanas separate individuals. In general, the distinction between an individual and acommunity might not be meaningful for AGIs. If there were enough AGIs, they
might be able to form a community suf ﬁcient to take control of the rest of the Earth.3 Responses to the Journey to the Singularity 29
Heylighen ( 2007 ) himself has argued that many of the features of the Internet are
virtually identical to the mechanisms used by the human brain. If the AGI is notcarefully controlled, it might end up in a position where it made up the majority ofthe“Global Brain ”and could undertake actions which the remaining parts of the
organism did not agree with.
3.3.1.3 Let Them Kill Us
Dietrich ( 2007 ) argues that humanity frequently harms other species, and that
people have also evolved to hurt other people by engaging in behaviors such aschild abuse, sexism, rape, and racism. Therefore, human extinction would not
matter, as long as the machines implemented only the positive aspects of humanity.
De Garis ( 2005 ) suggests that AGIs destroying humanity might not matter. He
writes that on a cosmic scale, with hundreds of billions of stars in our galaxy alone,the survival of the inhabitants of a single planet is irrelevant. As AGIs would bemore intelligent than us in every way, it would be better if they replaced humanity.
AGIs being more intelligent and therefore more valuable than humans equates
intelligence with value, but Bostrom ( 2004 ) suggests ways by which a civilization
of highly intelligent entities might lack things which we thought to have value. For
example, such entities might not be conscious in the ﬁrst place. Alternatively, there
are many things which we consider valuable for their own sake, such as humor,love, game-playing, art, sex, dancing, social conversation, philosophy, literature,scienti ﬁc discovery, food and drink, friendship, parenting, and sport. We value
these due to the fact that we have dispositions and preferences which have beenevolutionarily adaptive in the past, but for a future civilization few or none of themmight be, creating a world with very little of value. Bostrom ( 2012 ) proposes an
orthogonality thesis, by which an arti ﬁcial intelligence can have any combination of
intelligence level and goal, including goals that humans would intuitively deem to
be of no value.
3.3.1.4 “Do Nothing ”Proposals —Our View
As discussed above, completely ignoring the possibility of AGI risk at this stage
would seem to require a con ﬁdent belief in at least one of the following
propositions
1. AGI is very remote.
2. There is no major risk from AGI even if it is created.
3. Very little effective work can be done at this stage.
4. AGIs destroying humanity would not matter.30 K. Sotala and R. Yampolskiy
In the beginning of this paper, we mentioned several experts who considered it
plausible that AGI might be created in the next twenty to one hundred years; in thissection we have covered experts who disagree.
In general, there is a great deal of disagreement among people who have made
AGI predictions, and no clear consensus even among experts in the ﬁeld of arti ﬁcial
intelligence. The lack of expert agreement suggests that expertise in the ﬁeld does
not contribute to an ability to make reliable predictions.
2If the judgment of experts
is not reliable, then, probably, neither is anyone else ’s. This suggests that it is
unjusti ﬁed to be highly certain of AGI being near, but also of it notbeing near. We
thus consider it unreasonable to have a con ﬁdent belief in the ﬁrst proposition.
The second proposition also seems questionable. As discussed in the previous
chapter, AGIs seem very likely to obtain great power, possibly very quickly.
Furthermore, as also discussed in the previous chapter, the complexity and fragility
of value theses imply that it could be very dif ﬁcult to create AGIs which would not
cause immense amounts of damage if they had enough power.
It also does not seem like it is too early to work on the problem as we summarize
in Sect. 3.6, there seem to be a number of promising research directions which can
already be pursued. We also agree with Yudkowsky ( 2008 ), who points out that
research on the philosophical and technical requirements of safe AGI might showthat broad classes of possible AGI architectures are fundamentally unsafe, sug-
gesting that such architectures should be avoided. If this is the case, it seems better
to have that knowledge as early as possible, before there has been a great deal ofinvestment into unsafe AGI designs.
In response to the suggestion that humanity being destroyed would not matter,
we certainly agree that there is much to be improved in today ’s humanity, and that
our future descendants might have very little resemblance to ourselves. Regardless,we think that much about today ’s humans is valuable and worth preserving, and
that we should be able to preserve it without involving the death of present humans.
3.3.2 Integrate with Society
Integration proposals hold that AGI might be created in the next several decades,and that there are indeed risks involved. These proposals argue that the best way todeal with the problem is to make sure that our societal structures are equipped to
handle AGIs once they are created.
2Armstrong and Sotala ( 2012 ) point out that many of the task properties which have been found to
be conducive for developing reliable and useful expertise are missing in AGI timeline forecasting.
In particular, one of the most important factors is whether experts get rapid (preferably immediate)feedback, while a timeline prediction that is set many decades in the future might have beenentirely forgotten by the time that its correctness could be evaluated.3 Responses to the Journey to the Singularity 31
There has been some initial work toward integrating AGIs with existing legal
and social frameworks, such as considering questions of their legal position andmoral rights (Gunkel 2012 ).
3.3.2.1 Legal and Economic Controls
Hanson ( 2012 ) writes that the values of older and younger generations have often
been in con ﬂict with each other, and he compares this to a con ﬂict between humans
and AGIs. He believes that the best way to control AGI risk is to create a legalframework such that it is in the interest of both humans and AGIs to uphold it.Hanson ( 2009 ) suggests that if the best way for AGIs to get what they want is via
mutually agreeable exchanges, then humans would need to care less about what the
AGIs wanted. According to him, we should be primarily concerned with ensuringthat the AGIs will be law-abiding enough to respect our property rights. Miller(2012 ) summarizes Hanson ’s argument, and the idea that humanity could be con-
tent with a small fraction of the world ’s overall wealth and let the AGIs have the
rest. An analogy to this idea is that humans do not kill people who become oldenough to no longer contribute to production, even though younger people could inprinciple join together and take the wealth of the older people. Instead, old people
are allowed to keep their wealth even while in retirement. If things went well, AGIs
might similarly allow humanity to “retire ”and keep its accumulated wealth, even if
humans were no longer otherwise useful for AGIs.
Hall ( 2007a ) also says that we should ensure that the interactions between
ourselves and machines are economic, “based on universal rules of property and
reciprocity. ”Moravec ( 1999 ) likewise writes that governmental controls should be
used to ensure that humans bene ﬁt from AGIs. Without government intervention,
humans would be squeezed out of existence by more ef ﬁcient robots, but taxation
could be used to support human populations for a long time. He also recommends
laws which would require any AGIs to incorporate programming that made themsafe and subservient to human desires. Sandberg ( 2001 ) writes that relying only on
legal and economic controls would be problematic, but that a strategy which alsoincorporated them in addition to other approaches would be more robust than astrategy which did not.
However, even if AGIs were integrated with human institutions, it does not
guarantee that human values would survive. If humans were reduced to a position
of negligible power, AGIs might not have any reason to keep us around.
Economic arguments, such as the principle of comparative advantage, are
sometimes invoked to argue that AGI would ﬁnd it more bene ﬁcial to trade with us
than to do us harm. However, technological progress can drive the wages ofworkers below the level needed for survival, and there is already a possible threat oftechnological unemployment (Brynjolfsson and McAfee 2011 ). AGIs keeping
humans around due to gains from trade implicitly presumes that they would nothave the will or the opportunity to simply eliminate humans in order to replace them
with a better trading partner, and then trade with the new partner instead.32 K. Sotala and R. Yampolskiy
Humans already eliminate species with low economic value in order to make
room for more humans, such as when clearing a forest in order to build new homes.Clark ( 2007 ) uses the example of horses in Britain their population peaked in 1901,
with 3.25 million horses doing work such as plowing ﬁelds, hauling wagons and
carriages short distances, and carrying armies into battle. The internal combustion
engine replaced so many of them that by 1924 there were fewer than two million.Clark writes
There was always a wage at which all these horses could have remained employed. But that
wage was so low that it did not pay for their feed, and it certainly did not pay enough tobreed fresh generations of horses to replace them. Horses were thus an early casualty ofindustrialization (Clark 2007 ).
There are also ways to harm humans while still respecting their property rights,
such as by manipulating them into making bad decisions, or selling them addictivesubstances. If AGIs were suf ﬁciently smarter than humans, humans could be tricked
into making a series of trades that respected their property rights but left them with
negligible assets and caused considerable damage to their well-being.
A related issue is that AGIs might become more capable of changing our values
than we are capable of changing AGI values. Mass media already convey valuesthat have a negative impact on human well-being, such as idealization of rare bodytypes, which causes dissatisfaction among people who do not have those kinds ofbodies (Groesz et al. 2001 ; Agliata and Tantleff-Dunn 2004 ). AGIs with a deep
understanding of human psychology could engineer the spread of values whichshifted more power to them, regardless of their effect on human well-being.
Yet another problem is ensuring that the AGIs have indeed adopted the right
values. Making intelligent beings adopt speci ﬁc values is a dif ﬁcult process which
often fails. There could be an AGI with the wrong goals that would pretend tobehave correctly in society throughout the whole socialization process. AGIs couldconceivably preserve and conceal their goals far better than humans could.
Society does not know of any methods which would reliably instill our chosen
values in human minds, despite a long history of trying to develop them. Our
attempts to make AGIs adopt human values would be hampered by our lack of
experience and understanding of the AGI ’s thought processes, with even
tried-and-true methods for instilling positive values in humans possibly beingineffective. The limited success that we do have with humans is often backed up byvarious incentives as well as threats of punishment, both of which might fail in thecase of an AGI developing to become vastly more powerful than us.
Additionally, the values which a being is likely to adopt, or is even capable of
adopting, will depend on its mental architecture. We will demonstrate these claimswith examples from humans, who are not blank slates on whom arbitrary values can
be imposed with the right education. Although the challenge of instilling speci ﬁc
values in humans is very different from the challenge of instilling them in AGIs, ourexamples are meant to demonstrate the fact that the existing properties of a mindwill affect the process of acquiring values. Just as it is dif ﬁcult to make humans3 Responses to the Journey to the Singularity 33
permanently adopt some kinds of values, the kind of mental architecture that an
AGI has will affect its inclination to adopt various values.
Psychopathy is a risk factor for violence, and psychopathic criminals are much
more likely to reoffend than nonpsychopaths (Hare et al. 2000 ). Harris and Rice
(2006 ) argue that therapy for psychopaths is ineffective and may even make them
more dangerous, as they use their improved social skills to manipulate others moreeffectively. Furthermore, “cult brainwashing ”is generally ineffective and most cult
members will eventually leave (Anthony and Robbins 2004 ); and large-scale social
engineering efforts often face widespread resistance, even in dictatorships with fewscruples about which methods to use (Scott 1998 , Chap. 6 –7). Thus, while one can
try to make humans adopt values, this will only work to the extent that the indi-viduals in question are actually disposed toward adopting them.
3.3.2.2 Foster Positive Values
Kurzweil ( 2005 ), considering the possible effects of many future technologies,
notes that AGI may be a catastrophic risk. He generally supports regulation andpartial relinquishment of dangerous technologies, as well as research into theirdefensive applications. However, he believes that with AGI this may be insuf ﬁcient
and that, at the present time, it may be infeasible to develop strategies that would
guarantee safe AGI. He argues that machine intelligences will be tightly integratedinto our society and that, for the time being, the best chance of avoiding AGI risk isto foster positive values in our society. This will increase the likelihood that anyAGIs that are created will re ﬂect such positive values.
One possible way of achieving such a goal is moral enhancement (Douglas
2008 ), the use of technology to instill people with better motives. Persson and
Savulescu ( 2008 ,2012 ) argue that, as technology improves, we become more
capable of damaging humanity, and that we need to carry out moral enhancement in
order to lessen our destructive impulses.
3.3.2.3 “Integrate with Society ”Proposals —Our View
Proposals to incorporate AGIs into society suffer from the issue that some AGIs
may never adopt benevolent and cooperative values, no matter what the environ-
ment. Neither does the intelligence of the AGIs necessarily affect their values
(Bostrom 2012 ). Suf ﬁciently intelligent AGIs could certainly come to eventually
understand human values, but humans can also come to understand others ’values
while continuing to disagree with them.
Thus, in order for these kinds of proposals to work, they need to incorporate
strong enforcement mechanisms to keep non –safe AGIs in line and to prevent them
from acquiring signi ﬁcant power. This requires an ability to create
value-conforming AGIs in the ﬁrst place, to implement the enforcement. Even a
soft takeoff would eventually lead to AGIs wielding great power, so the34 K. Sotala and R. Yampolskiy
enforcement could not be left to just humans or narrow AIs.3In practice, this means
that integration proposals must be combined with some proposal for internal con-straints which is capable of reliably creating value-conforming AGIs. Integrationproposals also require there to be a soft takeoff in order to work, as having a small
group of AGIs which rapidly acquired enough power to take control of the world
would prevent any gradual integration schemes from working.
Therefore, because any effective integration strategy would require creating safe
AGIs, and the right safe AGI design could lead to a positive outcome even if therewere a hard takeoff, we believe that it is currently better to focus on proposals whichare aimed at furthering the creation of safe AGIs.
3.3.3 Regulate Research
Integrating AGIs into society may require explicit regulation. Calls for regulationare often agnostic about long-term outcomes but nonetheless recommend caution asa reasonable approach. For example, Hibbard ( 2005b ) calls for international reg-
ulation to ensure that AGIs will value the long-term well-being of humans, but doesnot go into much detail. Daley ( 2011 ) calls for a government panel for AGI issues.
Hughes ( 2001 ) argues that AGI should be regulated using the same mechanisms as
previous technologies, creating state agencies responsible for the task and fosteringglobal cooperation in the regulation effort.
Current mainstream academic opinion does not consider AGI a serious threat
(Horvitz and Selman 2009 ), so AGI regulation seems unlikely in the near future. On
the other hand, many AI systems are becoming increasingly autonomous, and anumber of authors are arguing that even narrow-AI applications should be equippedwith an understanding of ethics (Wallach and Allen 2009 ). Currently there are calls
to regulate AI in the form of high-frequency trading (Sobolewski 2012 ), and AI
applications that have a major impact on society might become increasingly reg-ulated. At the same time, legislation has a well-known tendency to lag behindtechnology, and regulating AI applications will probably not translate into regu-lating basic research into AGI.
3.3.3.1 Review Boards
Yampolskiy and Fox ( 2012 ) note that university research programs in the social and
medical sciences are overseen by institutional review boards. They propose settingup analogous review boards to evaluate potential AGI research. Research that wasfound to be AGI related would be restricted with measures ranging from
3For proposals which suggest that humans could use technology to remain competitive with AGIs
and thus prevent them from acquiring excessive amounts of power, see Sect. 3.4.3 Responses to the Journey to the Singularity 35
supervision and funding limits to partial or complete bans. At the same time,
research focusing on safety measures would be encouraged.
Posner ( 2004 , p. 221) suggests the enactment of a law which would require
scienti ﬁc research projects in dangerous areas to be reviewed by a federal catas-
trophic risks assessment board, and forbidden if the board found that the project
would create an undue risk to human survival.
Wilson ( 2013 ) makes possibly the most detailed AGI regulation proposal so far,
recommending a new international treaty where a body of experts would determinewhether there was a “reasonable level of concern ”about AGI or some other pos-
sibly dangerous research. States would be required to regulate research or eventemporarily prohibit it once experts agreed upon there being such a level of con-cern. He also suggests a number of other safeguards built into the treaty, such as the
creation of ethical oversight organizations for researchers, mechanisms for moni-
toring abuses of dangerous technologies, and an oversight mechanism for scienti ﬁc
publications.
3.3.3.2 Encourage Research into Safe AGI
In contrast, McGinnis ( 2010 ) argues that the government should not attempt to
regulate AGI development. Rather, it should concentrate on providing funding for
research projects intended to create safe AGI.
Goertzel and Pitt ( 2012 ) argue for an open-source approach to safe AGI
development instead of regulation. Hibbard ( 2008 ) has likewise suggested devel-
oping AGI via open-source methods, but not as an alternative to regulation.
Legg ( 2009 ) proposes funding safe AGI research via an organization that takes a
venture capitalist approach to funding research teams, backing promising groupsand cutting funding to any teams that fail to make signi ﬁcant progress. The focus of
the funding would be to make AGI as safe as possible.
3.3.3.3 Differential Technological Progress
Both review boards and government funding could be used to implement “differ-
ential intellectual progress ”
Differential intellectual progress consists in prioritizing risk-reducing intellectual progress
over risk-increasing intellectual progress. As applied to AI risks in particular, a plan of
differential intellectual progress would recommend that our progress on the scienti ﬁc,
philosophical, and technological problems of AI safety outpace our progress on theproblems of AI capability such that we develop safe superhuman AIs before we develop(arbitrary) superhuman AIs (Muehlhauser and Salamon 2012 ).
Examples of research questions that could constitute philosophical or scienti ﬁc
progress in safety can be found in later sections of this paper —for instance, the36 K. Sotala and R. Yampolskiy
usefulness of different internal constraints on ensuring safe behavior, or ways of
making AGIs reliably adopt human values as they learn what those values are like.
Bostrom ( 2002 ) used the term “differential technological progress ”to refer to
differential intellectual progress in technological development. Bostrom de ﬁned
differential technological progress as “trying to retard the implementation of dan-
gerous technologies and accelerate implementation of bene ﬁcial technologies,
especially those that ameliorate the hazards posed by other technologies ”.
One issue with differential technological progress is that we do not know what
kind of progress should be accelerated and what should be retarded. For example, amore advanced communication infrastructure could make AGIs more dangerous, asthere would be more networked machines that could be accessed via the Internet.Alternatively, it could be that the world will already be so networked that AGIs will
be a major threat anyway, and further advances will make the networks more
resilient to attack. Similarly, it can be argued that AGI development is dangerousfor as long as we have yet to solve the philosophical problems related to safe AGIdesign and do not know which AGI architectures are safe to pursue (Yudkowsky2008 ). But it can also be argued that we should invest in AGI development now,
when the related tools and hardware are still primitive enough that progress will beslow and gradual (Goertzel and Pitt 2012 ).
3.3.3.4 International Mass Surveillance
For AGI regulation to work, it needs to be enacted on a global scale. This requires
solving both the problem of effectively enforcing regulation within a country andthe problem of getting many different nations to all agree on the need for regulation.
Shulman ( 2009 ) discusses various factors in ﬂuencing the dif ﬁculty of AGI arms
control. He notes that AGI technology itself might make international cooperation
more feasible. If narrow AIs and early-stage AGIs were used to analyze the
information obtained from wide-scale mass surveillance and wiretapping, this mightmake it easier to ensure that nobody was developing more advanced AGI designs.
Shulman ( 2010b ) similarly notes that machine intelligences could be used to
enforce treaties between nations. They could also act as trustworthy inspectorswhich would be restricted to communicating only information about treaty viola-tions, thus not endangering state secrets even if they were allowed unlimited accessto them. This could help establish a “singleton ”regulatory regimen capable of
effectively enforcing international regulation, including AGI-related treaties.
Goertzel and Pitt ( 2012 ) also discuss the possibility of having a network of AGIs
monitoring the world in order to police other AGIs and to prevent any of them fromsuddenly obtaining excessive power.
Another proposal for international mass surveillance is to build an “AGI Nanny ”
(Goertzel 2012b ; Goertzel and Pitt 2012 ), a proposal discussed in Sect. 3.5.4 .
Large-scale surveillance efforts are ethically problematic and face major political
resistance, and it seems unlikely that current political opinion would support the
creation of a far-reaching surveillance network for the sake of AGI risk alone. The3 Responses to the Journey to the Singularity 37
extent to which such extremes would be necessary depends on exactly how easy it
would be to develop AGI in secret. Although several authors make the point thatAGI is much easier to develop unnoticed than something like nuclear weapons(McGinnis 2010 ; Miller 2012 ), cutting-edge high-tech research does tend to require
major investments which might plausibly be detected even by less elaborate
surveillance efforts.
To the extent that surveillance does turn out to be necessary, there is already a
strong trend toward a “surveillance society ”with increasing amounts of information
about people being collected and recorded in various databases (Wood and Ball2006 ). As a reaction to the increased surveillance, Mann et al. ( 2003 ) propose to
counter it with sousveillance —giving private individuals the ability to document
their life and subject the authorities to surveillance in order to protect civil liberties.
This is similar to the proposals of Brin ( 1998 ), who argues that technological
progress might eventually lead to a “transparent society, ”where we will need to
redesign our societal institutions in a way that allows us to maintain some of ourprivacy despite omnipresent surveillance. Miller ( 2012 ) notes that intelligence
agencies are already making major investments in AI-assisted analysis of surveil-lance data.
If social and technological developments independently create an environment
where large-scale surveillance or sousveillance is commonplace, it might be pos-
sible to take advantage of those developments in order to police AGI risk.
4Walker
(2008 ) argues that in order for mass surveillance to become effective, it must be
designed in such a way that it will not excessively violate people ’s privacy, for
otherwise the system will face widespread sabotage. Even under such conditions,there is no clear way to de ﬁne what counts as dangerous AGI. Goertzel and Pitt
(2012 ) point out that there is no clear division between narrow AI and AGI, and
attempts to establish such criteria have failed. They argue that since AGI has anebulous de ﬁnition, obvious wide-ranging economic bene ﬁts, and potentially rich
penetration into multiple industry sectors, it is unlikely to be regulated due to
speculative long-term risks.
AGI regulation requires global cooperation, as the noncooperation of even a
single nation might lead to catastrophe. Historically, achieving global cooperationon tasks such as nuclear disarmament and climate change has been very dif ﬁcult. As
with nuclear weapons, AGI could give an immense economic and militaryadvantage to the country that develops it ﬁrst, in which case limiting AGI research
might even give other countries an incentive to develop AGI faster (Miller 2012 ).
To be effective, regulation also needs to enjoy support among those being
regulated. If developers working in AGI-related ﬁelds only follow the letter of the
law, while privately viewing all regulations as annoying hindrances, and fears aboutAGI as overblown, the regulations may prove ineffective. Thus, it might not be
4An added bene ﬁt would be that this could also help avoid other kinds of existential risk, such as
the intentional creation of dangerous new diseases.38 K. Sotala and R. Yampolskiy
enough to convince governments of the need for regulation; the much larger group
of people working in the appropriate ﬁelds may also need to be convinced.
While Shulman ( 2009 ) argues that the unprecedentedly destabilizing effect of
AGI could be a cause for world leaders to cooperate more than usual, the opposite
argument can be made as well. Gubrud ( 1997 ) argues that increased automation
could make countries more self-reliant, and international cooperation considerablymore dif ﬁcult. AGI technology is also much harder to detect than, for example,
nuclear technology is —nuclear weapons require a substantial infrastructure to
develop, while AGI needs much less (McGinnis 2010 ; Miller 2012 ).
Miller ( 2012 ) even suggests that the mere possibility of a rival being close to
developing AGI might, if taken seriously, trigger a nuclear war. The nation that waslosing the AGI race might think that being the ﬁrst to develop AGI was suf ﬁciently
valuable that it was worth launching a ﬁrst strike for, even if it would lose most of
its own population in the retaliatory attack. He further argues that, although it wouldbe in the interest of every nation to try to avoid such an outcome, the ease ofsecretly pursuing an AGI development program undetected, in violation of treaty,could cause most nations to violate the treaty.
Miller also points out that the potential for an AGI arms race exists not only
between nations, but between corporations as well. He notes that the more AGIdevelopers there are, the more likely it is that they will all take more risks, with each
AGI developer reasoning that if they don ’t take this risk, somebody else might take
that risk ﬁrst.
Goertzel and Pitt ( 2012 ) suggest that for regulation to be enacted, there might
need to be an “AGI Sputnik ”—a technological achievement that makes the pos-
sibility of AGI evident to the public and policy makers. They note that after such amoment, it might not take very long for full human-level AGI to be developed,while the negotiations required to enact new kinds of arms control treaties wouldtake considerably longer.
So far, the discussion has assumed that regulation would be carried out effec-
tively and in the pursuit of humanity ’s common interests, but actual legislation is
strongly affected by lobbying and the desires of interest groups (Olson 1982 ;
Mueller 2003 , Chap. 22). Many established interest groups would have an eco-
nomic interest in either furthering or retarding AGI development, rendering thesuccess of regulation uncertain.
3.3.3.5 “Regulate Research ”Proposals —Our View
Although there seem to be great dif ﬁculties involved with regulation, there also
remains the fact that many technologies have been successfully subjected tointernational regulation. Even if one were skeptical about the chances of effectiveregulation, an AGI arms race seems to be one of the worst possible scenarios, onewhich should be avoided if at all possible. We are therefore generally supportive ofregulation, though the most effective regulatory approach remains unclear.3 Responses to the Journey to the Singularity 39
3.3.4 Enhance Human Capabilities
While regulation approaches attempt to limit the kinds of AGIs that will be created,
enhancement approaches attempt to give humanity and AGIs a level playing ﬁeld.
In principle, gains in AGI capability would not be a problem if humans could
improve themselves to the same level.
Alternatively, human capabilities could be improved in order to obtain a more
general capability to deal with dif ﬁcult problems. Verdoux ( 2010 ,2011 ) suggests
that cognitive enhancement could help in transforming previously incomprehensi-ble mysteries into tractable problems, and Verdoux ( 2010 ) particularly highlights
the possibility of cognitive enhancement helping to deal with the problems posedby existential risks. One problem with such approaches is that increasing human-
ity’s capability for solving problems will also make it easier to develop dangerous
technologies. It is possible that cognitive enhancement should be combined withmoral enhancement, in order to help foster the kind of cooperation that would helpavoid the risks of technology (Persson and Savulescu 2008 ,2012 ).
Moravec ( 1988 ,1999 ) proposes that humans could keep up with AGIs via “mind
uploading, ”a process of transferring the information in human brains to computer
systems so that human minds could run on a computer substrate. This technologymay arrive during a similar timeframe as AGI (Kurzweil 2005 ; Sandberg and
Bostrom 2008 ; Hayworth 2012 ; Koene 2012b ; Cattell and Parker 2012 ; Sandberg
2012 ). However, Moravec argues that mind uploading would come after AGIs, and
that unless the uploaded minds ( “uploads ”) would transform themselves to become
radically nonhuman, they would be weaker and less competitive than AGIs thatwere native to a digital environment Moravec ( 1992 ,1999 ). For these reasons,
Warwick ( 1998 ) also expresses doubt about the usefulness of mind uploading.
Kurzweil ( 2005 ) posits an evolution that will start with brain-computer inter-
faces, then proceed to using brain-embedded nanobots to enhance our intelligence,
andﬁnally lead to full uploading and radical intelligence enhancement. Koene
(2012a ) criticizes plans to create safe AGIs and considers uploading both a more
feasible and a more reliable approach.
Similar proposals have also been made without explicitly mentioning mind
uploading. Cade ( 1966 ) speculates on the option of gradually merging with
machines by replacing body parts with mechanical components. Turney ( 1991 )
proposes linking AGIs directly to human brains so that the two meld together intoone entity, and Warwick ( 1998 ,2003 ) notes that cyborgization could be used to
enhance humans.
Mind uploading might also be used to make human value systems more
accessible and easy to learn for AGIs, such as by having an AGI extrapolate theupload ’s goals directly from its brain, with the upload providing feedback.40 K. Sotala and R. Yampolskiy
3.3.4.1 Would We Remain Human?
Uploading might destroy parts of humanity that we value (Joy 2000 ; de Garis
2005 ). De Garis ( 2005 ) argues that a computer could have far more processing
power than a human brain, making it pointless to merge computers and humans.
The biological component of the resulting hybrid would be insigni ﬁcant compared
to the electronic component, creating a mind that was negligibly different from a“pure”AGI. Kurzweil ( 2001 ) makes the same argument, saying that although he
supports intelligence enhancement by directly connecting brains and computers,this would only keep pace with AGIs for a couple of additional decades.
The truth of this claim seems to depend on exactly how human brains are
augmented. In principle, it seems possible to create a prosthetic extension of a
human brain that uses the same basic architecture as the original brain and gradually
integrates with it (Sotala and Valpola 2012 ). A human extending their intelligence
using such a method might remain roughly human-like and maintain their originalvalues. However, it could also be possible to connect brains with computer pro-grams that are very unlike human brains, and which would substantially change theway the original brain worked. Even smaller differences could conceivably lead tothe adoption of “cyborg values ”distinct from ordinary human values (Warwick
2003 ).
Bostrom ( 2004 ) speculates that humans might outsource many of their skills to
nonconscious external modules and would cease to experience anything as a result.The value-altering modules would provide substantial advantages to their users, tothe point that they could outcompete uploaded minds who did not adopt themodules.
3.3.4.2 Would Evolutionary Pressures Change Us?
A willingness to integrate value-altering modules is not the only way by which a
population of uploads might come to have very different values from modern-dayhumans. This is not necessarily a bad, or even a very novel, development the valuesof earlier generations have often been different from the values of later generations(Hanson 2012 ), and it might not be a problem if a civilization of uploads enjoyed
very different things than a civilization of humans. Still, as there are possibleoutcomes that we would consider catastrophic, such as the loss of nearly all things
that have intrinsic value for us (Bostrom 2004 ), it is worth reviewing some of the
postulated changes in values.
For comprehensiveness, we will summarize all of the suggested effects that
uploading might have on human values, even if they are not obviously negative.Readers may decide for themselves whether or not they consider any of theseeffects concerning.
Hanson ( 1994 ) argues that employers will want to copy uploads who are good
workers, and that at least some uploads will consent to being copied in such a
manner. He suggests that the resulting evolutionary dynamics would lead to an3 Responses to the Journey to the Singularity 41
accelerated evolution of values. This would cause most of the upload population to
evolve to be indifferent or favorable to the thought of being copied, to be indifferenttoward being deleted as long as another copy of themselves remained, and to berelatively uninterested in having children “the traditional way ”(as opposed to
copying an already-existing mind). Although Hanson ’s analysis uses the example
of a worker-employer relationship, it should be noted that nations or families, oreven single individuals, could also gain a competitive advantage by copyingthemselves, thus contributing to the strength of the evolutionary dynamic.
Similarly, Bostrom ( 2004 ) writes that much of human life ’s meaning depends on
the enjoyment of things ranging from humor and love to literature and parenting.These capabilities were adaptive in our past, but in an upload environment theymight cease to be such and gradually disappear entirely.
Shulman ( 2010b ) likewise considers uploading-related evolutionary dynamics.
He notes that there might be a strong pressure for uploads to make copies ofthemselves in such a way that individual copies would be ready to sacri ﬁce
themselves to aid the rest. This would favor a willingness to copy oneself, and aview of personal identity which did not consider the loss of a single copy to bedeath. Beings taking this point of view could then take advantage of economicbeneﬁts of continually creating and deleting vast numbers of minds depending on
the conditions, favoring the existence of a large number of short-lived copies over a
somewhat less ef ﬁcient world of long-lived minds.
Finally, Sotala and Valpola ( 2012 ) consider the possibility of minds coalescing
via arti ﬁcial connections that linked several brains together in the same fashion as
the two hemispheres of ordinary brains are linked together. If this were to happen,considerable bene ﬁts might accrue to those who were ready to coalesce with other
minds. The ability to copy and share memories between minds might also blurdistinctions between individual minds. In the end, most humans might cease to beindividual, distinct people in any real sense of the word.
It has also been proposed that information security concerns could cause
undesirable dynamics among uploads, with signi ﬁcant advantages accruing to those
who could steal the computational resources of others and use them to create newcopies of themselves. If one could seize control of the hardware that an upload wasrunning on, it could be immediately replaced with a copy of a mind loyal to theattacker. It might even be possible to do this without being detected, if it waspossible to steal enough of an upload ’s personal information to impersonate it.
An attack targeting a critical vulnerability in some commonly used piece of
software might quickly hit a very large number of victims. As discussed in the
previous chapter, both theoretical arguments and actual cases of malware show thatlarge numbers of machines on the Internet could be infected in a very short time(Staniford et al. 2002 , Moore et al. 2002 ,2003 ). In a society of uploads, attacks
such as these would be not only inconvenient, but potentially fatal. Eckersley andSandberg ( 2013 ) offer a preliminary analysis of information security in a world of
uploads.42 K. Sotala and R. Yampolskiy
3.3.4.3 Would Uploading Help?
Even if the potential changes of values were deemed acceptable, it is unclear
whether the technology for uploading could be developed before developing AGI.
Uploading might require emulating the low-level details of a human brain with a
high degree of precision, requiring large amounts of computing power (Sandbergand Bostrom 2008 ; Cattell and Parker 2012 ). In contrast, an AGI might be designed
around high-level principles which have been chosen to be computationally cheapto implement on existing hardware architectures.
Yudkowsky ( 2008 ) uses the analogy that it is much easier to ﬁgure out the
principles of aerodynamic ﬂight and then build a Boeing 747 than it is to take a
living bird and “upgrade ”it into a giant bird that can carry passengers, all while
ensuring that the bird remains alive and healthy throughout the process. Likewise, it
may be much easier to ﬁgure out the basic principles of intelligence and build AGIs
than to upload existing minds.
On the other hand, one can also construct an analogy suggesting that it is easier
to copy a thing ’s function than it is to understand how it works. If a person does not
understand architecture but wants to build a sturdy house, it may be easier to createa replica of an existing house than it is to design an entirely new one that does notcollapse.
Even if uploads were created ﬁrst, they might not be able to harness all the
advantages of digitality, as many of these advantages depend on minds being easyto modify (Sotala 2012 ), which human minds may not be. Uploads will be able to
directly edit their source code as well as introduce simulated pharmaceutical andother interventions, and they could experiment on copies that are restored to anunmodi ﬁed state if the modi ﬁcations turn out to be unworkable (Shulman 2010b ).
Regardless, human brains did not evolve to be easy to modify, and it may bedifﬁcult to ﬁnd a workable set of modi ﬁcations which would drastically improve
them.
In contrast, in order for an AGI programmed using traditional means to be
manageable as a software project, it must be easy for the engineers to modify it.
5
Thus, even if uploading were developed before AGI, AGIs that were developedlater might still be capable of becoming more powerful than uploads. However,existing uploads already enjoying some of the advantages of the newly-createdAGIs would still make it easier for the uploads to control the AGIs, at least for awhile.
Moravec ( 1992 ) notes that the human mind has evolved to function in an
environment which is drastically different from a purely digital environment, andthat the only way to remain competitive with AGIs would be to transform intosomething that was very different from a human. This suggests that uploading
5However, this might not be true for AGIs created using some alternate means, such as arti ﬁcial
life (Sullins 2005 ).3 Responses to the Journey to the Singularity 43
might buy time for other approaches, but would be only a short-term solution in and
of itself.
If uploading technology were developed before AGI, it could be used to upload a
research team or other group and run them at a vastly accelerated rate as compared
to the rest of humanity. This would give them a considerable amount of extra time
for developing any of the other approaches. If this group were among the ﬁrst to be
successfully emulated and sped up, and if the speed-up would allow enough sub-jective time to pass before anyone else could implement their own version, theymight also be able to avoid trading safety for speed. However, such a group mightbe able to wield tremendous power, so they would need to be extremely reliable andtrustworthy.
3.3.4.4 “Enhance Human Capabilities ”Proposals —Our View
Of the various “enhance human capabilities ”approaches, uploading proposals seem
the most promising, as translating a human brain to a computer program wouldsidestep many of the constraints that come from modifying a physical system. Forexample, all relevant brain activity could be recorded for further analysis at anarbitrary level of detail, and any part of the brain could be instantly modi ﬁed
without a need for time-consuming and possibly dangerous invasive surgery.
Uploaded brains could also be more easily upgraded to take full advantage of morepowerful hardware, while humans whose brains were still partially biological wouldbe bottlenecked by the speed of the biological component.
Uploading does have several problems Uploading research might lead to AGI
being created before the uploads, in the long term uploads might have unfavorableevolutionary dynamics, and it seems likely that there will eventually be AGIs whichare capable of outperforming uploads in every ﬁeld of competence. Uploads could
also be untrustworthy even without evolutionary dynamics. At the same time,
however, uploading research doesn ’tnecessarily accelerate AGI research very
much, the evolutionary dynamics might not be as bad as they seem at the moment,and the advantages gained from uploading might be enough to help control unsafeAGIs until safe ones could be developed. Methods could also be developed forincreasing the trustworthiness of uploads (Shulman 2010b ). Uploading might still
turn out to be a useful tool for handling AGI risk.
3.3.5 Relinquish Technology
Not everyone believes that the risks involved in creating AGIs are acceptable.Relinquishment involves the abandonment of technological development that could
lead to AGI. This is possibly the earliest proposed approach, with Butler ( 1863 )44 K. Sotala and R. Yampolskiy
writing that “war to the death should be instantly proclaimed ”upon machines, for
otherwise they would end up destroying humans entirely. In a much-discussedarticle, Joy ( 2000 ) suggests that it might be necessary to relinquish at least some
aspects of AGI research, as well as nanotechnology and genetics research.
Hughes ( 2001 ) criticizes AGI relinquishment, while Kurzweil ( 2005 ) criticizes
broad relinquishment but supports the possibility of “ﬁne-grained relinquishment, ”
banning some dangerous aspects of technologies while allowing general work onthem to proceed. In general, most writers reject proposals for broad relinquishment.
3.3.5.1 Outlaw AGI
Weng et al. ( 2009 ) write that the creation of AGIs would force society to shift from
human-centric values to robot-human dual values. In order to avoid this, theyconsider the possibility of banning AGI. This could be done either permanently oruntil appropriate solutions are developed for mediating such a con ﬂict of values.
McKibben ( 2003 ), writing mainly in the context of genetic engineering, also sug-
gests that AGI research should be stopped.
Hughes ( 2001 ) argues that attempts to outlaw a technology will only make the
technology move to other countries. De Garis ( 2005 ) believes that differences of
opinion about whether to build AGIs will eventually lead to armed con ﬂict, to the
point of open warfare.
Annas et al. ( 2002 ) have similarly argued that genetic engineering of humans
would eventually lead to war between unmodi ﬁed humans and the engineered
“posthumans, ”and that cloning and inheritable modi ﬁcations should therefore be
banned. To the extent that one accepts their reasoning with regard to humans, itcould also be interpreted to apply to AGIs.
3.3.5.2 Restrict Hardware
Berglas ( 2012 ) suggests not only stopping AGI research, but also outlawing the
production of more powerful hardware. Berglas holds that it will be possible tobuild computers as powerful as human brains in the very near future, and that weshould therefore reduce the power of new processors and destroy existing ones.
6
Branwen ( 2012 ) argues that Moore ’s Law depends on the existence of a small
number of expensive and centralized chip factories, making them easy targets for
regulation and incapable of being developed in secret.
6Berglas (personal communication) has since changed his mind and no longer believes that it is
possible to effectively restrict hardware or otherwise prevent AGI from being created.3 Responses to the Journey to the Singularity 45
3.3.5.3 “Relinquish Technology ”Proposals —Our View
Relinquishment proposals suffer from many of the same problems as regulation
proposals, but to a greater extent. There is no historical precedent of general,
multiuse technology similar to AGI being successfully relinquished for good, nor
do there seem to be any theoretical reasons for believing that relinquishment pro-posals would work in the future. Therefore we do not consider them to be a viableclass of proposals.
3.4 External AGI Constraints
Societal approaches involving regulation or research into safe AGI assume thatproper AGI design can produce solutions to AGI risk. One category of suchsolutions is that of external constraints . These are restrictions that are imposed on
an AGI from the outside and aim to limit its ability to do damage.
Several authors have argued that external constraints are unlikely to work with
AGIs that are genuinely far more intelligent than us (Vinge 1993 ; Yudkowsky
2001 ;2008 ; Kurzweil 2005 ; Chalmers 2010 ; Armstrong et al. 2012 ). The consensus
seems to be that external constraints might buy time when dealing with less
advanced AGIs, but they are useless against truly superintelligent ones.
External constraints also limit the usefulness of an AGI, as a free-acting one
could serve its creators more effectively. This reduces the probability of the uni-versal implementation of external constraints on AGIs. AGIs might also be dan-gerous if they were con ﬁned or otherwise restricted. For further discussion of these
points, see Sect. 3.5.1 .
3.4.1 AGI Con ﬁnement
AGI con ﬁnement, or “AI boxing ”(Chalmers 2010 ; Yampolskiy 2012 ; Armstrong
et al. 2012 ), involves con ﬁning an AGI to a speci ﬁc environment and limiting its
access to the external world.
Yampolskiy (2012) makes an attempt to formalize the idea, drawing on previous
computer security research on the so-called con ﬁnement problem (Lampson 1973 ).
Yampolskiy de ﬁnes the AI con ﬁnement problem (AICP) as the challenge of
restricting an AGI to a con ﬁned environment from which it can ’t communicate
without authorization. A number of methods have been proposed for implementingAI con ﬁnement, many of which are extensively discussed in Armstrong et al.
(2012 ).
Chalmers ( 2010 ) and Armstrong et al. ( 2012 ) mention numerous caveats and
difﬁculties with AI-boxing approaches. A truly leakproof system that perfectly
isolated the AGI from an outside environment would prevent us from even46 K. Sotala and R. Yampolskiy
observing the AGI. If AGIs were given knowledge about human behavior or
psychology, they could still launch social engineering attacks on us (Chalmers2010 ; Armstrong et al. 2012 ). An AGI that was unaware of the existence of humans
would be less likely to launch such attacks, but also much more limited in the kinds
of tasks that it could be used for.
Even if the AGI remained con ﬁned, it could achieve enough in ﬂuence among
humans to prevent itself from being reset or otherwise modi ﬁed (Good 1970 ).
An AGI that people grew reliant on might also become impossible to reset ormodify.
3.4.1.1 Safe Questions
Yampolskiy (2012) proposes a hierarchy of security levels for communication with
the AGI, ranging from completely unrestricted communication to no communica-tion whatsoever. He suggests that the right level for communicating with AGIs is anintermediate one, based on asking “safe questions. ”These are de ﬁned as questions
for which a human could ﬁnd an answer without help from a superintelligence. For
instance, a researcher who was considering two possible cures for cancer could askthe AGI which one to pursue ﬁrst. Yampolskiy argues that the AGI is effectively
not changing the ﬁnal outcome, as the researcher would in principle be capable of
ﬁnding the better treatment himself. The AGI, by recommending one of the
approaches as the choice to pursue ﬁrst, is merely speeding up the process. Arm-
strong et al. ( 2012 ) also brie ﬂy discuss such approaches.
3.4.1.2 Virtual Worlds
While the con ﬁnement protocol supposes an AGI that can perceive our world
directly, Chalmers ( 2010 ) uses the term “leakproof singularity ”to refer to the
project of creating isolated virtual worlds which would disallow their inhabitantsfrom in ﬂuencing the external world. An AGI placed in such a world could be
monitored by humans to ensure that it was safe to release into our world. However,Chalmers notes that “for an AI system to be useful or interesting to us at all, it must
have some effects on us. At a minimum, we must be able to observe it. And themoment we observe a virtual environment, some information leaks out from that
environment into our environment and affects us ”.
Hibbard ( 2012b ) proposes a decision support system for evaluating AGI designs
in simulated environments that model our real environment, and attempts to showthat the AGI can be built to be safe.
Armstrong et al. ( 2012 ) discuss the notion that we could supplement this
approach with several layers of “sub-worlds ”virtual worlds stacked within each
other. In such a scenario, even if an AGI somehow manages to break out from onevirtual reality, it is still restricted by another and cannot directly in ﬂuence the world.
Ultimately, the authors conclude that this approach isn ’t very promising.3 Responses to the Journey to the Singularity 47
3.4.1.3 Resetting the AGI
For as long as an AGI remains con ﬁned or cooperative, its memory could be altered
from the outside. Turney ( 1991 ) mentions a proposal in which an AGI ’s memory
could be wiped after each job, thus limiting its potential. Armstrong et al. ( 2012 )
also mention the possibility of resetting an AGI after each run, though they note thatthere are many ways for the AGI to get around it, and that it is only useful forcertain kinds of AGI systems.
Good ( 1970 ) warns that a superhuman AGI may be smart enough to ﬁnd ways to
prevent itself from being successfully reset or shut down, including methods ofsocial manipulation. Warwick ( 1998 ) makes a similar point.
3.4.1.4 Checks and Balances
Armstrong et al. ( 2012 ) suggest a number of checks and balances that could be used
to catch the early stages of an AGI that was behaving dangerously or trying todeceive us. Test questions with known answers could be mixed in with questionswe genuinely didn ’t know the answer to, in order to establish an AGI ’s trustwor-
thiness. “Honey-pots, ”tempting resources with great value to the AGI, could also
be left in physical or mental locations that it was forbidden from accessing. Finally,
several different kinds of AGIs could be presented with the same problem, withtheir answers then being compared. Inconsistent answers would suggest a possibleattempt at deception.
Bostrom ( 2014 , Chap. 9) similarly discusses “tripwires ”, systems which make
diagnostic tests to the AGI and shut it down if they detect signs of dangerousactivity. Some possible tripwires includes ones monitoring the AGI ’s behavior, its
abilities, or possibly even content of the AGI ’s thoughts and planning.
3.4.1.5 “AI Con ﬁnement ”Proposals —Our View
Despite their limited ability to deal with AGIs more intelligent than us, AI-boxing
techniques seem to have value as a ﬁrst line of defense, and it may be worthwhile to
invest in developing off-the-shelf software packages for AI con ﬁnement that are
easy and cheap to use. A research project that developed AGI unexpectedly might
not have been motivated to make major investments in security, but the AGI might
still be suf ﬁciently limited in intelligence that con ﬁnement would work. Having a
defense that is easy to deploy will make it more likely that these kinds of projectswill implement better precautions.
However, at the same time there is a risk that this will promote a false sense of
security and make research teams think that they have carried out their duty to becautious merely because they are running elementary con ﬁnement protocols.
Although some con ﬁnement procedures can be implemented on top of an AGI that
was not expressly designed for con ﬁnement, they are much less reliable than with48 K. Sotala and R. Yampolskiy
an AGI that was built with con ﬁnement considerations in mind (Armstrong et al.
2012 )—and even then, relying solely on con ﬁnement is a risky strategy. We are
therefore somewhat cautious in our recommendation to develop con ﬁnement
techniques.
3.4.2 AGI Enforcement
One problem with AI con ﬁnement proposals is that humans are tasked with
guarding machines that may be far more intelligent than themselves (Good 1970 ).
One proposed solution for this problem is to give the task of watching AGIs to other
AGIs.
Armstrong ( 2007 ) proposes that the trustworthiness of a superintelligent system
could be monitored via a chain of less powerful systems, all the way down tohumans. Although humans couldn ’t verify and understand the workings of a
superintelligence, they could verify and understand an AGI just slightly above theirown level, which could in turn verify and understand an AGI somewhat above itsown level, and so on.
Chaining multiple levels of AI systems with progressively greater capacity
seems to be replacing the problem of building a safe AI with a multisystem, and
possibly more dif ﬁcult, version of the same problem. Armstrong himself admits that
there are several problems with the proposal. There could be numerous issues alongthe line, such as a break in the chain of communication or an inability of a system toaccurately assess the mind of another (smarter) system. There is also the problem ofcreating a trusted bottom for the chain in the ﬁrst place, which is not necessarily any
easier than creating a trustworthy superintelligence.
Hall ( 2007a ) writes that there will be a great variety of AGIs, with those that
were designed to be moral or aligned with human interests keeping the nonsafe
ones in check. Goertzel and Pitt ( 2012 ) also propose that we build a community of
mutually policing AGI systems of roughly equal levels of intelligence. If an AGIstarted to “go off the rails, ”the other AGIs could stop it. This might not prevent a
single AGI from undergoing an intelligence explosion, but a community of AGIsmight be in a better position to detect and stop it than humans would.
Having AGIs police each other is only useful if the group of AGIs actually has
goals and values that are compatible with human goals and values. To this end, the
appropriate internal constraints are needed.
The proposal of a society of mutually policing AGIs would avoid the problem of
trying to control a more intelligent mind. If a global network of mildly superin-telligent AGIs could be instituted in such a manner, it might detect and prevent anynascent takeoff. However, by itself such an approach is not enough to ensure safety3 Responses to the Journey to the Singularity 49
—it helps guard against individual AGIs “going off the rails, ”but it does not help in
a scenario where the programming of most AGIs is ﬂawed and leads to nonsafe
behavior. It thus needs to be combined with the appropriate internal constraints.
A complication is that a hard takeoff is a relative term—an event that happens
too fast for any outside observer to stop. Even if the AGI network were a hundred
times more intelligent than a network composed of only humans, there might stillbe a more sophisticated AGI that could overcome the network.
3.4.2.1 “AGI Enforcement ”Proposals —Our View
AGI enforcement proposals are in many respects similar to social integration
proposals, in that they depend on the AGIs being part of a society which is strong
enough to stop any single AGI from misbehaving. The greatest challenge is then tomake sure that most of the AGIs in the overall system are safe and do not uniteagainst humans rather than against misbehaving AGIs. Also, there might not be anatural distinction between a distributed AGI and a collection of many differentAGIs, and AGI design is in any case likely to make heavy use of earlier AI/AGItechniques. AGI enforcement proposals therefore seem like implementation vari-ants of various internal constraint proposals (Sect. 3.5), rather than independent
proposals.
3.5 Internal Constraints
In addition to external constraints, AGIs could be designed with internal motiva-
tions designed to ensure that they would take actions in a manner bene ﬁcial to
humanity. Alternatively, AGIs could be built with internal constraints that make
them easier to control via external means.
With regard to internal constraints, Yudkowsky ( 2008 ) distinguishes between
technical failure andphilosophical failure :
Technical failure is when you try to build an AI and it doesn ’t work the way you think it
does—you have failed to understand the true workings of your own code. Philosophical
failure is trying to build the wrong thing, so that even if you succeeded you would still failto help anyone or bene ﬁt humanity. Needless to say, the two failures are not mutually
exclusive (Yudkowsky 2008 ).
In practice, it is not always easy to distinguish between the two. Most of the
discussion below focuses on the philosophical problems of various proposals, but
some of the issues, such as whether or not a proposal is actually possible toimplement, are technical.50 K. Sotala and R. Yampolskiy
3.5.1 Oracle AI
AnOracle AI is a hypothetical AGI that executes no actions other than answering
questions. This is a proposal with many similarities to AGI con ﬁnement: both
involve restricting the extent to which the AGI is allowed to take independent
action. We consider the difference to be that an Oracle AI has been programmed to“voluntarily ”restrict its activities, whereas AGI con ﬁnement refers to methods for
restricting an AGI ’s capabilities even if it was actively attempting to take more
extensive action.
Trying to build an AGI that only answered questions might not be as safe as it
sounds, however. Correctly de ﬁning “take no actions ”might prove surprisingly
tricky (Armstrong et al. 2012 ), and the oracle could give ﬂawed advice even if it did
correctly restrict its actions.
Some possible examples of ﬂawed advice: As extra resources are useful for the
fulﬁllment of nearly all goals (Omohundro 2007 ,2008 ), the oracle may seek to
obtain more resources —such as computing power —in order to answer questions
more accurately. Its answers might then be biased toward furthering this goal, evenif this temporarily reduces the accuracy of its answers, if it believes this to increasethe accuracy of its answers in the long run. Another example is that if the oracle hadthe goal of answering as many questions as possible as fast as possible, it could
attempt to manipulate humans into asking it questions that were maximally simple
and easy to answer.
Holden Karnofsky has suggested that an Oracle AI could be safe if it was “just a
calculator, ”a system which only computed things that were asked of it, taking no
goal-directed actions of its own. Such a “Tool-Oracle AI ”would keep humans as
the ultimate decision makers. Furthermore, the ﬁrst team to create a Tool-Oracle AI
could use it to become powerful enough to prevent the creation of other AGIs(Karnofsky and Tallinn 2011 ; Karnofsky 2012 ).
An example of a Tool-Oracle AI approach might be Omohundro ’s(2012 )
proposal of “Safe-AI Scaffolding ”creating highly constrained AGI systems which
act within limited, predetermined parameters. These could be used to developformal veri ﬁcation methods and solve problems related to the design of more
intelligent, but still safe, AGI systems.
Oracle AIs might be considered a special case of domestic AGI (Bostrom 2014 ,
Chap. 9): AGIs which are built to only be interested in taking action ”on a small
scale, within a narrow context, and through a limited set of action modes ”.
3.5.1.1 Oracles Are Likely to Be Released
As with a boxed AGI, there are many factors that would tempt the owners of an
Oracle AI to transform it to an autonomously acting agent. Such an AGI would befar more effective in furthering its goals, but also far more dangerous.3 Responses to the Journey to the Singularity 51
Current narrow-AI technology includes high-frequency trading (HFT) algo-
rithms, which make trading decisions within fractions of a second, far too fast tokeep humans in the loop. HFT seeks to make a very short-term pro ﬁt, but even
traders looking for a longer-term investment bene ﬁt from being faster than their
competitors. Market prices are also very effective at incorporating various sources
of knowledge (Hanson 2000 ). As a consequence, a trading algorithm ’s performance
might be improved both by making it faster and by making it more capable ofintegrating various sources of knowledge. Most advances toward general AGI willlikely be quickly taken advantage of in the ﬁnancial markets, with little opportunity
for a human to vet all the decisions. Oracle AIs are unlikely to remain as pureoracles for long.
Similarly, Wallach and Allen ( 2012 ) discuss the topic of autonomous robotic
weaponry and note that the US military is seeking to eventually transition to a state
where the human operators of robot weapons are “on the loop ”rather than “in the
loop”. In other words, whereas a human was previously required to explicitly give
the order before a robot was allowed to initiate possibly lethal activity, in the futurehumans are meant to merely supervise the robot ’s actions and interfere if something
goes wrong. In practice, this may make it much harder for the human to control therobot ’s actions, if the robot makes a decision which the operator only has a very
short time to override.
Docherty & Goose ( 2012 ) report on a number of military systems which are
becoming increasingly autonomous, with the human oversight for automaticweapons defense systems —designed to detect and shoot down incoming missiles
and rockets —already being limited to accepting or overriding the computer ’s plan
of action in a matter of seconds. Although these systems are better described asautomatic, carrying out pre-programmed sequences of actions in a structuredenvironment, than autonomous, they are a good demonstration of a situation whererapid decisions are needed and the extent of human oversight is limited. A number
of militaries are considering the future use of more autonomous weapons.
In general, any broad domain involving high stakes, adversarial decision mak-
ing, and a need to act rapidly is likely to become increasingly dominated byautonomous systems. The extent to which the systems will need general intelli-gence will depend on the domain, but domains such as corporate management,fraud detection, and warfare could plausibly make use of all the intelligence theycan get. If one ’s opponents in the domain are also using increasingly autonomous
AI/AGI, there will be an arms race where one might have little choice but to give
increasing amounts of control to AI/AGI systems.
Miller ( 2012 ) also points out that if a person was close to death, due to natural
causes, being on the losing side of a war, or any other reason, they might turn evena potentially dangerous AGI system free. This would be a rational course of actionas long as they primarily valued their own survival and thought that even a smallchance of the AGI saving their life was better than a near-certain death.
Some AGI designers might also choose to create less constrained and more
free-acting AGIs for aesthetic or moral reasons, preferring advanced minds to have
more freedom.52 K. Sotala and R. Yampolskiy
3.5.1.2 Oracles Will Become Authorities
Even if humans were technically kept in the loop, they might not have the time,
opportunity, motivation, intelligence, or con ﬁdence to verify the advice given by an
Oracle AI. This may be a danger even with narrower AI systems. Friedman and
Kahn ( 1992 ) discuss APACHE, an expert system that provides medical advice to
doctors. They write that as the medical community puts more and more trust intoAPACHE, it may become common practice to act automatically on APACHE ’s
recommendations, and it may become increasingly dif ﬁcult to challenge the “au-
thority ”of the recommendations. Eventually, APACHE may in effect begin to
dictate clinical decisions.
Likewise, Bostrom and Yudkowsky ( 2013 ) point out that modern bureaucrats
often follow established procedures to the letter, rather than exercising their own
judgment and allowing themselves to be blamed for any mistakes that follow.Dutifully following all the recommendations of an AGI system would be an evenbetter way of avoiding blame.
Wallach and Allen ( 2012 ) note the existence of robots which attempt to auto-
matically detect the locations of hostile snipers and to point them out to soldiers. Tothe extent that these soldiers have come to trust the robots, they could be seen ascarrying out the robots ’orders. Eventually, equipping the robot with its own
weapons would merely dispense with the formality of needing to have a human to
pull the trigger.
Thus, even AGI systems that function purely to provide advice will need to be
explicitly designed to be safe in the sense of not providing advice that would goagainst human values (Wallach and Allen 2009 ). Yudkowsky ( 2012 ) further notes
that an Oracle AI might choose a plan that is beyond human comprehension, inwhich case there ’s still a need to design it as explicitly safe and conforming to
human values.
3.5.1.3 “Oracle AI ”Proposals —Our View
Much like with external constraints, it seems like Oracle AIs could be a useful
stepping stone on the path toward safe, freely acting AGIs. However, because anyOracle AI can be relatively easily turned into a free-acting AGI and because manypeople will have an incentive to do so, Oracle AIs are not by themselves a solution
to AGI risk, even if they are safer than free-acting AGIs when kept as pure oracles.
3.5.2 Top-Down Safe AGI
AGIs built to take autonomous actions will need to be designed with safe moti-
vations. Wallach and Allen divide approaches for ensuring safe behavior into
“top-down ”and“bottom-up ”approaches (Wallach and Allen 2009 ). They de ﬁne3 Responses to the Journey to the Singularity 53
“top-down ”approaches as ones that take a speci ﬁed ethical theory and attempt to
build a system capable of implementing that theory.
They have expressed skepticism about the feasibility of both pure top-down and
bottom-up approaches, arguing for a hybrid approach.7With regard to top-down
approaches, which attempt to derive an internal architecture from a given ethical
theory, Wallach ( 2010 )ﬁnds three problems
1.“Limitations already recognized by moral philosophers For example, in a util-
itarian calculation, how can consequences be calculated when information is
limited and the effects of actions cascade in never-ending interactions? Whichconsequences should be factored into the maximization of utility? Is there astopping procedure? ”(Wallach 2010 ).
2. The “frame problem ”refers to the challenge of discerning relevant from irrel-
evant information without having to consider all of it, as all information couldbe relevant in principle (Pylyshyn 1987 ; Dennett 1987 ). Moral decision-making
involves a number of problems that are related to the frame problem, such as
needing to know what effects different actions have on the world, and needing to
estimate whether one has suf ﬁcient information to accurately predict the con-
sequences of the actions.
3.“The need for background information. What mechanisms will the system
require in order to acquire the information it needs to make its calculations?How does one ensure that this information is up to date in real time? ”(Wallach
2010 ).
To some extent, these problems may be special cases of the fact that we do not
yet have AGI with good general learning capabilities creating an AGI would alsorequire solving the frame problem, for instance. These problems might therefore notall be as challenging as they seem at ﬁrst, presuming that we manage to develop
AGI in the ﬁrst place.
3.5.2.1 Three Laws
Probably the most widely known proposal for machine ethics is Isaac Asimov ’s
(1942 ) Three Laws of Robotics:
1. A robot may not injure a human being or, through inaction, allow a human being
to come to harm.
2. A robot must obey orders given to it by human beings except where such orders
would con ﬂict with the First Law.
3. A robot must protect its own existence as long as such protection does not
conﬂict with either the First or Second Law.
7For a de ﬁnition of “bottom-up ”approaches, see Sect. 5.3.54 K. Sotala and R. Yampolskiy
Asimov and other writers later expanded the list to include a number of addi-
tional laws, including the Zeroth Law.
A robot may not harm humanity, or through inaction allow humanity to come to harm.
Although the Three Laws are widely known and have inspired numerous imi-
tations, several of Asimov ’s own stories were written to illustrate the fact that the
laws contained numerous problems. They have also drawn heavy critique fromothers (Clarke 1993 ;1994 ; Weld and Etzioni 1994 ; Pynadath and Tambe 2002 ;
Gordon-Spears 2003 ; McCauley 2007 ; Weng et al. 2008 ; Wallach and Allen 2009 ;
Murphy and Woods 2009 ; Anderson 2011 ) and are not considered a viable
approach for safe AI. Among their chief shortcomings is the fact that they are tooambiguous to implement and, if de ﬁned with complete accuracy, contradict each
other in many situations.
3.5.2.2 Categorical Imperative
The best-known universal ethical axiom is Kant ’s categorical imperative. Many
authors have discussed using the categorical imperative as the foundation of AGImorality (Stahl 2002 ; Powers 2006 ; Wallach and Allen 2009 ; Beavers 2009 ,2012 ).
All of these authors conclude that Kantian ethics is a problematic goal for AGI,
though Powers ( 2006 ) still remains hopeful about its prospects.
3.5.2.3 Principle of Voluntary Joyous Growth
Goertzel ( 2004a ,b) considers a number of possible axioms before settling on what
he calls the “Principle of Voluntary Joyous Growth ”,d eﬁned as “Maximize hap-
piness, growth and choice ”. He starts by considering the axiom “maximize hap-
piness ”, but then ﬁnds this to be problematic and adds “growth ”, which he de ﬁnes
as“increase in the amount and complexity of patterns in the universe ”. Finally he
adds “choice ”in order to allow sentient beings to “choose their own destiny ”.
3.5.2.4 Utilitarianism
Classic utilitarianism is an ethical theory stating that people should take actions that
lead to the greatest amount of happiness and the smallest amount of suffering. Theprospects for AGIs implementing a utilitarian moral theory have been discussed byseveral authors. The consensus is that pure classical utilitarianism is problematicand does not capture all human values. For example, a purely utilitarian AGI could3 Responses to the Journey to the Singularity 55
reprogram the brains of humans so that they did nothing but experience the max-
imal amount of pleasure all the time, and that prospect seems unsatisfactory tomany.
8
3.5.2.5 Value Learning
Freeman ( 2009 ) describes a decision-making algorithm which observes people ’s
behavior, infers their preferences in the form of a utility function, and then attemptsto carry out actions which ful ﬁll everyone ’s preferences. The standard name for this
kind of an approach is inverse reinforcement learning (Ng and Russell 2000 ).
Russell ( 2015 ) argues for an inverse reinforcement learning approach, as there is
considerable data about human behavior and the attitudes behind it, there are solid
economic incentives to solve this problem, the problem does not seem intrinsicallyharder than learning how the rest of the world works, and because it seems possibleto deﬁne this goal so as to make the AGIs very careful to ensure that they ’re correct
about our preferences before taking any serious action.
Similarly, Dewey ( 2011 ) discusses value learners , AGIs which are provided a
probability distribution over possible utility functions that humans may have. Valuelearners then attempt to ﬁnd the utility functions with the best match for human
preferences. Hibbard ( 2012a ) builds on Dewey ’s work to offer a similar proposal.
One problem with conceptualizing human desires as utility functions is that
human desires change over time (van Gelder 1995 ) and also violate the axioms of
utility theory required to construct a coherent utility function (Tversky and Kah-neman 1981 ). While it is possible to treat inconsistent choices as random deviations
from an underlying “true”utility function that is then learned (Nielsen and Jensen
2004 ), this does not seem to properly describe human preferences. Rather, human
decision making and preferences seem to be driven by multiple competing systems,
only some of which resemble utility functions (Dayan 2011 ). Even if a true utility
function could be constructed, it does not take into account the fact that we havesecond-order preferences, or desires about our desires a drug addict may desire adrug, but also desire that he not desire it (Frankfurt 1971 ). Similarly, we often wish
that we had stronger desires toward behaviors which we consider good but cannotmake ourselves engage in. Taking second-order preferences into account leads towhat philosophers call “ideal preference ”theories of value.
Taking this into account, it has been argued that we should aim to build AGIs
which act according to humanity ’sextrapolated values (Yudkowsky 2004 ; Tarleton
2010 ; Muehlhauser and Helm 2012 ). Yudkowsky proposes attempting to discover
the“Coherent Extrapolated Volition ”(CEV) of humanity, which he de ﬁnes as
8Note that utilitarianism is not the same thing as having a utility function. Utilitarianism is a
speciﬁc kind of ethical system, while utility functions are general-purpose mechanisms for
choosing between actions and can in principle be used to implement very different kinds of ethicalsystems, such as egoism and possibly even rights-based theories and virtue ethics (Peterson 2010 ).56 K. Sotala and R. Yampolskiy
our wish if we knew more, thought faster, were more the people we wished we were, had
grown up farther together; where the extrapolation converges rather than diverges, whereour wishes cohere rather than interfere; extrapolated as we wish that extrapolated, inter-
preted as we wish that interpreted (Yudkowsky 2004 ).
CEV remains vaguely de ﬁned and has been criticized by several authors (Hib-
bard 2005a ; Goertzel 2010a ; Goertzel and Pitt 2012 ; Miller 2012 ). However,
Tarleton ( 2010 )ﬁnds CEV a promising approach, and suggests that CEV has ﬁve
desirable properties, and that many different kinds of algorithms could possess these
features
Meta-algorithm : Most of the AGI ’s goals will be obtained at runtime from
human minds, rather than explicitly programmed in before runtime.
Factually correct beliefs : The AGI will attempt to obtain correct answers to
various factual questions, in order to modify preferences or desires that arebased upon false factual beliefs.
Singleton : Only one superintelligent AGI is to be constructed, and it is to take
control of the world with whatever goal function is decided upon.
Reﬂection : Individual or group preferences are re ﬂected upon and revised.
Preference aggregation : The set of preferences of a whole group are to be
combined somehow.
At least two CEV variants have been proposed Coherent Aggregated Volition
(Goertzel 2010a ) and Coherent Blended Volition (Goertzel and Pitt 2012 ). Goertzel
and Pitt ( 2012 ) describe a methodology which was used to help end the apartheid in
South Africa. The methodology involves people with different views exploringdifferent future scenarios together and in great detail. By exploring different out-comes together, the participants build emotional bonds and mutual understanding,seeking an outcome that everyone can agree to live with. The authors characterize
the Coherent Blended Volition of a diverse group as analogous to the “conceptual
blend ”resulting from the methodology, incorporating the most essential elements
of the group into a harmonious whole.
Christiano ( 2012 ) attempts to sketch out a formalization of a value extrapolation
approach called “indirect normativity ”. It proposes a technique that would allow an
AI to approximate the kinds of values a group of humans would settle on if theycould spend an unbounded amount of time and resources considering the problem.
Other authors have begun preliminary work on simpler value learning systems,
designed to automatically learn moral principles. Anderson et al. ( 2005a ,2005b ,
2006 ) have built systems based around various moral duties and principles. As lists
of duties do not in and of themselves specify what to do when they con ﬂict, the
systems let human experts judge how each con ﬂict should be resolved, and then
attempt to learn general rules from the judgments. As put forth, however, thisapproach would have little ability to infer ethical rules which did not ﬁt the
framework of proposed duties. Improved computational models of ethical reason-ing, perhaps incorporating work from neuroscience and moral psychology, could
help address this. Potapov and Rodionov ( 2012 ) propose an approach by which an3 Responses to the Journey to the Singularity 57
AGI could gradually learn the values of other agents as its understanding of the
world improved.
A value extrapolation process seems dif ﬁcult to specify exactly, as it requires
building an AGI with programming that formally and rigorously de ﬁnes human
values. Even if it manages to avoid the ﬁrst issue in Wallach ’s(2010 ) list
(Sect. 3.5.2 ), top-down value extrapolation may fall victim to other issues, such as
computational tractability. One interpretation of CEV would seem to requiremodeling not only the values of everyone on Earth, but also the evolution of thosevalues as the people in question interacted with each other, became more intelligentand more like their ideal selves, chose which of their values they wanted to pre-serve, etc. Even if the AGI could eventually obtain the enormous amount ofcomputing power required to run this model, its behavior would need to be safe
from the beginning, or it could end up doing vast damage to humanity before
understanding what it was doing wrong.
Goertzel and Pitt ’s(2012 ) hybrid approach, in which AGIs cooperate with
humans in order to discover the values humans wish to see implemented, seemsmore likely to avoid the issue of computational tractability. However, it will fail towork in a hard takeoff situation where AGIs take control before being taught thecorrect human values. Another issue with Coherent Blended Volition is thatschemes which require absolute consensus are unworkable with large groups, as
anyone whose situation would be worsened by a change of events could block the
consensus. A general issue with value extrapolation approaches is that there may beseveral valid ways of de ﬁning a value extrapolation process, with no objective
grounds for choosing one rather than another.
Goertzel ( 2010a ) notes that in formal reasoning systems a set of initially
inconsistent beliefs which the system attempts to resolve might arrive at somethingvery different than the initial belief set, even if there existed a consistent belief setthat was closer to the original set. He suggests that something similar might happen
when attempting to make human values consistent, though whether this would be a
bad thing is unclear.
3.5.2.6 Approval-Directed Agents
Christiano ( 2014a ,b,c,2015 ) proposes “approval-directed agents ”. Instead of
having explicit goals which they would attempt to optimize for, approval-directed
agents would choose their next action using a procedure such as
Estimate the expected rating [an overseer] would give each action if he considered it at
length. Take the action with the highest expected rating. (Christiano 2014a ,b,c)
Christiano ( 2014a ,b,c) suggests that this approach would:
allow us to start with simple overseers and simple approval-directed agents,
rather than having to specify all of an AI ’s goals at once58 K. Sotala and R. Yampolskiy
avoiding getting locked into bad design decisions: “if an approval-directed AI
encounters an unforeseen situation, it will respond in the way that we mostapprove of ”
fail gracefully: ﬂawed interpretations of the overseer ’s goals might be noticed
faster, as the AI wouldn ’t try to actively hide them unless the overseer delib-
erately approved of such actions
3.5.2.7 “Top-Down Safe AGI ”Proposals —Our View
Of the various top-down proposals, value learning and proposals and
approval-directed agents seem to be the only ones which properly take into accountthe complexity of value thesis (see the pervious chapter), as they attempt tospeciﬁcally take into account considerations such as “Would humanity have
endorsed this course of action if it had known the consequences? ”Although there
are many open questions concerning the computational tractability as well as the
general feasibility of the value learning approaches, they together with theapproval-directed ones seem like some of the most important ones to work on.
3.5.3 Bottom-up and Hybrid Safe AGI
Wallach ( 2010 )d eﬁnes bottom-up approaches as ones that favor evolving or
simulating the mechanisms that give rise to our moral decisions. Another alternativeis hybrid approaches, combining parts of both top-down and bottom-up approaches.
A problem with pure bottom-up approaches is that techniques such as arti ﬁcial
evolution or merely rewarding the AGI for the right behavior may cause it tobehave correctly in tests, but would not guarantee that it would behave safely in anyother situation. Even if an AGI seems to have adopted human values, the actual
processes driving its behavior may be very different from the processes that would
be driving the actions of a human who behaved similarly. It might then behave very
unexpectedly in situations which are different enough (Yudkowsky 2008 ,2011 ).
Armstrong et al. ( 2012 ) discuss various problems related to such approaches and
offer examples of concepts which seem straightforward to humans but are not assimple as they may seem on the surface. One of their examples relates to theconcept of time
If the [AGI] had the reasonable-sounding moral premise that “painlessly killing a human
being, who is going to die in a micro-second anyway, in order to gain some other good, isnot a crime, ”we would not want it to be able to rede ﬁne millennia as seconds (Armstrong
et al. 2012 ).
All humans have an intuitive understanding of time and no experience with
beings who could arbitrarily rede ﬁne their own clocks and might not share the same3 Responses to the Journey to the Singularity 59
concept of time. Such differences in the conceptual grounding of an AGI ’s values
and of human values might not become apparent until too late.
3.5.3.1 Evolutionary Invariants
Human morality is to a large extent shaped by evolution, and evolutionary
approaches attempt to replicate this process with AGIs.
Hall ( 2007a ,2011 ) argues that self-improving AGIs are likely to exist in com-
petition with many other kinds of self-improving AGIs. Properties that give AGIs asigniﬁcant disadvantage might then be strongly selected against and disappear. We
could attempt to identify evolutionary invariants , or evolutionarily stable strategies,
which would both survive in a competitive environment and cause an AGI to treat
humans well.
Hall ( 2011 ) lists self-interest, long planning horizons, knowledge, an under-
standing of evolutionary ethics, and guaranteed honesty as invariants that are likelyto make an AGI more moral as well as to persist even under intense competition. Hesuggests that, although self-interest may sound like a bad thing in an AGI,non-self-interested creatures are dif ﬁcult to punish. Thus, enlightened self-interest
might be a good thing for an AGI to possess, as it will provide an outside com-
munity with both a stick and a carrot to control it with.
Similarly, Waser ( 2008 ) suggests that minds which are intelligent enough will,
due to game-theoretical and other considerations, become altruistic and cooperative.Waser ( 2011 ) proposed the principle of Rational Universal Benevolence (RUB), the
idea that the moral course of action is cooperation while letting everyone freelypursue their own goals. Waser proposes that, instead of making human-friendlybehavior an AGI ’s only goal, the AGI would be allowed to have and form its own
goals. However, its goals and actions would be subject to the constraint that they
should respect the principle of RUB and not force others into a life those others
would disagree with.
Kornai ( 2014 ) cites Gewirth ’s (Gewirth 1978 ) work on the principle of generic
consistency, which holds that respecting others ’rights to freedom and well-being is
a logically necessary conclusion for any rational agents. Kornai suggests that if theprinciple is correct, then AGIs would respect humanity ’s rights to freedom and
well-being, and that AGIs which failed to respect the principle would be outcom-peted by ones which did.
Although these approaches expect AGI either to evolve altruism or to ﬁnd it the
most rational approach, true altruism or even pure tit-for-tat (Axelrod 1987 ) isn’t
actually the best strategy in evolutionary terms. Rather, a better strategy isMachiavellian tit-for-tat cultivating an appearance of altruism and cooperation
when it bene ﬁts oneself, and acting sel ﬁshly when one can get away with it.
Humans seem strongly disposed toward such behavior (Haidt 2006 ).
Another problem is that tit-for-tat as a good strategy assumes that both players
are equally powerful and both have the same options at their disposal. If the AGI
became far more powerful than most humans, it might no longer be in its interests60 K. Sotala and R. Yampolskiy
to treat humans favorably (Fox and Shulman 2010 ). This hypothesis can be tested
by looking at human behavior if exploiting the weak is an evolutionarily usefulstrategy, then humans should engage in it when given the opportunity. Humanswho feel powerful do indeed devalue the worth of the less powerful and view them
as objects of manipulation (Kipnis 1972 ). They also tend to ignore social norms
(van Kleef et al. 2011 ) and to experience less distress and compassion toward the
suffering of others (van-Kleef 2008 ).
Thus, even if an AGI cooperated with other similarly powerful AGIs, the group
of AGIs might still decide to collectively exploit humans. Similarly, even thoughthere might be pressure for AGIs to make themselves more transparent and easilyinspected by others, this only persists for as long as the AGI needs others more thanthe others need the AGI.
3.5.3.2 Evolved Morality
Another proposal is to create AGIs via algorithmic evolution, selecting in each
generation the AGIs which are not only the most intelligent, but also the mostmoral. These ideas are discussed to some extent by Wallach and Allen ( 2009 ).
3.5.3.3 Reinforcement Learning
In machine learning, reinforcement learning (not to be confused with inverse
reinforcement learning, discussed above in the value learning section) is a model inwhich an agent takes various actions and is differentially rewarded for the actions,after which it learns to perform the actions with the greatest expected reward. Inpsychology, it refers to agents being rewarded for certain actions and thus learning
behaviors which they have a hard time breaking, even if some other kind of
behavior is more bene ﬁcial for them later on.
Applied to AGI, the machine learning sense of reinforcement involves teaching
an AGI to behave in a safe manner by rewarding it for ethical choices, and letting itlearn for itself the underlying rules of what constitutes ethical behavior. In an earlyexample of this kind of proposal, McCulloch ( 1956 ) described an “ethical machine ”
that could infer the rules of chess by playing the game, and suggested that it couldalso learn ethical behavior this way.
Hibbard ( 2001 ,2005a ) suggested using reinforcement learning to give AGIs
positive emotions toward humans. Early AGIs would be taught to recognize hap-piness and unhappiness in humans, and the results of this learning would behard-wired as emotional values in more advanced AGIs. This training processwould then be continued —for example, by letting the AGIs predict how human
happiness would be affected by various actions and using those predictions asemotional values.
A reinforcement learner is supplied with a reward signal, and it always has the
explicit goal of maximizing the sum of this reward, any way it can. In order for this3 Responses to the Journey to the Singularity 61
goal to align with human values, humans must engineer the environment so that the
reinforcement learner is prevented from receiving rewards if human goals are notfulﬁlled (Dewey 2011 ). A reinforcement-learning AGI only remains safe for as long
as humans are capable of enforcing this limitation, and will become unpredictable if
it becomes capable of overcoming it. Hibbard (Hibbard 2012d ) has retracted his
earlier reinforcement learning –based proposals, as they would allow the AGI to
maximize its reinforcement by modifying humans to be maximally happy, evenagainst their will (Dewey 2011 ).
3.5.3.4 Human-like AGI
Another kind of proposal involves building AGIs that can learn human values by
virtue of being similar to humans.
Connectionist systems, based on arti ﬁcial neural nets, are capable of learning
patterns from data without being told what the patterns are. As some connectionistmodels have learned to classify problems in a manner similar to humans (McLeodet al. 1998 ; Plaut 2003 ; Thomas and McClelland 2008 ), it has been proposed that
connectionist AGI might learn moral principles that are too complex for humans tospecify as explicit rules.
9This idea has been explored by Guarini ( 2006 ) and
Wallach and Allen ( 2009 ).
Sotala ( 2015 ) brie ﬂy surveys research into human concept learning, and argues
that current research suggests that human concepts might be generated using arelatively limited set of mechanisms. He proposes a research program to map thesemechanisms in more detail and to then build AGIs which would learn similarconcepts as humans, including moral concepts.
One speci ﬁc proposal that draws upon connectionism is to make AGIs act
according to virtue ethics (Wallach and Allen 2009 ). These authors note that pre-
vious writers discussing virtuous behavior have emphasized the importance of
learning moral virtues through habit and practice. As it is impossible to exhaus-tively de ﬁne a virtue, virtue ethics has traditionally required each individual to learn
the right behaviors through “bottom-up processes of discovery or learning ”. Models
that mimicked the human learning process well enough could then potentially learnthe same behaviors as humans do.
Another kind of human-inspired proposal is the suggestion that something like
Stan Franklin ’s LIDA architecture (Franklin and Patterson 2006 ; Ramamurthy et al.
2006 ; Snaider et al. 2011 ), or some other approach based on Bernard Baars ’s(2002 ,
2005 )“global workspace ”theory, might enable moral reasoning. In the LIDA
architecture, incoming information is monitored by specialized attention codelets ,
each of which searches the input for speci ﬁc features. In particular, moral codelets
might look for morally relevant factors and ally themselves with other codelets to
9But it should be noted that there are also promising nonconnectionist approaches for modeling
human classi ﬁcation behavior —see, e.g., Tenenbaum et al. ( 2006 ,2011 ).62 K. Sotala and R. Yampolskiy
promote their concerns to the level of conscious attention. Ultimately, some
coalitions will win enough support to accomplish a speci ﬁc kind of decision
(Wallach and Allen 2009 ).
Goertzel and Pitt ( 2012 ) consider human memory systems (episodic, sensori-
motor, declarative, procedural, attentional, and intentional) and ways by which
human morality might be formed via their interaction. They brie ﬂy discuss the way
that the OpenCog AGI system (Hart and Goertzel 2008 ; Goertzel 2012a ) imple-
ments similar memory systems and how those systems could enable it to learnmorality. Similarly, Goertzel and Bugaj ( 2008 ) discuss the stages of moral devel-
opment in humans and suggest ways by which they could be replicated in AGIsystems, using the speci ﬁc example of Novamente, a proprietary version of
OpenCog.
Waser ( 2009 ) also proposes building an AGI by studying the results of evolution
and creating an implementation as close to the human model as possible. Shanahan(2015 , Chap. 5) similarly suggests building a human-like AGI that might then
acquire human values and abilities, such as empathy.
Human-inspired AGI architectures would intuitively seem the most capable of
learning human values, though what would be human-like enough remains an openquestion. It is possible that even a relatively minor variation from the norm couldcause an AGI to adopt values that most humans would consider undesirable.
Getting the details right might require an extensive understanding of the human
brain.
There are also humans who have drastically different ethics than the vast
majority of humanity and argue for the desirability of outcomes such as theextinction of mankind (Benatar 2006 ; Dietrich 2007 ). There remains the possibility
that even AGIs which reasoned about ethics in a completely human-like mannerwould reach such conclusions.
Humans also have a long track record of abusing power, or of undergoing major
behavioral changes due to relatively small injuries —the“safe Homo sapiens ”
problem also remains unsolved. On the other hand, it seems plausible thathuman-like AGIs could be explicitly engineered to avoid such problems.
The easier that an AGI is to modify, the more powerful it might become (Sotala
2012 ), and very close recreations of the human brain may turn out to be dif ﬁcult to
extensively modify and upgrade. Human-inspired safe AGIs might then end upoutcompeted by AGIs which were easier to modify, and which might or might notbe safe.
Even if human-inspired architectures could be easily modi ﬁed, the messiness of
human cognitive architecture means that it might be dif ﬁcult to ensure that their
values remain bene ﬁcial during modi ﬁcation. For instance, in LIDA-like architec-
tures, bene ﬁcial behavior will depend on the correct coalitions of morality codelets
winning each time. If the system undergoes drastic changes, this can be verydifﬁcult if not impossible to ensure.
Most AGI builders will attempt to create a mind that displays considerable
advantages over ordinary humans. Some such advantages might be best achieved
by employing a very nonhuman architecture (Moravec 1992 ), so there will be3 Responses to the Journey to the Singularity 63
reasons to build AGIs that are not as human-like. These could also end up out-
competing the human-like AGIs.
3.5.3.5 “Bottom-up and Hybrid Safe AGI ”Proposals —Our View
We are generally very skeptical about pure bottom-up methods, as they only allow a
very crude degree of control over an AGI ’s goals, giving it a motivational system
which can only be relied on to align with human values in the very speci ﬁc
environments that the AGI has been tested in. Evolutionary invariants seem inca-pable of preserving complexity of value, and they might not even be capable ofpreserving human survival. Reinforcement learning, on the other hand, depends on
the AGI being incapable of modifying the environment against the will of its human
controllers. Therefore, none of these three approaches seems workable.
Human-like AGI might have some promise, depending on exactly how fragile
human values were. If the AGI reasoning process could be made suf ﬁciently
human-like, there is the possibility that the AGI could remain relatively safe,though less safe than a well-executed value extrapolation –based AGI.
3.5.4 AGI Nanny
A more general proposal, which could be achieved by either top-down, bottom-up,
or hybrid methods, is the proposal of an “AGI Nanny ”(Goertzel 2012b ; Goertzel
and Pitt 2012 ). This is an AGI that is somewhat more intelligent than humans and is
designed to monitor Earth for various dangers, including more advanced AGI.
The AGI Nanny would be connected to powerful surveillance systems and
would control a massive contingent of robots. It would help abolish problems such
as disease, involuntary death, and poverty, while preventing the development oftechnologies that could threaten humanity. The AGI Nanny would be designed notto rule humanity on a permanent basis, but to give us some breathing room and timeto design more advanced AGIs. After some predetermined amount of time, it wouldcede control of the world to a more intelligent AGI.
Goertzel and Pitt ( 2012 ) brie ﬂy discuss some of the problems inherent in the
AGI Nanny proposal. The AGI would have to come to power in an ethical way, and
might behave unpredictably despite our best efforts. It might also be easier to create
a dramatically self-improving AGI than to create a more constrained AGI Nanny.
3.5.4.1 “AGI Nanny ”Proposals —Our View
Upon asserting control, the AGI Nanny would need to have precisely speci ﬁed
goals, so that it would stop other AGIs from taking control but would also not harm
human interests. It is not clear to what extent de ﬁning these goals would be easier64 K. Sotala and R. Yampolskiy
than de ﬁning the goals of a more free-acting AGI (Muehlhauser and Salamon
2012 ). Overall, the AGI Nanny seems to have promise, but it ’s unclear whether it
can be made to work.
3.5.5 Motivational Scaffolding
Motivational scaffolding (Bostrom 2014 , Chap. 9) is a proposal in which an AGI is
originally given a temporary, relatively simple goal system that is easy to specifybut not intended as the ﬁnal system. As the AGI ’s capabilities and knowledge
mature, the AGI ’s motivational system is replaced with a more sophisticated one
that then guides it on its path towards full superintelligence.
Bostrom ( 2014 ) considers this approach to have some potential, but also notes
several problems with it. The AGI might resist having its initial goals replaced, theAGI might grow into a superintelligence with the old goals in place, and it isunclear whether installing the ﬁnal goal system in the fully-developed AGI would
be particularly easy.
3.5.6 Formal Veri ﬁcation
Formal veri ﬁcation methods prove speci ﬁc properties about various algorithms. If
the complexity and fragility of value theses hold, it follows that safe AGI requires theability to verify that proposed changes to the AGI will not alter its goals or values. Ifeven a mild drift from an AGI ’s original goals might lead to catastrophic conse-
quences, then utmost care should be given to ensuring that the goals will not change
inadvertently. This is particularly the case if there are no external feedback mech-
anisms which would correct the drift. Before modifying itself, an AGI could attemptto formally prove that the changes would not alter its existing goals, and wouldtherefore keep them intact even during extended self-modi ﬁcation (Yudkowsky
2008 ). Such proofs could be required before self-modi ﬁcation was allowed to occur,
and the system could also be required to prove that this verify-before-modi ﬁcation
property itself would always be preserved during self-modi ﬁcation.
Spears ( 2006 ) combines machine learning and formal veri ﬁcation methods to
ensure that AIs remain within the bounds of prespeci ﬁed constraints after having
learned new behaviors. She attempts to identify “safe”machine learning operators,
which are guaranteed to preserve the system ’s constraints.
One AGI system built entirely around the concept of formal veri ﬁcation is the
Gödel machine (Schmidhuber 2009 ; Steunebrink and Schmidhuber 2011 ). It con-
sists of a solver , which attempts to achieve the goals set for the machine, and a
searcher , which has access to a set of axioms that completely describe the machine.
The searcher may completely rewrite any part of the machine, provided that it can
produce a formal proof showing that such a rewrite will further the system ’s goals.3 Responses to the Journey to the Singularity 65
Goertzel ( 2010b ) proposes GOLEM (Goal-Oriented LEarning
Meta-Architecture), a meta-architecture that can be wrapped around a variety ofdifferent AGI systems. GOLEM will only implement changes that are predicted tobe more effective at achieving the original goal of the system. Goertzel argues that
GOLEM is likely to be both self-improving and steadfast: either it pursues the same
goals it had at the start, or it stops acting altogether.
Unfortunately, formalizing the AGI ’s goals in a manner that will allow formal
veriﬁcation methods to be used is a challenging task. Within cryptography, many
communications protocols have been proven secure, only for successful attacks tobe later developed against their various implementations. While the formal proofswere correct, they contained assumptions which did not accurately capture the waythe protocols worked in practice (Degabriele et al. 2011 ). Proven theorems are only
as good as their assumptions, so formal veri ﬁcation requires good models of the
AGI hardware and software.
3.5.6.1 “Formal Veri ﬁcation ”Proposals —Our View
Compared to the relatively simple domain of cryptographic security, verifying
things such as “Does this kind of a change to the AGI ’s code preserve its goal of
respecting human values? ”seems like a much more open-ended and dif ﬁcult task,
one which might even prove impossible. Regardless, it is the only way of achievinghigh con ﬁdence that a system is safe, so it should at least be attempted.
3.5.7 Motivational Weaknesses
Finally, there is a category of internal constraints that, while not making an AGI ’s
values safer, make it easier to control AGI via external constraints.
3.5.7.1 High Discount Rates
AGI systems could be given a high discount rate, making them value short-term
goals and gains far more than long-term goals and gains (Shulman 2010a ; Arm-
strong et al. 2012 ). This would inhibit the AGI ’s long-term planning, making it
more predictable. However, an AGI can also reach long-term goals through a seriesof short-term goals (Armstrong et al. 2012 ). Another possible problem is that it
could cause the AGI to pursue goals which were harmful for humanity ’s long-term
future. Humanity may arguably be seen as already behaving in ways that imply anexcessively high discount rate, such as by consuming ﬁnite natural resources
without properly taking into account the well-being of future generations.66 K. Sotala and R. Yampolskiy
3.5.7.2 Easily Satiable Goals
Shulman ( 2010a ) proposes designing AGIs in such a way that their goals are easy to
satisfy. For example, an AGI could receive a near-maximum reward for simply
continuing to receive an external reward signal, which could be cut if humans
suspected misbehavior. The AGI would then prefer to cooperate with humans ratherthan trying to attack them, even if it was very sure of its chances of success.
10
Likewise, if the AGI could receive a maximal reward with a relatively smallfraction of humanity ’s available resources, it would have little to gain from seizing
more resources.
Bostrom ( 2014 ) mentions a variation of this idea, where the AGI would receive a
continuous stream of cryptographic reward tokens from a human controller. Its
utility function would be de ﬁned so that it would receive “99% of its maximum
utility from the ﬁrst reward token; 99% of its remaining utility potential from the
second reward token; and so on ”. Bostrom also discusses what might go wrong
with this scheme: possibly the AGI might not trust the human ’s promises to con-
tinue sending the rewards, or it might suspect that the human was in danger of beingincapacitated unless the AGI took action.
An extreme form of this kind of a deal is Orseau and Ring ’s(2011 )“Simpleton
Gambit, ”in which an AGI is promised everything that it would ever want, on the
condition that it turn itself into a harmless simpleton. Orseau and Ring consider
several hypothetical AGI designs, many of which seem likely to accept the gambit,given certain assumptions.
In a related paper, Ring and Orseau ( 2011 ) consider the consequences of an AGI
being able to modify itself to receive the maximum possible reward. They show thatcertain types of AGIs will then come to only care about their own survival.Hypothetically, humans could promise not to threaten such AGIs in exchange forthem agreeing to be subject to AI-boxing procedures. For this to work, the system
would have to believe that humans will take care of its survival against external
threats better than it could itself. Hibbard ( 2012a ,c) discusses the kinds of AGIs
that would avoid the behaviors described by Ring and Orseau (Ring and Orseau2011 ; Orseau and Ring 2011 ).
3.5.7.3 Calculated Indifference
Another proposal is to make an AGI indifferent to a speci ﬁc event. For instance, the
AGI could be made indifferent to the detonation of explosives attached to its
10On the other hand, this might incentivize the AGI to deceive its controllers into believing it was
behaving properly, and also to actively hide any information which it even suspected might beinterpreted as misbehavior.3 Responses to the Journey to the Singularity 67
hardware, which might enable humans to have better control over it (Armstrong
2010 ; Armstrong et al. 2012 ).
3.5.7.4 Programmed Restrictions
Goertzel and Pitt ( 2012 ) suggest we ought to ensure that an AGI does not
self-improve too fast, because AGIs will be harder to control as they become moreand more cognitively superior to humans. To limit the rate of self-improvement inAGIs, perhaps AGIs could be programmed to extensively consult humans and otherAGI systems while improving themselves, in order to ensure that no unwantedmodiﬁcations would be implemented.
Omohundro ( 2012 ) discusses a number of programmed restrictions in the form
of constraints on what the AGI is allowed to do, with formal proofs being used toensure that an AGI will not violate its safety constraints. Such limited AGI systemswould be used to design more sophisticated AGIs.
Programmed restrictions are problematic, as the AGI might treat these merely as
problems to solve in the process of meeting its goals, and attempt to overcome them(Omohundro 2008 ). Making an AGI not want to quickly self-improve might not
solve the problem by itself. If the AGI ends up with a second-order desire to rid
itself of such a disinclination, the stronger desire will eventually prevail (Suber
2002 ). Even if the AGI wanted to maintain its disinclination toward rapid
self-improvement, it might still try to circumvent the goal in some other way, suchas by creating a copy of itself which did not have that disinclination (Omohundro2008 ). Regardless, such limits could help control less sophisticated AGIs.
3.5.7.5 Legal Machine Language
Weng et al. ( 2008 ,2009 ) propose a “legal machine language ”which could be used
to formally specify actions which the AGI is allowed or disallowed to do.Governments could then enact laws written in legal machine language, allowingthem to be programmed into robots.
3.5.7.6 “Motivational Weaknesses ”Proposals —Our View
Overall, motivational weaknesses seem comparable to external constraints possibly
useful and worth studying, but not something to rely on exclusively, particularly inthe case of superintelligent AGIs. As with external constraints and Oracle AIs, anarms race situation might provide a considerable incentive to loosen or remove suchconstraints.68 K. Sotala and R. Yampolskiy
3.6 Conclusion
We began this paper by noting that a number of researchers are predicting AGI in
the next twenty to one hundred years. One must not put excess trust in this time
frame as Armstrong and Sotala ( 2012 ) show, experts have been terrible at pre-
dicting AGI. Muehlhauser and Salamon ( 2012 ) consider a number of methods other
than expert opinion that could be used for predicting AGI, but ﬁnd that they too
provide suggestive evidence at best.
It would be a mistake, however, to leap from “AGI is very hard to predict ”to
“AGI must be very far away ”. Our brains are known to think about uncertain,
abstract ideas like AGI in “far mode, ”which also makes it feel like AGI must be
temporally distant (Trope and Liberman 2010 ), but something being uncertain is
not strong evidence that it is far away . When we are highly ignorant about
something, we should widen our error bars in both directions. Thus, we shouldn ’t
be highly con ﬁdent that AGI will arrive this century, and we shouldn ’t be highly
conﬁdent that it won ’t.
Next, we explained why AGIs may be an existential risk. A trend toward
automatization would give AGIs increased in ﬂuence in society, and there might be
a discontinuity in which they gained power rapidly. This could be a disaster forhumanity if AGIs don ’t share our values, and in fact it looks dif ﬁcult to cause them
to share our values because human values are complex and fragile, and therefore
problematic to specify.
The recommendations given for dealing with the problem can be divided into
proposals for societal action, external constraints, and internal constraints(Table 3.1). Many proposals seem to suffer from serious problems, or seem to be of
limited effectiveness. Others seem to have enough promise to be worth exploring.We will conclude by reviewing the proposals which we feel are worthy of furtherstudy.
As a brief summary of our views, in the medium term, we think that the pro-
posals of AGI con ﬁnement (Sect. 3.4.1 ), Oracle AI (Sect. 3.5.1 ), and motivational
weaknesses (5.6) would have promise in helping create safer AGIs. These proposalsshare in common the fact that, although they could help a cautious team ofresearchers create an AGI, they are not solutions to the problem of AGI risk, as theydo not prevent others from creating unsafe AGIs, nor are they suf ﬁcient in guar-
anteeing the safety of suf ﬁciently intelligent AGIs. Regulation (Sect. 3.3.3 ) as well
as human capability enhancement (Sect. 3.3.4 ) could also help to somewhat reduce
AGI risk. In the long run, we will need the ability to guarantee the safety of freely
acting AGIs. For this goal, value learning (Sect. 3.5.2.5 ) would seem like the most
reliable approach if it could be made to work, with human-like architecture(Sect. 3.5.3.4 ) a possible alternative which seems less reliable but possibly easier to
build. Formal veri ﬁcation (Sect. 3.5.6 ) seems like a very important tool in helping
to ensure the safety of our AGIs, regardless of the exact approach that we choose.
Of the societal proposals, we are supportive of the calls to regulate AGI
development, but we admit there are many practical hurdles which might make this3 Responses to the Journey to the Singularity 69
infeasible. The economic and military potential of AGI, and the dif ﬁculty of
enforcing compliance to regulations and arms treaties could lead to unstoppablearms races.
Weﬁnd ourselves in general agreement with the authors who advocate funding
additional research into safe AGI as the primary solution. Such research will alsoTable 3.1 Responses to catastrophic AGI risk
Societal proposals
Do nothing AGI is distant
Little risk, no action neededLet them kill us
Integrate to society Legal and economic controls
Foster positive values
Regulate research Review boards
Encourage safety researchDifferential technological progressInternational mass surveillance
Enhance human capabilitiesRelinquish technology Outlaw AGI
Restrict hardware
AGI design proposalsExternal constraints Internal constraints
AGI
conﬁnementSafe questions Oracle AI
Virtual worlds Top-down approaches Three lawsResetting the
AGICategorical imperative
Checks and
balancesPrinciple of Voluntary
Joyous Growth
AGI
enforcementUtilitarianism
Value learningApproval-directed agents
Bottom-up and hybrid
approachesEvolutionary invariants
Evolved moralityReinforcement learningHuman-like AGI
AGI NannyFormal veri ﬁcation
Motivational scaffoldingMotivational weaknesses High discount rates
Easily satiable goalsCalculated indifferenceProgrammed restrictionsLegal Machine Language70 K. Sotala and R. Yampolskiy
help establish the kinds of constraints which would make it possible to successfully
carry out integration proposals.
Uploading approaches, in which human minds are made to run on computers
and then augmented, might buy us some time to develop safe AGI. However, it is
unclear whether they can be developed before AGI, and large-scale uploading could
create strong evolutionary trends which seem dangerous in and of themselves. AsAGIs seem likely to eventually outpace uploads, uploading by itself is probably nota sufﬁcient solution. What uploading could do is to reduce the initial advantages
that AGIs enjoy over (partially uploaded) humanity, so that other responses to AGIrisk can be deployed more effectively.
External constraints are likely to be useful in controlling AGI systems of limited
intelligence, and could possibly help us develop more intelligent AGIs while
maintaining their safety. If inexpensive external constraints were readily available,
this could encourage even research teams skeptical about safety issues to implementthem. Yet it does not seem safe to rely on these constraints once we are dealing witha superhuman intelligence, and we cannot trust everyone to be responsible enoughto contain their AGI systems, especially given the economic pressures to “release ”
AGIs. For such an approach to be a solution for AGI risk in general, it would haveto be adopted by all successful AGI projects, at least until safe AGIs were devel-oped. Much the same is true of attempting to design Oracle AIs. In the short term,
such efforts may be reinforced by research into motivational weaknesses, internal
constraints that make AGIs easier to control via external means.
In the long term, the internal constraints that show the most promise are value
extrapolation approaches and human-like architectures. Value extrapolationattempts to learn human values and interpret them as we would wish them to beinterpreted. These approaches have the advantage of potentially maximizing thepreservation of human values, and the disadvantage that such approaches mayprove intractable, or impossible to properly formalize. Human-like architectures
seem easier to construct, as we can simply copy mechanisms that are used within
the human brain, but it seems hard to build such an exact match as to reliablyreplicate human values. Slavish reproductions of the human psyche also seem likelyto be outcompeted by less human, more ef ﬁcient architectures.
Both approaches would bene ﬁt from better formal veri ﬁcation methods, so that
AGIs which were editing and improving themselves could verify that the modi ﬁ-
cations did not threaten to remove the AGIs ’motivation to follow their original
goals. Studies which aim to uncover the roots of human morals and preferences also
seem like candidates for research that would bene ﬁt the development of safe AGI,
as do studies into computational models of ethical reasoning.
We reiterate that when we talk about “human values ”, we are not making the
claim that human values would be static, nor that current human values would be
ideal. Nor do we wish to imply that the values of other sentient beings would beunimportant. Rather, we are seeking to guarantee the implementation of some verybasic values, such as the avoidance of unnecessary suffering, the preservation ofhumanity, and the prohibition of forced brain reprogramming. We believe the vast
majority of humans would agree with these values and be sad to see them lost.3 Responses to the Journey to the Singularity 71
Acknowledgementss Special thanks to Luke Muehlhauser for extensive assistance throughout
the writing process. We would also like to thank Abram Demski, Alexei Turchin, Alexey Potapov,Anders Sandberg, Andras Kornai, Anthony Berglas, Aron Vallinder, Ben Goertzel, Ben Noble,
Ben Sterrett, Brian Rabkin, Bill Hibbard, Carl Shulman, Dana Scott, Daniel Dewey, David Pearce,
Evelyn Mitchell, Evgenij Thorstensen, Frank White, gwern branwen, Harri Valpola, Jaan Tallinn,Jacob Steinhardt, James Babcock, James Miller, Joshua Fox, Louie Helm, Mark Gubrud, MarkWaser, Michael Anissimov, Michael Vassar, Miles Brundage, Moshe Looks, Randal Koene,
Robin Hanson, Risto Saarelma, Steve Omohundro, Suzanne Lidström, Steven Kaas, Stuart
Armstrong, Tim Freeman, Ted Goertzel, Toni Heinonen, Tony Barrett, Vincent Müller, VladimirNesov, Wei Dai, and two anonymous reviewers as well as several users of lesswrong.com for theirhelpful comments.
References
Agliata, Daniel, and Stacey Tantleff-Dunn. 2004. “The Impact of Media Exposure on Males ’Body
Image ”. Journal of Social and Clinical Psychology 23(1): 7 –22. doi: 10.1521/jscp.23.1.7.26988 .
Alexander, Scott. 2015. “AI researchers on AI risk ”. Slate Star Codex [blog]. http://slatestarcodex.
com/2015/05/22/ai-researchers-on-ai-risk/ .
Anderson, Monica. 2010. “Problem Solved: Unfriendly AI ”. H + Magazine, December 15. http://
hplusmagazine.com/2010/12/15/problem-solved-unfriendly-ai/ .
Anderson, Michael, Susan Leigh Anderson, and Chris Armen, eds. 2005a. Machine Ethics: Papers
from the 2005 AAAI Fall Symposium. Technical Report, FS-05-06. AAAI Press, Menlo Park,
CA. http://www.aaai.org/Library/Symposia/Fall/fs05-06 .
Anderson, Michael, Susan Leigh Anderson, and Chris Armen. 2005b. “MedEthEx: Toward a
Medical Ethics Advisor. ”In Caring Machines: AI in Eldercare: Papers from the 2005 AAAI
Fall Symposium, edited by Timothy Bickmore, 9 –16. Technical Report, FS-05-02. AAAI
Press, Menlo Park, CA. http://aaaipress.org/Papers/Symposia/Fall/2005/FS-05-02/FS05-02-
002.pdf .
Anderson, Michael, Susan Leigh Anderson, and Chris Armen. 2006. “An Approach to Computing
Ethics. ”IEEE Intelligent Systems 21(4): 56 –63. doi: 10.1109/MIS.2006.64 .
Anderson, Susan Leigh. 2011. “The Unacceptability of Asimov ’s Three Laws of Robotics as a
Basis for Machine Ethics ”. In Anderson and Anderson 2011, 285 –296.
Annas, George J., Lori B. Andrews, and Rosario M. Isasi. 2002. “Protecting the Endangered
Human: Toward an International Treaty Prohibiting Cloning and Inheritable Alterations ”.
American Journal of Law & Medicine 28(2 –3): 151 –178.
Anthony, Dick, and Thomas Robbins. 2004. “Conversion and ‘Brainwashing ’in New Religious
Movements ”. In The Oxford Handbook of New Religious Movements, 1st ed., edited by
James R. Lewis, 243 –297. New York: Oxford University Press. doi: 10.1093/oxfordhb/
9780195369649.003 . 0012.
Armstrong, Stuart. 2007. “Chaining God: A Qualitative Approach to AI, Trust and Moral
Systems ”. Unpublished manuscript, October 20. Accessed December 31, 2012. http://www.
neweuropeancentury.org/GodAI.pdf .
Armstrong, Stuart. 2010. Utility Indifference. Technical Report, 2010-1. Oxford: Future of
Humanity Institute, University of Oxford. http://www.fhi.ox.ac.uk/reports/2010-1.pdf .
Armstrong, Stuart, Anders Sandberg, and Nick Bostrom. 2012. “Thinking Inside the Box:
Controlling and Using an Oracle AI ”. Minds and Machines 22(4): 299 –324. doi: 10.1007/
s11023-012-9282-2 .
Armstrong, Stuart, and Kaj Sotala. 2012. “How We ’re Predicting AI —or Failing To ”. In Beyond
AI: Arti ﬁcial Dreams, edited by Jan Romportl, Pavel Ircing, Eva Zackova, Michal Polak, and
Radek Schuster, 52 –75. Pilsen: University of West Bohemia. Accessed February 2, 2013.
http://www.kky.zcu.cz/en/publications/1/JanRomportl\_2012\_BeyondAIArti ﬁcial.pdf .72 K. Sotala and R. Yampolskiy
Asimov, Isaac. 1942. “Runaround ”. Astounding Science-Fiction, March, 94 –103.
Axelrod, Robert. 1987. “The Evolution of Strategies in the Iterated Prisoner ’s Dilemma ”.I n
Genetic Algorithms and Simulated Annealing, edited by Lawrence Davis, 32 –41. Los Altos,
CA: Morgan Kaufmann.
Baars, Bernard J. 2002. “The Conscious Access Hypothesis: Origins and Recent Evidence ”.
Trends in Cognitive Sciences 6(1): 47 –52. doi: 10.1016/S1364-6613(00)01819-2 .
Baars, Bernard J. 2005. “Global Workspace Theory of Consciousness: Toward a Cognitive
Neuroscience of Human Experience ”. In The Boundaries of Consciousness: Neurobiology and
Neuropathology, edited by Steven Laureys, 45 –53. Progress in Brain Research 150. Boston:
Elsevier.
Beavers, Anthony F. 2009. “Between Angels and Animals: The Question of Robot Ethics; or, Is
Kantian Moral Agency Desirable? ”Paper presented at the Annual Meeting of the Association
for Practical and Professional Ethics, Cincinnati, OH, March.
Beavers, Anthony F. 2012. “Moral Machines and the Threat of Ethical Nihilism ”. In Lin, Patrick,
Keith Abney, and George A. Bekey, eds. Robot Ethics: The Ethical and Social Implications of
Robotics. Intelligent Robotics and Autonomous Agents. Cambridge, MA: MIT Press, 333 –344.
Benatar, David. 2006. Better Never to Have Been: The Harm of Coming into Existence. New
York: Oxford University Press.
Berglas, Anthony. 2012. “Artiﬁcial Intelligence Will Kill Our Grandchildren (Singularity) ”.
Unpublished manuscript, draft 9, January. Accessed December 31, 2012. http://berglas.org/
Articles/AIKillGrandchildren/AIKillGrandchildren.html .
Bostrom, Nick. 2002. “Existential Risks: Analyzing Human Extinction Scenarios and Related
Hazards. ”Journal of Evolution and Technology 9. http://www.jetpress.org/volume9/risks.html .
Bostrom, Nick. 2004. “The Future of Human Evolution ”. In Two Hundred Years After Kant, Fifty
Years After Turing, edited by Charles Tandy, 339 –371. Vol. 2. Death and Anti-Death. Palo
Alto, CA: Ria University Press.
Bostrom, Nick. 2012. “The Superintelligent Will: Motivation and Instrumental Rationality in
Advanced Arti ﬁcial Agents ”.I n“Theory and Philosophy of AI, ”edited by Vincent C. Müller.
Special issue, Minds and Machines 22(2): 71 –85. doi: 10.1007/s11023-012-9281-3 .
Bostrom, Nick. 2014. Superintelligence: Paths, dangers, strategies. Oxford University Press.
Bostrom, Nick, and Eliezer Yudkowsky. 2013. “The Ethics of Arti ﬁcial Intelligence ”.I n
Cambridge Handbook of Arti ﬁcial Intelligence, edited by Keith Frankish and William Ramsey.
New York: Cambridge University Press.
Branwen, Gwern. 2012. “Slowing Moore ’s Law: Why You Might Want to and How You Would
Do It ”. gwern.net. December 11. Accessed December 31, 2012. http://www.gwern.net/Slowing
%20Moore ’s%20Law .
Brin, David. 1998. The Transparent Society: Will Technology Force Us to Choose Between
Privacy and Freedom? Reading, MA: Perseus Books.
Bringsjord, Selmer, and Alexander Bringsjord. 2012. “Belief in the Singularity is Fideistic ”.I n
Eden, Amnon, Johnny Søraker, James H. Moor, and Eric Steinhart, eds. Singularity
Hypotheses: A Scienti ﬁc and Philosophical Assessment. The Frontiers Collection. Berlin:
Springer.
Brooks, Rodney A. 2008. “I, Rodney Brooks, Am a Robot ”. IEEE Spectrum 45(6): 68 –71. doi: 10.
1109/MSPEC.2008.4531466 .
Brynjolfsson, Erik, and Andrew McAfee. 2011. Race Against The Machine: How the Digital
Revolution is Accelerating Innovation, Driving Productivity, and Irreversibly Transforming
Employment and the Economy. Lexington, MA: Digital Frontier. Kindle edition.
Bryson, Joanna, and Phil Kime. 1998. “Just Another Artifact: Ethics and the Empirical Experience
of AI ”. Paper presented at the Fifteenth Internation Congress on Cybernetics, Namur, Belgium.
http://www.cs.bath.ac.uk/ ∼jjb/web/aiethics98.html .
Butler, Samuel [Cellarius, pseud.]. 1863. “Darwin Among the Machines ”. Christchurch Press,
June 13. http://www.nzetc.org/tm/scholarly/tei-ButFir-t1-g1-t1-g1-t4-body.html .
Cade, C. Maxwell. 1966. Other Worlds Than Ours. 1st ed. London: Museum.3 Responses to the Journey to the Singularity 73
Cattell, Rick, and Alice Parker. 2012. Challenges for Brain Emulation: Why is Building a Brain so
Difﬁcult? Synaptic Link, February 5. http://synapticlink.org/Brain%20Emulation%
20Challenges.pdf .
Chalmers, David John. 2010. “The Singularity: A Philosophical Analysis ”. Journal of
Consciousness Studies 17 (9 –10): 7 –65.http://www.ingentaconnect.com/content/imp/jcs/
2010/00000017/f0020009/art00001 .
Christiano, Paul F. 2012. “‘Indirect Normativity ’Write-up ”. Ordinary Ideas (blog), April 21.
http://ordinaryideas.wordpress.com/2012/04/21/indirect-normativity-write-up/ .
Christiano, Paul F. 2014a Approval-directed agents. December 1 https://medium.com/ai-control/
model-free-decisions-6e6609f5d99e .
Christiano, Paul F. 2014b. Approval-directed search. December 14 https://medium.com/
@paulfchristiano/approval-directed-search-63457096f9e4 .
Christiano, Paul F. 2014c. Approval-directed bootstrapping. December 20 https://medium.com/ai-
control/approval-directed-bootstrapping-5d49e886c14f .
Christiano, Paul F. 2015. Learn policies or goals? April 21 https://medium.com/ai-control/learn-
policies-or-goals-348add76b8eb .
Clark, Gregory. 2007. A Farewell to Alms: A Brief Economic History of the World. 1st ed.
Princeton, NJ: Princeton University Press.
Clarke, Roger. 1993. “Asimov ’s Laws of Robotics: Implications for Information Technology, Part
1”. Computer 26(12): 53 –61. doi: 10.1109/2.247652 .
Clarke, Roger. 1994. “Asimov ’s Laws of Robotics: Implications for Information Technology, Part
2”. Computer 27 (1): 57 –66. doi: 10.1109/2.248881 .
Daley, William. 2011. “Mitigating Potential Hazards to Humans from the Development of
Intelligent Machines ”. Synthese 2:44 –50.http://www.synesisjournal.com/vol2\_g/2011\_2\_44-
50\_Daley.pdf .
Davis, Ernest. 2012. “The Singularity and the State of the Art in Arti ﬁcial Intelligence ”. Working
Paper, New York, May 9. Accessed July 22, 2013. http://www.cs.nyu.edu/ ∼davise/papers/
singularity.pdf .
Dayan, Peter. 2011. “Models of Value and Choice ”. In Neuroscience of Preference and Choice:
Cognitive and Neural Mechanisms, edited by Raymond J. Dolan and Tali Sharot, 33 –52.
Waltham, MA: Academic Press.
De Garis, Hugo. 2005. The Artilect War: Cosmists vs. Terrans: A Bitter Controversy Concerning
Whether Humanity Should Build Godlike Massively Intelligent Machines.Palm Springs, CA:
ETC Publications.
Degabriele, Jean Paul, Kenny Paterson, and Gaven Watson. 2011. “Provable Security in the Real
World ”. IEEE Security & Privacy Magazine 9(3): 33 –41. doi: 10.1109/MSP.2010.200 .
Dennett, Daniel C. 1987. “Cognitive Wheels: The Frame Problem of AI ”. In Pylyshyn 1987, 41 –
64.
Dennett, Daniel C. 2012. “The Mystery of David Chalmers ”. Journal of Consciousness Studies 19
(1–2): 86 –95.http://ingentaconnect.com/content/imp/jcs/2012/00000019/F0020001/art00005 .
Deutsch, David. 2011. The Beginning of In ﬁnity: Explanations that Transform the World. 1st ed.
New York: Viking.
Dewey, Daniel. 2011. “Learning What to Value ”. In Schmidhuber, Jürgen, Kristinn R. Thórisson,
and Moshe Looks, eds. Arti ﬁcial General Intelligence: 4th International Conference, AGI
2011, Mountain View, CA, USA, August 3 –6, 2011. Proceedings. Lecture Notes in Computer
Science 6830. Berlin: Springer, 309 –314.
Dietrich, Eric. 2007. “After The Humans Are Gone ”. Philosophy Now, May –June. http://
philosophynow.org/issues/61/After\_The\_Humans\_Are\_Gone .
Docherty, Bonnie, and Steve Goose. 2012. Losing Humanity: The Case Against Killer Robots.
Cambridge, MA: Human Rights Watch and the International Human Rights Clinic, November19.http://www.hrw.org/sites/default/ ﬁles/reports/arms1112ForUpload\_0\_0.pdf .
Douglas, Thomas. 2008. “Moral Enhancement ”. Journal of Applied Philosophy 25(3): 228 –245.
doi:10.1111/j.1468-5930.2008.00412.x .74 K. Sotala and R. Yampolskiy
Eckersley, Peter, and Anders Sandberg. 2013. Is Brain Emulation Dangerous? Journal of Arti ﬁcial
General Intelligence 4.3: 170 –194.
Fox, Joshua, and Carl Shulman. 2010. “Superintelligence Does Not Imply Benevolence ”.I n
Mainzer, Klaus, ed. 2010. ECAP10: VIII European Conference on Computing and Philosophy.
Munich: Dr. Hut.
Frankfurt, Harry G. 1971. “Freedom of the Will and the Concept of a Person ”. Journal of
Philosophy 68 (1): 5 –20. doi: 10.2307/2024717 .
Franklin, Stan, and F. G. Patterson Jr. 2006. “The LIDA Architecture: Adding New Modes of
Learning to an Intelligent, Autonomous, Software Agent ”. In IDPT-2006 Proceedings.San
Diego, CA: Society for Design & Process Science. http://ccrg.cs.memphis.edu/assets/papers/
zo-1010-lida-060403.pdf .
Freeman, Tim. 2009. “Using Compassion and Respect to Motivate an Arti ﬁcial Intelligence ”.
Unpublished manuscript, March 8. Accessed December 31, 2012. http://fungible.com/respect/
paper.html .
Friedman, Batya, and Peter H. Kahn. 1992. “Human Agency and Responsible Computing:
Implications for Computer System Design ”. Journal of Systems and Software 17 (1): 7 –14.
doi:10.1016/0164-1212(92)90075-U .
Future of Life Institute. 2015. Research Priorities for Robust and Bene ﬁcial Arti ﬁcial Intelligence:
an Open Letter. http://futureo ﬂife.org/misc/open\_letter .
Gewirth, Alan. 1978. Reason and Morality. Chicago: University of Chicago Press.
Goertzel, Ben. 2004a. “Encouraging a Positive Transcension: Issues in Transhumanist Ethical
Philosophy ”. Dynamical Psychology. http://www.goertzel.org/dynapsyc/2004/Positive
Transcension.htm .
Goertzel, Ben. 2004b. “Growth, Choice and Joy: Toward a Precise De ﬁnition of a Universal
Ethical Principle ”. Dynamical Psychology. http://www.goertzel.org/dynapsyc/2004/Growth
ChoiceJoy.htm .
Goertzel, Ben. 2010a. “Coherent Aggregated Volition: A Method for Deriving Goal System Content
for Advanced, Bene ﬁcial AGIs ”. The Multiverse According to Ben (blog), March 12. http://
multiverseaccordingtoben.blogspot.ca/2010/03/coherent-aggregated-volitiontoward.html .
Goertzel, Ben. 2010b. “GOLEM: Toward an AGI Meta-Architecture Enabling Both Goal
Preservation and Radical Self-Improvement ”. Unpublished manuscript, May 2. Accessed
December 31, 2012. http://goertzel.org/GOLEM.pdf .
Goertzel, Ben. 2012a. “CogPrime: An Integrative Architecture for Embodied Arti ﬁcial General
Intelligence ”. OpenCog Foundation. October 2. Accessed December 31, 2012. http://wiki.
opencog.org/w/CogPrime\_Overview .
Goertzel, Ben. 2012b. “Should Humanity Build a Global AI Nanny to Delay the Singularity Until
It’s Better Understood? ”Journal of Consciousness Studies 19(1 –2): 96 –111. http://
ingentaconnect.com/content/imp/jcs/2012/00000019/F0020001/art00006 .
Goertzel, Ben, and Stephan Vladimir Bugaj. 2008. “Stages of Ethical Development in Arti ﬁcial
General Intelligence Systems ”. In Wang, Pei, Ben Goertzel, and Stan Franklin, eds. Arti ﬁcial
General Intelligence 2008: Proceedings of the First AGI Conference. Frontiers in Arti ﬁcial
Intelligence and Applications 171. Amsterdam: IOS, 448 –459.
Goertzel, Ben, and Joel Pitt. 2012. “Nine Ways to Bias Open-Source AGI Toward Friendliness ”.
Journal of Evolution and Technology 22(1): 116 –131. http://jetpress.org/v22/goertzel-pitt.htm .
Gomes, Lee. 2015. Facebook AI Director Yann LeCun on His Quest to Unleash Deep Learning
and Make Machines Smarter. IEEE Spectrum. http://spectrum.ieee.org/automaton/robotics/
artiﬁcial-intelligence/facebook-ai-director-yann-lecun-on-deep-learning#qaTopicEight .
Good, Irving John. 1970. “Some Future Social Repercussions of Computers ”. International Journal
of Environmental Studies 1(1 –4): 67 –79. doi: 10.1080/00207237008709398 .
Gordon-Spears, Diana F. 2003. “Asimov ’s Laws: Current Progress ”. In Formal Approaches to
Agent-Based Systems: Second International Workshop, FAABS 2002, Greenbelt, MD, USA,
October 29 –31, 2002. Revised Papers, edited by Michael G. Hinchey, James L. Rash, Walter F.
Truszkowski, Christopher Rouff, and Diana F. Gordon-Spears, 257 –259. Lecture Notes in
Computer Science 2699. Berlin: Springer. doi: 10.1007/978-3-540-45133-4\_23 .3 Responses to the Journey to the Singularity 75
Groesz, Lisa M., Michael P. Levine, and Sarah K. Murnen. 2001. “The Effect of Experimental
Presentation of Thin Media Images on Body Satisfaction: A Meta-Analytic Review ”.
International Journal of Eating Disorders 31(1): 1 –16. doi: 10.1002/eat.10005 .
Guarini, Marcello. 2006. “Particularism and the Classi ﬁcation and Reclassi ﬁcation of Moral
Cases ”. IEEE Intelligent Systems 21 (4): 22 –28. doi: 10.1109/MIS.2006.76 .
Gubrud, Mark Avrum. 1997. “Nanotechnology and International Security ”. Paper presented at the
Fifth Foresight Conference on Molecular Nanotechnology, Palo Alto, CA, November 5 –8.
http://www.foresight.org/Conferences/MNT05/Papers/Gubrud/ .
Gunkel, David J. 2012. The Machine Question: Critical Perspectives on AI, Robotics, and Ethics.
Cambridge, MA: MIT Press.
Haidt, Jonathan. 2006. The Happiness Hypothesis: Finding Modern Truth in Ancient Wisdom. 1st
ed. New York: Basic Books.
Hall, John Storrs. 2007a. Beyond AI: Creating the Conscience of the Machine. Amherst, NY:
Prometheus Books.
Hall, John Storrs. 2011. “Ethics for Self-Improving Machines ”. In Anderson and Anderson 2011,
512–523.
Hanson, Robin. 1994. “If Uploads Come First: The Crack of a Future Dawn ”. Extropy 6(2). http://
hanson.gmu.edu/uploads.html .
Hanson, Robin. 2000. “Shall We Vote on Values, But Bet on Beliefs? ”Unpublished manuscript,
September. Last revised October 2007. http://hanson.gmu.edu/futarchy.pdf .
Hanson, Robin. 2008. “Economics of the Singularity ”. IEEE Spectrum 45 (6): 45 –50. doi: 10.1109/
MSPEC.2008.4531461 .
Hanson, Robin. 2009. “Prefer Law to Values ”. Overcoming Bias (blog), October 10. http://www.
overcomingbias.com/2009/10/prefer-law-to-values.html .
Hanson, Robin. 2012. “Meet the New Con ﬂict, Same as the Old Con ﬂict”. Journal of
Consciousness Studies 19(1 –2): 119 –125. http://www.ingentaconnect.com/content/imp/jcs/
2012/00000019/F0020001/art00008 .
Hare, Robert D., Danny Clark, Martin Grann, and David Thornton. 2000. “Psychopathy and the
Predictive Validity of the PCL-R: An International Perspective ”. Behavioral Sciences & the
Law 18(5): 623 –645. doi: 10.1002/1099-0798(200010)18:5<623::AID-BSL409>3.0.CO;2-W .
Harris, Grant T., and Marnie E. Rice. 2006. “Treatment of Psychopathy: A Review of Empirical
Findings ”. In Handbook of Psychopathy, edited by Christopher J. Patrick, 555 –572. New
York: Guilford.
Hart, David, and Ben Goertzel. 2008. “OpenCog: A Software Framework for Integrative Arti ﬁcial
General Intelligence ”. Unpublished manuscript. http://www.agiri.org/OpenCog\_AGI-08.pdf .
Hayworth, Kenneth J. 2012. “Electron Imaging Technology for Whole Brain Neural Circuit
Mapping ”. International Journal of Machine Consciousness 4(1): 87 –108. doi: 10.1142/
S1793843012500060 .
Heylighen, Francis. 2007. “Accelerating Socio-Technological Evolution: From Ephemeralization
and Stigmergy to the Global Brain ”. In Globalization as Evolutionary Process: Modeling
Global Change, edited by George Modelski, Tessaleno Devezas, and William R. Thompson,
284–309. Rethinking Globalizations 10. New York: Routledge.
Heylighen, Francis. 2012. “Brain in a Vat Cannot Break Out. ”Journal of Consciousness Studies 19
(1–2): 126 –142. http://www.ingentaconnect.com/content/imp/jcs/2012/00000019/F0020001/
art00009 .
Hibbard, Bill. 2001. “Super-Intelligent Machines ”. ACM SIGGRAPH Computer Graphics 35 (1):
13–15.http://www.siggraph.org/publications/newsletter/issues/v35/v35n1.pdf .
Hibbard, Bill. 2005a. “Critique of the SIAI Collective Volition Theory ”. Unpublished manuscript,
December. Accessed December 31, 2012. http://www.ssec.wisc.edu/ ∼billh/g/SIAI\_CV\_
critique.html .
Hibbard, Bill. 2005b. “The Ethics and Politics of Super-Intelligent Machines ”. Unpublished
manuscript, July. Microsoft Word ﬁle, accessed December 31, 2012. https://sites.google.com/
site/whibbard/g/SI\_ethics\_politics.doc .76 K. Sotala and R. Yampolskiy
Hibbard, Bill. 2008. “Open Source AI. ”In Wang, Pei, Ben Goertzel, and Stan Franklin, eds.
Artiﬁcial General Intelligence 2008: Proceedings of the First AGI Conference. Frontiers in
Artiﬁcial Intelligence and Applications 171. Amsterdam: IOS, 473 –477.
Hibbard, Bill. 2012a. “Avoiding Unintended AI Behaviors ”. In Bach, Joscha, Ben Goertzel, and
Matthew Iklé, eds. Arti ﬁcial General Intelligence: 5th International Conference, AGI 2012,
Oxford, UK, December 8 –11, 2012. Proceedings. Lecture Notes in Arti ﬁcial Intelligence 7716.
New York: Springer. doi: 10.1007/978-3-642-35506-6, 107 –116.
Hibbard, Bill. 2012b. “Decision Support for Safe AI Design ”. In Bach, Joscha, Ben Goertzel, and
Matthew Iklé, eds. Arti ﬁcial General Intelligence: 5th International Conference, AGI 2012,
Oxford, UK, December 8 –11, 2012. Proceedings. Lecture Notes in Arti ﬁcial Intelligence 7716.
New York: Springer. doi: 10.1007/978-3-642-35506-6, 117 –125.
Hibbard, Bill. 2012c. “Model-Based Utility Functions ”. Journal of Arti ﬁcial General Intelligence 3
(1): 1 –24. doi: 10.2478/v10229-011-0013-5 .
Hibbard, Bill. 2012d. The Error in My 2001 VisFiles Column, September. Accessed December 31,
2012. http://www.ssec.wisc.edu/ ∼billh/g/vis ﬁles\_error.html .
Horvitz, Eric J., and Bart Selman. 2009. Interim Report from the AAAI Presidential Panel on
Long- Term AI Futures. Palo Alto, CA: AAAI, August. http://www.aaai.org/Organization/
Panel/panelnote.pdf .
Hughes, James. 2001. “Relinquishment or Regulation: Dealing with Apocalyptic Technological
Threats ”. Hartford, CT, November 14.
IEEE Spectrum. 2008. “Tech Luminaries Address Singularity ”:“The Singularity; Special Report ”.
(June).
Jenkins, Anne. 2003. “Artiﬁcial Intelligence and the Real World ”. Futures 35 (7): 779 –786. doi: 10.
1016/S0016-3287(03)00029-6 .
Joy, Bill. 2000. “Why the Future Doesn ’t Need Us ”. Wired, April. http://www.wired.com/wired/
archive/8.04/joy.html .
Karnofsky, Holden. 2012. “Thoughts on the Singularity Institute (SI) ”. Less Wrong (blog), May
11.http://lesswrong.com/lw/cbs/thoughts\_on\_the\_singularity\_institute\_si/ .
Karnofsky, Holden, and Jaan Tallinn. 2011. “Karnofsky & Tallinn Dialog on SIAI Ef ﬁcacy”.
Accessed December 31, 2012. http://xa.yimg.com/kq/groups/23070378/1331435883/name/
Jaan+Tallinn+2011+05+-+revised.doc .
Kipnis, David. 1972. “Does Power Corrupt? ”. Journal of Personality and Social Psychology 24(1):
33–41. doi: 10.1037/h0033390 .
Koene, Randal A. 2012a. “Embracing Competitive Balance: The Case for Substrate-Independent
Minds and Whole Brain Emulation ”. In Eden, Amnon, Johnny Søraker, James H. Moor, and
Eric Steinhart, eds. Singularity Hypotheses: A Scienti ﬁc and Philosophical Assessment. The
Frontiers Collection. Berlin: Springer.
Koene, Randal A. 2012b. “Experimental Research in Whole Brain Emulation: The Need for
Innovative in Vivo Measurement Techniques ”. International Journal of Machine Conscious-
ness 4(1): 35 –65. doi: 10.1142/S1793843012400033 .
Kornai, András. 2014. Bounding the impact of AGI. Journal of Experimental & Theoretical
Artiﬁcial Intelligence, 26(3), 417 –438.
Kurzweil, Ray. 2001. “Response to Stephen Hawking ”. Kurzweil Accelerating Intelligence.
September 5. Accessed December 31, 2012. http://www.kurzweilai.net/response-to-stephen-
hawking .
Kurzweil, Ray. 2005. The Singularity Is Near: When Humans Transcend Biology. New York:
Viking.
Lampson, Butler W. 1973. “A Note on the Con ﬁnement Problem ”. Communications of the ACM
16(10): 613 –615. doi: 10.1145/362375.362389 .
Legg, Shane. 2009. “Funding Safe AGI ”. Vetta Project (blog), August 3. http://www.vetta.org/
2009/08/funding-safe-agi/ .
Madrigal, Alexis C. 2015. The case against killer robots, from a guy actually working on arti ﬁcial
intelligence. http://fusion.net/story/54583/the-case-against-killer-robots-from-a-guy-actually-
building-ai/ .3 Responses to the Journey to the Singularity 77
Mann, Steve, Jason Nolan, and Barry Wellman. 2003. “Sousveillance: Inventing and Using
Wearable Computing Devices for Data Collection in Surveillance Environments ”. Surveillance
& Society 1(3): 331 –355. http://library.queensu.ca/ojs/index.php/surveillance-and-society/
article/view/3344 .
McCauley, Lee. 2007. “AI Armageddon and the Three Laws of Robotics ”. Ethics and Information
Technology 9(2): 153 –164. doi: 10.1007/s10676-007-9138-2 .
McCulloch, W. S. 1956. “Toward Some Circuitry of Ethical Robots; or, An Observational Science
of the Genesis of Social Evaluation in the Mind-like Behavior of Artifacts ”. Acta Biotheoretica
11(3–4): 147 –156. doi: 10.1007/BF01557008 .
McDermott, Drew. 2012. “Response to ‘The Singularity ’by David Chalmers ”. Journal of
Consciousness Studies 19(1 –2): 167 –172. http://www.ingentaconnect.com/content/imp/jcs/
2012/00000019/F0020001/art00011 .
McGinnis, John O. 2010. “Accelerating AI ”. Northwestern University Law Review 104 (3): 1253 –
1270. http://www.law.northwestern.edu/lawreview/v104/n3/1253/LR104n3McGinnis.pdf .
McKibben, Bill. 2003. Enough: Staying Human in an Engineered Age. New York: Henry Holt.
McLeod, Peter, Kim Plunkett, and Edmund T. Rolls. 1998. Introduction to Connectionist
Modelling of Cognitive Processes. New York: Oxford University Press.
Miller, James D. 2012. Singularity Rising: Surviving and Thriving in a Smarter, Richer, and More
Dangerous World. Dallas, TX: BenBella Books.
Moore, David, Vern Paxson, Stefan Savage, Colleen Shannon, Stuart Staniford, and Nicholas
Weaver. 2003. “Inside the Slammer Worm ”. IEEE Security & Privacy Magazine 1(4): 33 –39.
doi:10.1109/MSECP.2003.1219056 .
Moore, David, Colleen Shannon, and Jeffery Brown. 2002. “Code-Red: A Case Study on the
Spread and Victims of an Internet Worm ”. In Proceedings of the Second ACM SIGCOMM
Workshop on Internet Measurment (IMW ’02), 273 –284. New York: ACM Press. doi: 10.1145/
637201.637244 .
Moravec, Hans P. 1988. Mind Children: The Future of Robot and Human Intelligence. Cambridge,
MA: Harvard University Press.
Moravec, Hans P. 1992. “Pigs in Cyberspace ”. Field Robotics Center. Accessed December 31,
2012. http://www.frc.ri.cmu.edu/ ∼hpm/project.archive/general.articles/1992/CyberPigs.html .
Moravec, Hans P. 1999. Robot: Mere Machine to Transcendent Mind. New York: Oxford
University Press.
Muehlhauser, Luke, and Louie Helm. 2012. “The Singularity and Machine Ethics ”. In Eden,
Amnon, Johnny Søraker, James H. Moor, and Eric Steinhart, eds. Singularity Hypotheses: A
Scienti ﬁc and Philosophical Assessment. The Frontiers Collection. Berlin: Springer.
Muehlhauser, Luke, and Anna Salamon. 2012. “Intelligence Explosion: Evidence and Import ”.I n
Eden, Amnon, Johnny Søraker, James H. Moor, and Eric Steinhart, eds. SingularityHypotheses: A Scienti ﬁc and Philosophical Assessment.The Frontiers Collection. Berlin:
Springer.
Mueller, Dennis C. 2003. Public Choice III. 3rd ed. New York: Cambridge University Press.
Müller, Vincent C., and Nick Bostrom. 2014. Future progress in arti ﬁcial intelligence: A survey of
expert opinion. Fundamental Issues of Arti ﬁcial Intelligence.
Murphy, Robin, and David D. Woods. 2009. “Beyond Asimov: The Three Laws of Responsible
Robotics ”. IEEE Intelligent Systems 24(4): 14 –20. doi: 10.1109/MIS.2009.69 .
Napier, William. 2008. “Hazards from Comets and Asteroids ”. In Bostrom, Nick, and Milan M.
Ćirkovi ć, eds. Global Catastrophic Risks. New York: Oxford University Press, 222 –237.
Ng, Andrew Y., and Stuart J. Russell. 2000. Algorithms for inverse reinforcement learning. In Icml
(pp. 663 –670).
Nielsen, Thomas D., and Finn V. Jensen. 2004. “Learning a Decision Maker ’s Utility Function
from (Possibly) Inconsistent Behavior ”. Arti ﬁcial Intelligence 160(1 –2): 53 –78. doi: 10.1016/j.
artint.2004.08.003 .
Nordmann, Alfred. 2007. “If and Then: A Critique of Speculative NanoEthics ”. NanoEthics 1(1):
31–46. doi: 10.1007/s11569-007-0007-6 .78 K. Sotala and R. Yampolskiy
Nordmann, Alfred. 2008. “Singular Simplicity ”. IEEE Spectrum, June. http://spectrum.ieee.org/
robotics/robotics-software/singular-simplicity .
Olson, Mancur. 1982. The Rise and Decline of Nations: Economic Growth, Stag ﬂation, and Social
Rigidities. New Haven, CT: Yale University Press.
Omohundro, Stephen M. 2007. “The Nature of Self-Improving Arti ﬁcial Intelligence ”. Paper
presented at Singularity Summit 2007, San Francisco, CA, September 8 –9.http://selfaware
systems.com/2007/10/05/paper-on-the-nature-of-self-improving-arti ﬁcial-intelligence/ .
Omohundro, Stephen M. 2008. “The Basic AI Drives ”. In Wang, Pei, Ben Goertzel, and Stan
Franklin, eds. Arti ﬁcial General Intelligence 2008: Proceedings of the First AGI Conference.
Frontiers in Arti ﬁcial Intelligence and Applications 171. Amsterdam: IOS, 483 –492.
Omohundro, Stephen M. 2012. “Rational Arti ﬁcial Intelligence for the Greater Good ”. In Eden,
Amnon, Johnny Søraker, James H. Moor, and Eric Steinhart, eds. Singularity Hypotheses: A
Scienti ﬁc and Philosophical Assessment. The Frontiers Collection. Berlin: Springer.
Orseau, Laurent, and Mark Ring. 2011. “Self-Modi ﬁcation and Mortality in Arti ﬁcial Agents ”.I n
Schmidhuber, Jürgen, Kristinn R. Thórisson, and Moshe Looks, eds. Arti ﬁcial General
Intelligence: 4th International Conference, AGI 2011, Mountain View, CA, USA, August 3 –6,
2011. Proceedings. Lecture Notes in Computer Science 6830. Berlin: Springer, 1 –10.
Persson, Ingmar, and Julian Savulescu. 2008. “The Perils of Cognitive Enhancement and the
Urgent Imperative to Enhance the Moral Character of Humanity ”. Journal of Applied
Philosophy 25(3): 162 –177. doi: 10.1111/j.1468-5930.2008.00410.x .
Persson, Ingmar, and Julian Savulescu. 2012. Un ﬁt for the Future. Oxford: Oxford University
Press. doi: 10.1093/acprof:oso/9780199653645.001.0001 .
Peterson, Nathaniel R., David B. Pisoni, and Richard T. Miyamoto. 2010. “Cochlear Implants and
Spoken Language Processing Abilities: Review and Assessment of the Literature ”. Restorative
Neurology and Neuroscience 28(2): 237 –250. doi: 10.3233/RNN-2010-0535 .
Plaut, David C. 2003. “Connectionist Modeling of Language: Examples and Implications ”.I n
Mind, Brain, and Language: Multidisciplinary Perspectives, edited by Marie T. Banich and
Molly Mack, 143 –168. Mahwah, NJ: Lawrence Erlbaum.
Posner, Richard A. 2004. Catastrophe: Risk and Response. New York: Oxford University Press.Potapov, Alexey, and Sergey Rodionov. 2012. “Universal Empathy and Ethical Bias for Arti ﬁcial
General Intelligence ”. Paper presented at the Fifth Conference on Arti ﬁcial General
Intelligence (AGI –12), Oxford, December 8 –11. Accessed June 27, 2013. http://aideus.com/
research/doc/preprints/04\_paper4\_AGIImpacts12.pdf .
Powers, Thomas M. 2006. “Prospects for a Kantian Machine ”. IEEE Intelligent Systems 21(4):
46–51. doi: 10.1109/MIS.2006.77 .
Pylyshyn, Zenon W., ed. 1987. The Robot ’s Dilemma: The Frame Problem in Arti ﬁcial
Intelligence. Norwood, NJ: Ablex.
Pynadath, David V., and Milind Tambe. 2002. “Revisiting Asimov ’s First Law: A Response to the
Call to Arms ”. In Intelligent Agents VIII: Agent Theories, Architectures, and Languages 8th
International Workshop, ATAL 2001 Seattle, WA, USA, August 1 –3, 2001 Revised Papers,
edited by John-Jules Ch. Meyer and Milind Tambe, 307 –320. Berlin: Springer. doi: 10.1007/3-
540-45448-9\_22 .
Ramamurthy, Uma, Bernard J. Baars, Sidney K. D ’Mello, and Stan Franklin. 2006. “LIDA: A
Working Model of Cognition ”. In Proceedings of the Seventh International Conference on
Cognitive Modeling, edited by Danilo Fum, Fabio Del Missier, and Andrea Stocco, 244 –249.
Trieste, Italy: Edizioni Goliardiche. http://ccrg.cs.memphis.edu/assets/papers/ICCM06-UR.pdf .
Ring, Mark, and Laurent Orseau. 2011. “Delusion, Survival, and Intelligent Agents ”.I n
Schmidhuber, Jürgen, Kristinn R. Thórisson, and Moshe Looks, eds. Arti ﬁcial General
Intelligence: 4th International Conference, AGI 2011, Mountain View, CA, USA, August 3 –6,
2011. Proceedings. Lecture Notes in Computer Science 6830. Berlin: Springer, 11 –20.
Russell, Stuart J. 2015. Will They Make Us Better People? Edge.org. http://edge.org/response-
detail/26157 .3 Responses to the Journey to the Singularity 79
Russell, Stuart J., Dewey, Daniel, Tegmark, Max. 2015. Research priorities for robust and
beneﬁcial arti ﬁcial intelligence. http://futureo ﬂife.org/static/data/documents/research\_priorities.
pdf.
Sandberg, Anders. 2001. “Friendly Superintelligence ”. Accessed December 31, 2012. http://www.
aleph.se/Nada/Extro5/Friendly%20Superintelligence.htm .
Sandberg, Anders. 2012. “Models of a Singularity ”. In Eden, Amnon, Johnny Søraker, James H.
Moor, and Eric Steinhart, eds. Singularity Hypotheses: A Scienti ﬁc and Philosophical
Assessment. The Frontiers Collection. Berlin: Springer.
Sandberg, Anders, and Nick Bostrom. 2008. Whole Brain Emulation: A Roadmap. Technical
Report, 2008-3. Future of Humanity Institute, University of Oxford. http://www.fhi.ox.ac.uk/
wpcontent/uploads/brain-emulation-roadmap-report1.pdf .
Schmidhuber, Jürgen. 2009. “Ultimate Cognition à la Gödel ”. Cognitive Computation 1(2): 177 –
193. doi: 10.1007/s12559-009-9014-y .
Scott, James C. 1998. Seeing Like a State: How Certain Schemes to Improve the Human Condition
Have Failed. New Haven, CT: Yale University Press.
Shanahan, Murray. 2015. The Technological Singularity. MIT Press (forthcoming).Shulman, Carl. 2009. “Arms Control and Intelligence Explosions ”. Paper presented at the 7th
European Conference on Computing and Philosophy (ECAP), Bellaterra, Spain, July 2 –4.
Shulman, Carl. 2010a. Omohundro ’s“Basic AI Drives ”and Catastrophic Risks. The Singularity
Institute, San Francisco, CA. http://intelligence.org/ ﬁles/BasicAIDrives.pdf .
Shulman, Carl. 2010b. Whole Brain Emulation and the Evolution of Superorganisms. The
Singularity Institute, San Francisco, CA. http://intelligence.org/ ﬁles/WBE-Superorgs.pdf .
Snaider, Javier, Ryan Mccall, and Stan Franklin. 2011. “The LIDA Framework as a General Tool
for AGI ”. In Schmidhuber, Jürgen, Kristinn R. Thórisson, and Moshe Looks, eds. Arti ﬁcial
General Intelligence: 4th International Conference, AGI 2011, Mountain View, CA, USA,
August 3 –6, 2011. Proceedings. Lecture Notes in Computer Science 6830. Berlin: Springer,
133–142.
Soares, N., & Benja Fallenstein. 2014. Aligning Superintelligence with Human Interests: A
Technical Research Agenda. Tech. rep. Machine Intelligence Research Institute, 2014. URL:http://intelligence.org/ ﬁles/TechnicalAgenda.pdf .
Sobolewski, Matthias. 2012. “German Cabinet to Agree Tougher Rules on High-Frequency
Trading ”. Reuters, September 25. Accessed December 31, 2012. http://in.reuters.com/article/
2012/09/25/germany-bourse-rules-idINL5E8KP8BK20120925 .
Sotala, Kaj. 2012. “
Advantages of Arti ﬁcial Intelligences, Uploads, and Digital Minds ”.
International Journal of Machine Consciousness 4(1): 275 –291. doi: 10.1142/
S1793843012400161 .
Sotala, Kaj. 2015. Concept learning for safe autonomous AI. In Workshops at the Twenty-Ninth
AAAI Conference on Arti ﬁcial Intelligence.
Sotala, Kaj, and Harri Valpola. 2012. “Coalescing Minds: Brain Uploading-Related Group Mind
Scenarios ”. International Journal of Machine Consciousness 4(1): 293 –312. doi: 10.1142/
S1793843012400173 .
Sotala, Kaj, and Roman V. Yampolskiy. 2013. Responses to catastrophic AGI risk: a survey.
Technical report 2013-2. Berkeley, CA: Machine Intelligence Research Institute.
Sotala, Kaj, and Roman V. Yampolskiy. 2015. Responses to catastrophic AGI risk: a survey.
Physica Scripta, 90(1), 018001.
Spears, Diana F. 2006. “Assuring the Behavior of Adaptive Agents ”. In Agent Technology from a
Formal Perspective, edited by Christopher Rouff, Michael Hinchey, James Rash, Walter
Truszkowski, and Diana F. Gordon-Spears, 227 –257. NASA Monographs in Systems and
Software Engineering. London: Springer. doi: 10.1007/1-84628-271-3\_8 .
Stahl, Bernd Carsten. 2002. “Can a Computer Adhere to the Categorical Imperative? A
Contemplation of the Limits of Transcendental Ethics in IT ”. In, edited by Iva Smit and
George E. Lasker, 13 –18. Vol. 1. Windsor, ON: International Institute for Advanced Studies in
Systems Research/Cybernetics.80 K. Sotala and R. Yampolskiy
Staniford, Stuart, Vern Paxson, and Nicholas Weaver. 2002. “How to 0wn the Internet in Your
Spare Time ”. In Proceedings of the 11th USENIX Security Symposium, edited by Dan Boneh,
149–167. Berkeley, CA: USENIX. http://www.icir.org/vern/papers/cdc-usenix-sec02/ .
Steunebrink, Bas R., and Jürgen Schmidhuber. 2011. “A Family of Gödel Machine Implemen-
tations ”. In Schmidhuber, Jürgen, Kristinn R. Thórisson, and Moshe Looks, eds. Arti ﬁcial
General Intelligence: 4th International Conference, AGI 2011, Mountain View, CA, USA,
August 3 –6, 2011. Proceedings. Lecture Notes in Computer Science 6830. Berlin: Springer,
275–280.
Suber, Peter. 2002. “Saving Machines from Themselves: The Ethics of Deep Self-Modi ﬁcation ”.
Accessed December 31, 2012. http://www.earlham.edu/ ∼peters/writing/selfmod.htm .
Sullins, John P. 2005. “Ethics and Arti ﬁcial life: From Modeling to Moral Agents ”. Ethics &
Information Technology 7 (3): 139 –148. doi: 10.1007/s10676-006-0003-5 .
Tarleton, Nick. 2010. Coherent Extrapolated Volition: A Meta-Level Approach to Machine Ethics.
The Singularity Institute, San Francisco, CA. http://intelligence.org/ ﬁles/CEV-MachineEthics.
pdf.
Tenenbaum, Joshua B., Thomas L. Grif ﬁths, and Charles Kemp. 2006. “Theory-Based Bayesian
Models of Inductive Learning and Reasoning ”.I n“Probabilistic Models of Cognition ”. Special
issue, Trends in Cognitive Sciences 10(7): 309 –318. doi: 10.1016/j.tics.2006.05.009 .
Tenenbaum, Joshua B., Charles Kemp, Thomas L. Grif ﬁths, and Noah D. Goodman. 2011. “How
to grow a mind: Statistics, structure, and abstraction ”. science 331, 6022: 1279 –1285.
Thomas, Michael S. C., and James L. McClelland. 2008. “Connectionist Models of Cognition ”.I n
The Cambridge Handbook of Computational Psychology, edited by Ron Sun, 23 –58.
Cambridge Handbooks in Psychology. New York: Cambridge University Press.
Trope, Yaacov, and Nira Liberman. 2010. “Construal-level Theory of Psychological Distance ”.
Psychological Review 117(2): 440 –463. doi: 10.1037/a0018963 .
Turney, Peter. 1991. “Controlling Super-Intelligent Machines ”. Canadian Arti ﬁcial Intelligence,
July 27, 3 –4, 12, 35.
Tversky, Amos, and Daniel Kahneman. 1981. “The Framing of Decisions and the Psychology of
Choice ”. Science 211 (4481): 453 –458. doi: 10.1126/science.7455683 .
Van Gelder, Timothy. 1995. “What Might Cognition Be, If Not Computation? ”Journal of
Philosophy 92(7): 345 –381. http://www.jstor.org/stable/2941061 .
Van Kleef, Gerben A., Astrid C. Homan, Catrin Finkenauer, Seval Gundemir, and Eftychia
Stamkou. 2011. “Breaking the Rules to Rise to Power: How Norm Violators Gain Power in the
Eyes of Others ”. Social Psychological and Personality Science 2(5): 500 –507. doi: 10.1177/
1948550611398416 .
Van Kleef, Gerben A., Christopher Oveis, Ilmo van der Löwe, Aleksandr LuoKogan, Jennifer
Goetz, and Dacher Keltner. 2008. “Power, Distress, and Compassion: Turning a Blind Eye to
the Suffering of Others ”. Psychological Science 19(12): 1315 –1322. doi: 10.1111/j.1467-9280.
2008.02241.x .
Verdoux, Philippe. 2010. “Risk Mysterianism and Cognitive Boosters ”. Journal of Futures Studies
15 (1): 1 –20. Accessed February 2, 2013. http://www.jfs.tku.edu.tw/15-1/A01.pdf .
Verdoux, Philippe. 2011. “Emerging Technologies and the Future of Philosophy ”. Metaphilos-
ophy 42(5): 682 –707. doi: 10.1111/j.1467-9973.2011.01715.x .
Vinge, Vernor. 1993. “The Coming Technological Singularity: How to Survive in the Post-Human
Era”. In Vision-21: Interdisciplinary Science and Engineering in the Era of Cyberspace, 11 –22.
NASA Conference Publication 10129. NASA Lewis Research Center. http://ntrs.nasa.gov/
archive/nasa/casi.ntrs.nasa.gov/19940022855\_1994022855.pdf .
Walker, Mark. 2008. “Human Extinction and Farsighted Universal Surveillance ”. Working Paper,
September. Accessed December 31, 2012. http://www.nmsu.edu/ ∼philos/documents/sept-
2008-smart-dust- ﬁnal.doc .
Wallach, Wendell. 2010. “Robot Minds and Human Ethics: The Need for a Comprehensive Model
of Moral Decision Making ”.I n“Robot Ethics and Human Ethics, ”edited by Anthony Beavers.
Special issue, Ethics and Information Technology 12(3): 243 –250. doi: 10.1007/s10676-010-
9232-8 .3 Responses to the Journey to the Singularity 81
Wallach, Wendell, and Colin Allen. 2009. Moral Machines: Teaching Robots Right from Wrong.
New York: Oxford University Press. doi: 10.1093/acprof:oso/9780195374049.001.0001 .
Wallach, Wendell, and Colin Allen. 2012. “Framing Robot Arms Control ”. Ethics and Information
Technology. doi:10.1007/ s10676-012-9303-0.
Wang, Pei. 2012. “Motivation Management in AGI Systems ”. In Bach, Joscha, Ben Goertzel, and
Matthew Iklé, eds. Arti ﬁcial General Intelligence: 5th International Conference, AGI 2012,
Oxford, UK, December 8 –11, 2012. Proceedings. Lecture Notes in Arti ﬁcial Intelligence 7716.
New York: Springer. doi: 10.1007/978-3-642-35506-6, 352 –361.
Warwick, Kevin. 1998. In the Mind of the Machine: Breakthrough in Arti ﬁcial Intelligence.
London: Arrow.
Warwick, Kevin. 2003. “Cyborg Morals, Cyborg Values, Cyborg Ethics ”. Ethics and Information
Technology 5(3): 131 –137. doi: 10.1023/B:ETIN.0000006870.65865.cf .
Waser, Mark R. 2008. “Discovering the Foundations of a Universal System of Ethics as a Road to
Safe Arti ﬁcial Intelligence ”. In Biologically Inspired Cognitive Architectures: Papers from the
AAAI Fall Symposium, 195 –200. Technical Report, FS-08-04. AAAI Press, Menlo Park, CA.
http://www.aaai.org/Papers/Symposia/Fall/2008/FS-08-04/FS08-04-049.pdf .
Waser, Mark R. 2009. “A Safe Ethical System for Intelligent Machines ”. In Biologically Inspired
Cognitive Architectures: Papers from the AAAI Fall Symposium, edited by Alexei V.
Samsonovich, 194 –199. Technical Report, FS-09-01. AAAI Press, Menlo Park, CA. http://
aaai.org/ocs/index.php/FSS/FSS09/paper/view/934 .
Waser, Mark R. 2011. “Rational Universal Benevolence: Simpler, Safer, and Wiser than ‘Friendly
AI”’. In Schmidhuber, Jürgen, Kristinn R. Thórisson, and Moshe Looks, eds. Arti ﬁcial General
Intelligence: 4th International Conference, AGI 2011, Mountain View, CA, USA, August 3 –6,
2011. Proceedings. Lecture Notes in Computer Science 6830. Berlin: Springer, 153 –162.
Weld, Daniel, and Oren Etzioni. 1994. “The First Law of Robotics (A Call to Arms) ”.I n
Proceedings of the Twelfth National Conference on Arti ﬁcial Intelligence, edited by Barbara
Hayes-Roth and Richard E. Korf, 1042 –1047. Menlo Park, CA: AAAI Press. http://www.aaai.
org/Papers/AAAI/1994/AAAI94-160.pdf .
Weng, Yueh-Hsuan, Chien-Hsun Chen, and Chuen-Tsai Sun. 2008. “Safety Intelligence and Legal
Machine Language: Do We Need the Three Laws of Robotics? ”In Service Robot
Applications, edited by Yoshihiko Takahashi. InTech. doi: 10.5772/6057 .
Weng, Yueh-Hsuan, Chien-Hsun Chen, and Chuen-Tsai Sun. 2009. “Toward the Human –Robot
Coexistence Society: On Safety Intelligence for Next Generation Robots ”. International Journal
of Social Robotics 1(4): 267 –282. doi: 10.1007/s12369-009-0019-1 .
Whitby, Blay. 1996. Re ﬂections on Arti ﬁcial Intelligence: The Legal, Moral, and Ethical
Dimensions. Exeter, UK: Intellect Books.
Whitby, Blay, and Kane Oliver. 2000. “How to Avoid a Robot Takeover: Political and Ethical
Choices in the Design and Introduction of Intelligent Artifacts ”. Paper presented at Symposium
on Arti ﬁcial Intelligence, Ethics and (Quasi-) Human Rights at AISB-00, University of
Birmingham, England. http://www.sussex.ac.uk/Users/blayw/BlayAISB00.html .
Wilson, Grant. 2013. Minimizing global catastrophic and existential risks from emerging
technologies through international law. Va. Envtl. LJ, 31, 307.
Wood, David Murakami, and Kirstie Ball, eds. 2006. A Report on the Surveillance Society: For
the Information Commissioner, by the Surveillance Studies Network. Wilmslow, UK: Of ﬁce of
the Information Commissioner, September. http://www.ico.org.uk/about\_us/research/ ∼/media/
documents/library/Data\_Protection/Practical\_application/SURVEILLANCE\_SOCIETY\_
SUMMARY\_06.ashx .
Yampolskiy, Roman V. 2012. “Leakproo ﬁng the Singularity: Arti ﬁcial Intelligence Con ﬁnement
Problem ”. Journal of Consciousness Studies 2012(1 –2): 194 –214. http://www.ingentaconnect.
com/content/imp/jcs/2012/00000019/F0020001/art00014 .
Yampolskiy, Roman V. 2013. “What to Do with the Singularity Paradox? ”Studies in Applied
Philosophy, Epistemology and Rational Ethics vol 5, pp. 397 –413. Springer Berlin Heidelberg.
Yampolskiy, Roman V., and Joshua Fox. 2012. “Safety Engineering for Arti ﬁcial General
Intelligence ”. Topoi. doi: 10.1007/s11245-012-9128-9 .82 K. Sotala and R. Yampolskiy
Yudkowsky, Eliezer. 2001. Creating Friendly AI 1.0: The Analysis and Design of Benevolent
Goal Architectures. The Singularity Institute, San Francisco, CA, June 15. http://intelligence.
org/ﬁles/CFAI.pdf .
Yudkowsky, Eliezer. 2004. Coherent Extrapolated Volition. The Singularity Institute, San
Francisco, CA, May. http://intelligence.org/ ﬁles/CEV.pdf .
Yudkowsky, Eliezer. 2008. “Artiﬁcial Intelligence as a Positive and Negative Factor in Global
Risk”. In Bostrom, Nick, and Milan M. Ćirkovi ć, eds. Global Catastrophic Risks. New York:
Oxford University Press, 308 –345.
Yudkowsky, Eliezer. 2011. Complex Value Systems are Required to Realize Valuable Futures.
The Singularity Institute, San Francisco, CA. http://intelligence.org/ ﬁles/ComplexValues.pdf .
Yudkowsky, Eliezer. 2012. “Reply to Holden on ‘Tool AI ”’. Less Wrong (blog), June 12. http://
lesswrong.com/lw/cze/reply\_to\_holden\_on\_tool\_ai/ .3 Responses to the Journey to the Singularity 83