Introduction to the technological singularity

Chapter 1
Introduction to the Technological
Singularity
Stuart Armstrong
The term “technological singularity ”has been much used and abused over the
years. It ’s clear that it ’s connected with the creation of arti ﬁcial intelligences (AIs),
machines with cognitive skills that rival or surpass those of humanity. But there is
little consensus or consistency beyond that. In a narrow sense it can refer to AIs
being capable of recursive self-improvement: of redesigning themselves to be morecapable, and using that improved capability to further redesign themselves, and soon. This might lead to an intelligence explosion, a “singularity ”of capability, with
the AIs rapidly surpassing human understanding and slipping beyond our control.But the term has been used in a variety of other ways, from simply denoting abreakdown in our ability to predict anything after AIs are created, through tomessianic and quasi-religious visions of a world empowered by super-machines
(Sandberg ’s2010 , has identi ﬁed nine common de ﬁnition, corresponding to eigh-
teen basic models).
It is in the simplest sense, though, that the term is the most useful. A singularity
in a model —a breakdown of our ability to predict beyond that point. It need not
mean that the world goes crazy, or even that the model does. But it does mean thatour standard tools become inadequate for understanding and shaping what comesafter. New tools are needed.
In a sense, the prevalence of robots and AI on our TV and computer screens have
made us lessprepared for the arrival of true AIs. Human ﬁction is written by
humans —for the moment, at least —for humans, about human concerns, and with
human morality and messages woven into them. Consider the potential for AIs orrobots to take over human jobs. In ﬁction, the focus is variously on hard-working
humans dispossessed by the AIs, on humans coping well or badly with the extraleisure (with a moral message thrown in one way or the other), or, maybe, on therobots themselves, portrayed as human-equivalents, suffering as slaves. On top of
S. Armstrong ( ✉)
Future of Humanity Institute, Oxford University, Oxford, UK
e-mail: stuart.armstrong@philosophy.ox.ac.uk
© Springer-Verlag GmbH Germany 2017
V. Callaghan et al. (eds.), The Technological Singularity ,
The Frontiers Collection, DOI 10.1007/978-3-662-54033-6\_11
that, all the humans (or human-equivalent robots) typically have the exact same
cultural values as the writers of the work.
Contrast that with a more mechanical or analytic interpretation of the problem:
what will happen when human capital (what we used to call “employees ”) can be
mass produced as needed for any purpose by companies? Or by political parties, by
charities, by think tanks, by surveillance organisations, by supermarkets, by parkingservices, by publishers, by travel agencies? When everything in the world that iscurrently limited by human focus and attention —which is most things, today —
suddenly becomes a thousand times easier. Every moment of every day would betransformed, even something as simple as a lock: why bother with a key if apersonal assistant can wait alongside or inside the door, opening as needed (andonly as needed). On top of that, add the shock of (some? most? all?) jobs getting
destroyed at the same time as there is a huge surge in manufacturing, services, and
production of all kind. Add further the social dislocations and political responses,and the social and cultural transformations that will inevitably follow. It is clear thatindustry, economy, entertainment, society, culture, politics, and people ’s very
conception of themselves and their roles, could be completely transformed by thearrival of AIs.
Though the point should not be belaboured (and there are certainly uncertainties
and doubts as to how exactly things might play out) it is clear that our ﬁction and
our conception of AIs as “yet another technology ”could leave us woefully
unprepared for how transformative they could be. Thus, “singularity ”is not an
inappropriate term: a reminder of the possible magnitude of the change.
This book is the second in a series looking at such an AI-empowered singularity.
Theﬁrst,“Singularity Hypotheses: A Scienti ﬁc and Philosophical Assessment ”
looked squarely at the concept, analysing its features and likelihood from a varietyof technical and philosophical perspectives. This book, in contrast, is focusedsquarely on managing the AI transition, how to best deal with the wondrous pos-
sibilities and extreme risks it may entail.
If we are to have a singularity, we want to have the best singularity we can.
1.1 Why the “Singularity ”Is Important
Though the concept of machine intelligence is fascinating both from a storytelling
and a philosophical point of view, it is the practical implications that are of highest
importance in preparing for it. Some of these implications have already been hintedat: the possibility of replacing “human capital ”(workers and employees) with
AI-powered versions of this. While humans require a long, 18+ years of maturationto reach capability, and can perform one task at a time, an AI, once trained, could becopied at will to perform almost any task. The fallibilities of the human condition —
our limited memories, our loss of skills due to ageing, our limited short termmemories, our slow thinking speeds on many dif ﬁcult and important problems, and
so on —need not concern AIs. Indeed, AIs would follow a radically different2 S. Armstrong
learning cycle than humans: different humans must learn the same skill individu-
ally, while skills learnt by an AI could then become available for every AI toperform at comparable ability —just as once the basic calculator was invented, all
computers could perform superhuman feats of multiplication from then on.
This issue, though not exactly prominent, has already been addressed in many
areas, with suggestions ranging from a merging of humans and AIs (just assmartphone or cars are an extension of many people ’s brains today), through to
arguments about AIs complementing rather than substituting for human labour, andradical political ideas to use the AI-generated surplus for guaranteeing a basicincome to everyone. In a sense, the arguments are nothing new, variants of theperennial debates about technological unemployment. What is often underesti-mated, though, is the speed with which the change could happen. Precisely because
AIs could have general intelligence —therefore the ability to adapt to new situations
—they could be rapidly used in jobs of all varieties and descriptions, with whole
categories of human professions vanishing every time the AI discovered a newskill.
But those issues, though interesting, pale in comparison with the possibility of
AIs becoming superintelligent. That is the real transformative opportunity/risk. Butwhat do we mean exactly by this superintelligence?
1.2 Superintelligence, Superpowers
One could de ﬁne intelligence as the ability to achieve goals across a wide variety of
different domains. This de ﬁnition naturally includes the creativity, ﬂexibility, and
learning needed to achieve these goals, and if there are other intelligent agents inthose domains, it naturally also includes understanding, negotiating with, and
manipulating other agents.
What would a “super ”version of that mean? Well, that the agent has a
“super-ability ”to achieve its goals. This is close to being a tautology —the winning
agent is the agent that wins —so exploring the concept generally involves mining
analogies for insight. One way to do so would be to look down: chimps anddolphins are quite intelligent, yet they have not constructed machinery, cities, orrockets, and are indeed currently dependent on human goodwill for their continuedsurvival. Therefore it seems possible that intelligences as far “above ”us as we are
above chimps or cows, would be able to dominate us more completely than we
dominate them.
We could imagine such an intelligence simply by endowing it with excellent
predictive powers. With good predictions, draining all money from the stockmar-kets becomes trivial. An agent could always know exactly how much risk to take,exactly who would win any debate, war, or election, exactly what words would bethe most likely to convince humans of its goals, and so on. Perfect prediction wouldmake such an AI irresistible; any objections along the lines of “but wouldn ’t people
do X to stop it ”runs up against the objection of “the AI could predict if people1 Introduction to the Technological Singularity 3
would try X, and would either stop X or ﬁnd an approach so people wouldn ’t want
to do X ”.
That superpredictor is almost certainly fanciful. Chaos is a very real factor in the
world, reducing the ability of any entity to accurately predict everything. But an AI
need not be able to predict everything , just to be able to predict enough. The stock
market, for instance, might be chaotic to our human eyes, and might be mainlychaotic in reality, but there might be enough patterns for an AI to exploit. Thereforethe concept of superintelligence is really an entwining of two ideas: that the AIcould be much better than us at certain skills, and that those skills could lead togreat abilities and power the world. So the question is whether there is a dimin-ishing return to intelligence (meaning that humans will probably soon plumb thelimits of what ’s possible), or is its potential impact extreme? And if the return to
intelligence is limited, is that true both in the short and the long terms?
We cannot be sure at this point, because these questions are precisely about areas
where humans have dif ﬁculty reasoning —if we knew what we would do with
superintelligence, we ’d do that already. Some have argued that, in a certain sense,
humans are as intelligent as possible (just as all Turing-complete agents areequivalent), and that the only difference between agents is time, resources, andimplementation details. Therefore, there is no further leap in intelligence possible:everything is on a smooth continuum.
Though that argument sounds super ﬁcially plausible, it can be undermined by
turning it on its head and looking back into the past. It might be that there is anupper limit on social intelligence —maybe the most convincing demagogue or con
man wouldn ’t be much better than what humans can achieve today. But scienti ﬁc
and technological intelligence seem different. It seems perfectly plausible for asuperintelligence in 1880 to develop airplanes, car, effective mass production,possibly nuclear weapons and mass telecommunications. With skillful deployment,these could have made it superpowered at that time. Do we think that our era is
somehow more resistant? Even if the curve of intelligence is smooth, it doesn ’t
mean the curve of impact is.
Another analogy for superintelligence, created by this author, is of a super-
committee. Consisting of AIs trained to top human level in a variety of skills (e.g.Einstein-level for scienti ﬁc theorising, Bill Clinton-level for social interactions),
and networked together perfectly, run at superhuman speeds, with access to allresearch published today in many domains. It seems likely to accumulate vastamounts of power. Even if it is limited over the short term (though recall its
potential to be copied into most jobs currently done by humans), it ’s long-term
planning could be excellent, slowly and inexorably driving the world in thedirection it desires.
That is a potentially superintelligent entity constructed from human-level intel-
ligences. Most of the other paths to superintelligence involves the AIs turning theirintelligence towards developing their own software and hardware, and henceachieving a recursive feedback loop, boosting their abilities to ever greater levels (issuch a feedback even loop possible? Add that question to the pile of unknowns).
There is, however, another possibility for superintelligence: it is possible that the4 S. Armstrong
range of human intelligences, that seems so broad to us, is actually fairly narrow. In
this view, humans are products of evolution, with some thin rational ability layeredon top of a mind developed mainly for moving about in the savannah and dealingwith tribal politics. And a small part of that rational ability was later co-opted by
culture for such tasks as mathematics and scienti ﬁc reasoning, something humans
would be very bad at in any objective sense. Potentially, the AIs could develop at asteady pace, but move from being dumber than almost all humans, to radicallysmarter than everyone, in the space of a single small upgrade.
Finally, though we ’ve been describing an AI becoming “superpowered ”via
being superintelligent, this need not be the case. There are ways that a relatively“dumb ”AI could become superpowered. Various scenarios can be imagined, for an
AI to dominate human society as it currently stands. For instance, a small AI with
the ability to hack could take over a computer, copy itself there, take over two
computers from its current basis, copy themselves there, and so on, thus taking overmost of the internet in short order (a smart botnet). Alternately, there are ways thatcertain narrow abilities could lend themselves to AI takeover, such as weapondesign and manufacturing (especially biological weapons such as targeted pan-demics). A more thorough look at potential AI superpower can be found in NickBostrom ’s book ( 2014 ).
It seems therefore that there is a potential for extremely superintelligent AIs
being developed, following soon after AIs of “human-level ”intelligence. Whether
or not this is possible depends on certain facts about the nature of intelligence, thepotential return on new knowledge, and the vulnerabilities of human society, thatwe just can ’t know at the present.
1.3 Danger, Danger!
Of course, just because an AI could become extremely powerful, does not mean that
it need be dangerous. Ability and morality are not correlated in humans (and evenless so in the alien mind of an AI), so the AI could be extremely powerful whilebeing extremely well-intentioned.
There are arguments to doubt that happy picture, however. Those arguments are
presented mainly in the preceding book (Eden et al. 2012 ), and in other works such
as Bostrom ( 2014 ) and Armstrong ( 2014 ), so the point won ’t be belaboured here
too much. The risk is not that the AI could, by luck or design, end up “evil”in the
human sense. The risk is rather that the AI ’s goals, while harmless initially, become
dangerous when the AI becomes powerful. In fact, most goals are dangerous at highpower. Consider the trivial example of a spam ﬁlter that becomes superintelligent.
Its task is to cut down on the number of spam messages that people receive. Withgreat power, one solution to this requirement is to arrange to have all spammerskilled. Or to shut down the internet. Or to have everyone killed. Or imagine an AIdedicated to increasing human happiness, as measured by the results of surveys, or
by some biochemical marker in their brain. The most ef ﬁcient way of doing this is1 Introduction to the Technological Singularity 5
to publicly execute anyone who marks themselves as unhappy on their survey, or to
forcibly inject everyone with that biochemical marker. We can ’t be sloppy when it
comes to AI goal design —if we want “happiness ”, we have to fully unambiguously
deﬁne it.
This is a general feature of AI motivations: goals that seem safe for a weak or
controlled AI, can lead to extremely pathological behaviour if the AI becomespowerful. As the AI gains in power, it becomes more and more important that itsgoals be fully compatible with human ﬂourishing, or the AI could enact a patho-
logical solution rather than one that we intended. Humans don ’t expect this kind of
behaviour, because our goals include a lot of implicit information, and we take“ﬁlter out the spam ”to include “and don ’t kill everyone in the world ”, without
having to articulate it. But the AI might be an extremely alien mind: we cannot
anthropomorphise it, or expect it to interpret things the way we would. We have to
articulate all the implicit limitations. Which may mean coming up with a solutionto, say, human value and ﬂourishing —a task philosophers have been failing at for
millennia —and cast it unambiguously and without error into computer code.
Note that the AI may have a perfect understanding that when we programmed in
“ﬁlter out the spam ”, we implicitly meant “don’t kill everyone in the world ”. But
the AI has no motivation to go along with the spirit of the law: its goals are the letteronly, the bit we actually programmed into it. Another worrying feature is that the AI
would be motivated to hide its pathological tendencies as long as it is weak, and
assure us that all was well, through anything it says or does. This is because it willnever be able to achieve its goals if it is turned off, so it must lie and play nice to getanywhere. Only when we can no longer control it, would it be willing to act openlyon its true goals —we so we better have made those safe.
1.4 Uncertainties and Safety
The analogies and arguments of the preceding sections are somewhat frustrating. “It
seems ”,“possibly ”,“it might ”,“maybe ”, and so on. Why is there so much
uncertainty and so many caveats? Is everyone so vague about what AIs could be ordo?
Certainly not. Vagueness is not the norm for AI predictions. Instead, the norm is
for predictors to be bold, dramatic, con ﬁdent—and wrong. We can comfortably say
that most AI predictions are wrong, simply because they contradict each other. For
a simple taste, see the following graph showing a variety of predictions for the datesat which we might have AI (for more details on this data see “AI timeline pre-
dictions: are we getting better? ”, see Chap. 14in this book):6 S. Armstrong
As can be seen, the predictions are all over the place, not showing any particular
consensus, with wide intervals between them, and a few of the dates already passed.Digging into the details of the predictions —and the weak arguments that underpins
the strong con ﬁdence —paints an even more dismal picture.
But, stepping back a bit, why should we expect anyone to be accurate about
predicting AI? We don ’t have an AI. We don ’t know how to build one. We don ’t
know what features it would have if it were built. We don ’t know whether con-
sciousness, intuitions, and emotions would be necessary for an AI (also, we don ’t
know what consciousness is). We don ’t know if it would need to be physically
embodied, how fast it could run, whether it would make new discoveries at thespeed of thought or at the slower speed of experiment, and so on. If predictions in
politics are so poor —and they are, as Tetlock ( 2005 ) has demonstrated —and
politics involve very understandable human-centric processes, why would weexpect predictions in AI to be any better? We are indeed trying to predict a futuretechnology, dependent on future unknown algorithms and the solutions to wholehost of problems we can ’t yet even see.
There actually has been quite a bit of work on the quality of expert predictions.
A lot of different analysis have been done, by such people as Shanteau et al. ( 2002 ),
Khaneman ( 2011 ), Cooke ( 2004 ), Klein ( 1997 ), and others (one disturbing feature
of the expertise literature is how little the various researchers seem to know about
each other). And AI predictions have none of the features that would predict goodpredictions; to use Shanteau ’s table on what predicts good or bad expert perfor-
mance, with the features of AI timeline predictions marked in red:1 Introduction to the Technological Singularity 7
Feedback, the most important component in good expert prediction, is almost
completely absent (immediate feedback, the crown jewel of expert competence, iscompletely absent). Experts disagree strongly on AI dates and the formats of AI, aswe’ve seen. Finally, though better predictions are made by decomposing the
problem, few predictors do so (possibly because of the magnitude of the problem).
A big warning here: it is easy to fall into the rhetorical trap of thinking “if it’s
uncertain, then we don ’tn e e dt ow o r r ya b o u ti t ”.T h a t “argument ”, or variants of it, is
often used to dismiss concerns about AI, even by very intelligent critics. Considering theargument deployed in other areas is enough to illustrate its weakness: “the enemy army
could come down the right pass or the left one, we really don ’t know, so let ’s not worry
about either ”,o r“the virus you caught may or may not be infectious, I wouldn ’tw o r r y
about it if I were you. ”In fact, claiming that AIs are safe or impossible (which is what
“don’tw o r r y ”amounts to) is a very con ﬁdent and speci ﬁc prediction about the future of
AI development. Hence, almost certainly wrong. Uncertainty is not safety ’sa l l y .
References
Armstrong, Stuart. Smarter than us: The rise of machine intelligence. MIRI, 2014.
Bostrom, Nick. Superintelligence: Paths, dangers, strategies. OUP Oxford, 2014.Cooke, Roger M., and Louis HJ Goossens. “Expert judgement elicitation for risk assessments of
critical infrastructures. ”Journal of risk research 7.6 (2004): 643 –656.
Eden, Amnon H., Eric Steinhart, David Pearce, and James H. Moor. “Singularity hypotheses: an
overview. ”In Singularity Hypotheses, pp. 1 –12. Springer Berlin Heidelberg, 2012. Harvard
Kahneman, Daniel. Thinking, fast and slow. Macmillan, 2011.
Klein, Gary. “The recognition-primed decision (RPD) model: Looking back, looking forward. ”
Naturalistic decision making (1997): 285 –292.
Sandberg, Anders. “An overview of models of technological singularity. ”Roadmaps to AGI and
the Future of AGI Workshop, Lugano, Switzerland, March. Vol. 8. 2010.
Shanteau, James, et al. “Performance-based assessment of expertise: How to decide if someone is
an expert or not. ”European Journal of Operational Research 136.2 (2002): 253 –263.
Tetlock, Philip. Expert political judgment: How good is it? How can we know?. Princeton
University Press, 2005.
8 S. Armstrong


Can the Singularity Be Patented? (And Other IP Conundrums for Converging Technologies)

Chapter 10
Can the Singularity Be Patented?
(And Other IP Conundrumsfor Converging Technologies)
David Koepsell
10.1 Introduction
Many of the societal controls and policy considerations discussed in relation to the
development of “The Singularity ”or arti ﬁcial general intelligence (AGI) relate to
potential harms and risks. There is one important area, however, where current
legislation and policies pose a risk to its development: the laws of intellectualproperty. Intellectual property (IP) is now taken for granted as somehow necessaryfor innovation, but it poses a considerable threat to both the eventual developmentof the Singularity, and to our rights, duties, and obligations once such a Singularity
emerges. If we are to properly anticipate and guide the development of AGI in
positive ways, we must come to terms with the role that IP plays in either promotingor hindering innovation, the impact it will have on AGIs as well as their developers,and manners in which might better manage IP to allay some of the anticipated risks.
Many people credit the development of the legal institution we call “intellectual
property ”with helping to propel us into the modern, technological age. Before
intellectual property (IP) was invented, there was no means available for people toprevent others from utilizing their ideas, aside from force. This is because ideas
cannot ordinarily be monopolized, except by secret-keeping, and even then they are
prone to independent discovery or thought. Because of the inherent uncontainabilityof ideas, states have, within the past couple hundred years, devised legal monop-olies over expressions in various forms under the belief that such monopolies wouldhelp encourage innovation and immigration of highly skilled and inventive people.
D. Koepsell ( ✉)
Research and Strategic Initiatives, COMISIÓN NACIONAL DE BIOÉTICA,
Mexico city, Mexico
e-mail: drkoepsell@gmail.com
URL: http://davidkoepsell.com
D. Koepsell
Universidad Autonoma Metropolitan, Xochimilco, Mexico
© Springer-Verlag GmbH Germany 2017
V. Callaghan et al. (eds.), The Technological Singularity ,
The Frontiers Collection, DOI 10.1007/978-3-662-54033-6\_10181
Theﬁrst such monopolies were “letters patent ”issued by sovereigns to those who
devised new and useful arts. These letters patent entitled their holders to monop-olize a market for a term of years. Over time, various states began to formalize thesesorts of processes, and introduced also monopolies for artistic works, believing that
these incentives would help to encourage technological and economic growth.
Modern IP includes copyrights, patents, and trademarks. Each works slightly
differently, and covers speci ﬁc types of expressions. Patents are monopolies granted
by the state to inventors of new, useful, and nonobvious utilitarian objects. Themonopoly term for patents runs 20 years, after which the invention lapses againinto the public domain. The copyright term, which covers aesthetic expressions,lasts the lifetime of the author plus an additional 70 years. Trademarks cover tradenames, and fall outside of this discussion. Our interests here focus on the effects of
both patents and copyright on the emergence and promise of the singularity, as it
has been hypothesized by Ray Kurzweil and others. Both copyrights and patents arepertinent due to some oddities in the law of IP that I began to explore nearly20 years ago, and which remain unresolved. But more to the point, patents are theprimary threat, at least in their current forms, to both the emergence of the sin-gularity and its achieving its full promise.
10.2 A Singular Promise
The singularity can be understood in numerous ways, and people more technicallyinclined have devised excellent explications of its potential nature. For our pur-poses, let us examine it in its most general possible forms and then discuss howvarious legal institutions such as copyrights and patents may impact on its eventualachievement. The technological singularity will involve a quantum leap in our
technologies in such a way that society itself will change. This could happen
through the realization of true arti ﬁcial intelligence, which, if it can be created, will
in all likelihood be unbounded by the biological limitations imposed upon our ownintelligences. The singularity may also be achieved by way of nanotechnology orsome other radical new approach to our material world such that objects can beprogrammed: our whole physical environment could be alterable either by our-selves, or merged with our arti ﬁcial intelligences. Finally, the singularity may
involve some transhumanist future in which we ourselves become the objects of our
technological change, capable of super-intelligence, strength, or longevity …or
even immortality. The singularity may involve some combination of all of these,either to completion or partially realized. In any event, any or all of these tech-nological achievements will likely usher in radical challenges to our present cul-tures and institutions. How we manage our current institutions, including laws, willlikely have some impact too on the eventual realization of the singularity, and howit may then affect us once achieved, as we shall see.182 D. Koepsell
Technological advances have necessarily wrought signi ﬁcant re ﬂection, if not
always rapid change, in the ways that we consider the roles of law in the ﬁeld of
innovation. Speci ﬁcally, of late there has been some shifting of approaches to
innovation and how best to encourage and protect investments. Open Source is now
a serious option for even the biggest, most established companies seeking to create
economic climates more suitable to commercial exploitation of some niches. This isperhaps because legal monopolies may discourage cooperation and competition,both of which may simultaneously be viable and pro ﬁtable means of building a
nascent technology ’s acceptance and success. Even early in the growth of the
automobile industry, competing companies realized that legal monopolies couldstiﬂe the development of their technologies, and so entered into “patent pools ”and
other cooperative means to prevent “patent thickets ”and encourage competition
and the technological growth it can promote. The technological singularity, if it is
underway, or if it is still far off on the horizon, will likely lead to similar legal,social, and institutional innovations if the current law of IP threatens to sti ﬂes its
achievement, as I suggest below.
10.3 Intellectual Property
I have spent the better part of 15 years criticizing IP law in its current form for avariety of reasons, some pragmatic, and others essential. The law as it standsembodies a very confused metaphysics (by metaphysics, I mean the manner inwhich we classify its objects), and increasingly these metaphysical errors lead toabsurd results. I will discuss my general issues with IP law brie ﬂy, and apply them
to two particular issues I see as arising from IP ’s odd metaphysics and application
speciﬁcally for the singularity hypothesis, namely: (1) does patent law as it now
stands allow the patenting of arti ﬁcial intelligences, and is this an ethical or practical
problem, and (2) will IP protection apply to the creative products of arti ﬁcial
intelligences, and should it? These two complex but unresolved issues in the law ofIP pose real potential impediments to its realization in the near future.
10.3.1 Some General IP Problems in Converging
Technologies
I have written at some length about some of the practical and theoretical oddities
and problems associated with IP law, some of which are particularly noticeable withthe advent of digital and bio-technologies in the past few decades. I summed muchof this up in my monograph, Innovation and Nanotechnology: Converging Tech-
nologies and the End of Intellectual Property . There I concluded that the
train-wreck of IP, which began with the courts and patent of ﬁces granting10 Can the Singularity Be Patented? … 183
simultaneous patents and copyrights (two, previously mutually-exclusive cate-
gories) to software, was culminating in the present innovative climate with amandate for innovators to simply ignore and work around IP law entirely. This isbecause the courts and legislators who have drafted the current IP laws have
interpreted them in ways that, while they might please IP lawyers and monopolists
who have the means to acquire vast patent portfolios and thus venture capital (orworse, litigation awards), do not understand the underlying science or metaphysicsof artifacts and nature. The result has been a slow usurpation of what many call the“scienti ﬁc commons ”and its gradual conversion into private property. This is
especially problematic, I argue, for developing the technologies that will lead to thesingularity.
Over time, the categories that originally bounded the objects of copyright and
patent have been revealed to be ﬂawed. Copyright was originally applied to
“non-utilitarian ”expressions, like works of art. Patents have been applied to
“utilitarian ”expressions, namely “Whoever invents or discovers any new and useful
process, machine, manufacture, or composition of matter, or any new and usefulimprovement thereof, may obtain a patent therefor, subject to the conditions andrequirements of this title. ”These criteria have been interpreted over time to apply to
nearly anything, including elements on the periodic table, business methods, soft-ware, algorithms, genes, and other seemingly problematic sorts of things have been
granted patents that have been upheld. Even lifeforms may be patented, as long as
they are not human, and as long as they have been “engineered ”through some
human intervention. Three explicit exclusions (other than the statutory exclusion ofpeople) created by famous court cases and adopted by patent of ﬁces more or less
around the world include: abstract ideas, products of nature, and natural phenom-ena. While these exclusions should have seemed more or less obvious under theoriginal conceptions of patent law (which was to promote invention of new anduseful things) the catch-all type of language in the US Patent Act ’s section 101
quoted above, and applied through various treaties to patents world-wide, have
resulted in a blurring of the lines delineating nature from artifact, and courts haveattempted through these exclusions to redraw those lines.
Unfortunately, the exclusions sought to be clari ﬁed have been further muddied
through poor metaphysics in courts and patent of ﬁces. What counts as a “product of
nature, ”for instance, is now more or less nothing. For instance, if you want to
patent a product of nature, like the molecule O
2(gaseous oxygen, created through
such natural processes as photosynthesis) just come up with some man-made way
to synthesize it. Thus, if I discover that I can concentrate O 2through electrolysis of
water, separating the O 2from the H 2in it, then under the current interpretation of
the patent laws in the US and Europe, I can patent not just these new processes, butthe products themselves. Suddenly, O
2is an invention. All of which poses a real
problem in nanotechnology, and converging technologies in general, because as thescales of manufacturing new and useful things shrink, the building blocks com-prising new products as the singularity nears will be at the molecular level.184 D. Koepsell
Consider the patent thickets that have emerged in the cell phone market, and now
multiply by hundreds. This will slow down the convergence of technologies likenanotechnology and synthetic biology, and hinder the emergence of the singularity.
Indeed the law of IP seems to be reaching a crisis point as technologies converge
upon the singularity. Written expressions and machines appear to be converging,
blurring the line between aesthetic and utilitarian expressions in general. Biologyand manufactures are converging, blurring the lines between artifacts and organ-isms. And the scales at which the building blocks of new artifacts are created arebecoming smaller, making fundamental elements, deemed at one time to beproducts of nature, to be the medium for manufacturing nearly everything. Beyondthese conundrums, however, lurk even more insidious ethical issues when (orshould) the singularity emerge. What, for instance, will we do about patenting
machines (or software) when they acquire sentience, and will we apply IP to the
products of their imaginations?
10.3.2 Some Gaps in IP Relating to the Singularity
Before the twentieth century, there was no question about it, life forms could not be
patented. This was largely because of the fact that technology had not yet advanced
to the state of being able to consciously create new, non-obvious, and useful
lifeforms. In 1930, the ﬁrst US Plant Patent Act changed all that. Lobbied for by
plant breeders as well as the likes of Thomas Edison himself, the Plant Patent Actprovided the ﬁrst monopoly protection under the Patent Act: 35 U.S.C. section 161:
“Whoever invents or discovers and asexually reproduces any distinct and new
variety of plant, including cultivated sports, mutants, hybrids, and newly foundseedlings, other than a tuber propagated plant or a plant found in an uncultivated
state, may obtain a patent therefor, subject to the conditions and requirements of
this title. ”This was the ﬁrst legislation to recognize and grant state-monopolies to
living creatures. And why not? If the patent law states explicitly that any new,useful, and non-obvious composition of matter or manufacture should be able to bepatented, then there ’s no particular reason embedded in these criteria to exclude
lifeforms of any kind. Still, it wasn ’t until 1980 that the US Supreme Court
extended this reasoning to non-plant life, in the seminal case of Diamond v.
Chakrabarty .
Chakrabarty had“engineered ”through selective breeding a bacterium that could
digest petroleum, which would certainly be a useful new, non-obvious invention,helpful for cleaning up oil spills, etc. but the US Patent and Trademark Of ﬁce
refused to grant a patent for the organism citing no previous history, and thusimplicit policy against patenting life. But the Supreme Court disagreed, and foundthe new organism patent-eligible, while stating that there were still some explicitexclusions to patent-eligibility, including: abstract ideas, products of nature, andnatural phenomena. Interesting for the singularity hypothesis are the following: to10 Can the Singularity Be Patented? … 185
what extent might the building-blocks of converging technologies fall under one of
theChakrabarthy exclusions, and should all engineered life-forms be patent-
eligible?
One glaring gap in the law as it relates to converging technologies remains in the
deﬁnitions of “abstract ideas, ”“products of nature, ”and“natural phenomena. ”
Vagueness in the categories will continue to effect innovation in nanotechnologyand arti ﬁcial intelligence. The singularity, if it is technically achievable, will break
down barriers among these categories, requiring us to reevaluate philosophicallyand practically what counts as abstract ideas, products of nature, and natural phe-nomena. Will arti ﬁcial intelligences exhibit properties of nature, even if they are
artiﬁcially produced. Will evolutionary forces, perhaps directing their future forms,
require us to consider them to be under the guidance and direction of natural
phenomena? If so, then do current IP exemptions mean that we cannot claim IP
rights over future products? If life was a big exception under IP law prior toChakrabarthy , and human life remains so now, what are the necessary and suf ﬁ-
cient features of life that make it excludable?
We can deduce from the con ﬂict between the statutory exclusion of human life
from patentability, and the acceptance of other lifeforms as patentable afterChakrabarthy , that something special about humans prohibits owning them through
IP. Of course, constitutional prohibitions against slavery should cover some forms
of human ownership, preventing owning any one human, but extending this notion
to IP con ﬂates IP and ordinary property to an unfair extent. Simply put: IP does not
confer ownership rights over tokens. That is to say, a patent holder doesn ’t have an
ownership interest in any instance of his or her invention. Rather, they own the
right to exclude others from pro ﬁting from the manufacture and ﬁrst sale of the
thing over which they hold IP rights. So IP ownership over engineered orpartially-engineered humans ought not to be prohibited by the same rationale thatprohibits slavery.
So what values would rationally prohibit ownership over types of some technical
artefacts or processes and not others? One answer may lie in deontology and someKantian concern over the effects of claims to ownership of types regarding sentientcreatures as opposed to others. Are we entitled to uselifeforms as instrumentalities,
or means to particular ends, or must we respect them (or at least some of them) aspossessing certain inherent dignities which IP rights might offend? On its face, weclearly reject an expansive view of this thesis, since there are numerous lifeforms,even arguably “sentient ”ones that we feel free to exploit as instrumentalities to
varying degrees.
10.4 Limits to Ownership and Other Monopolies
Not everything is susceptible to just claims of monopoly or other control. Some
limits to what may be owned are moral, and others are practical. Still others are
legal creations, designed to achieve certain utilitarian ends. Most recently, the186 D. Koepsell
Supreme Court in the U.S. has attempted to de ﬁne better some existing limits to
what could be claimed by patent. The Abstract Idea ,Natural Phenomena , and Laws
of Nature exceptions have been grappled with in the Bilski ,Mayo , and Myriad cases
between 2010 and 2014. First we will consider each of these exceptions and recent
cases brie ﬂy, then examine their effect, if any, on singularity technology.
Intellectual property law was created in order to encourage and advance the
march of the “useful arts. ”By granting temporary monopolies over the production
and sale of new inventions and discoveries through various patent laws, govern-ments attempt to promote economic development. IP law walks a delicate balance,however, between promoting invention and not impeding science. Scienti ﬁc dis-
covery typically focuses upon exploring and understanding the laws of nature,natural phenomena, and abstract ideas that govern the universe. This is perhaps a
critical reason why the courts have carved out these subject matter exceptions to
patent eligibility. Setting aside the moral problems posed by allowing monopoliesover mere discoveries of naturally-occurring phenomena or laws, a practical andunwanted consequence might be to prevent researchers in basic sciences fromconducting their research freely.
There have been certain realms in which such a monopoly was never deemed
proper. The stated purpose of IP law is to promote invention, innovation, techno-logical development and thus economic development. Science is typically the ﬁeld
viewed most appropriate for basic research into the working of nature, and the
impetus for scienti ﬁc discovery differs from that of commerce and industry. Science
is impelled largely by scienti ﬁc culture, which rewards discoveries about nature and
its laws through academic careers, prizes, and often through the recognition and joymany of those who pursue scienti ﬁc careers receive from uncovering nature ’s
mysteries. Although competition and egos are also powerful motivators for scien-tists to pursue their work, for science to work correctly, a degree of openness andrecognition of limits of knowledge and the contingency of present understanding,
ensure that humility and cooperation remain part of the scienti ﬁc culture as well.
By recognizing that abstract ideas, laws of nature, and natural phenomena cannot
be patented, the US Supreme Court has upheld, despite the lack of explicitexclusion in the Patent Act, a boundary between science and technology; betweenwhat may be owned and what may not. The boundaries that have arisen in IP lawwill mean very little when it comes to the singularity, because the singularitypromises to erase preexisting boundaries in revolutionary ways. Currently, one canpatent life, just not humans. Currently, one can patent isolated and modi ﬁed natural
phenomena and products of nature, but not if they are, apparently, morphologically
identical to natural products. What remains free for all are natural products quanatural products, and natural phenomena that have not been otherwise modi ﬁed by
man to serve some end. Where then does this leave the following possibility:converging technologies that result in arti ﬁcially intelligent, self-replicating entities
capable of modi ﬁcation through “evolutionary ”means? Under current IP regimes,
what repercussions will there be? What will be ownable through patent, and whatnot?10 Can the Singularity Be Patented? … 187
10.5 Owning the Singularity
Under the current IP regime described above, an arti ﬁcially-created intelligent agent
would be ownable, as long as it isn ’t somehow composed by cloning human DNA,
which is so far illegal, and humans are speci ﬁcally exempt from patent, even if
engineered in some way. Such a product quali ﬁes for patent as long as it meets the
criteria of new, non-obvious, and useful. It is trivial that, given a lack of explicitexception for intelligent inventions, such an agent would be susceptible to IPclaims. After all, who would invent such a useful thing without the patent incentive,one might argue. Be that as it may, patenting such an agent has signi ﬁcant moral
repercussions and practical implications that we should consider.
Our hypothetical agent of the singularity will be able to be legally monopolized
under patent by its creator. This means the following: no one may reproduce that
agent without the patent holder ’s license. This may seem unproblematic at ﬁrst; you
made your intelligent agent through your own ingenuity and inventiveness, so whyshould anyone else be able to pro ﬁt? But remaining questionable is whether that
agent ’s ability (forgetting for the moment whether it has any rights) to reproduce is
thwarted by the patent law. More complicated is the question of whether, if thatagent can create new, non-obvious, and useful things of its own, that agent, thecreator, or some other agent will have the ability to patent those new creations.
Whether we create some new, non-obvious, and useful agent through
silicon-based arti ﬁcial intelligence, uplifting other creatures, or through nano- or
bio-tech, those agents will be ownable, to a degree. To the extent that they willthemselves be impeded from reproducing duplicates of themselves, their rights assentient being will be curtailed. They will be prevented from expressing one of thebasic rights of sentient beings: self-reproduction. More curious will be the status oftheir inventions, if any. If they are sentient beings, and capable of creating new,non-obvious, and useful new things, will they be able to patent them, or will their
own inventors? If the former, then curtailing their right to self-replicating repro-
duction seems unfair, whereas if the latter, then this seriously alters the manner inwhich we think inventors gain rights to the fruits of their intellectual labor.
A signi ﬁcant and perhaps insurmountable problem for IP in general will be the
question of who shall rightly own the fruits of the inventiveness of inventivemachines. It ought to cause us to question the foundations of IP law itself and itsutilitarian intentions. A singularity will be able to compete with us humans in allrealms, including invention and creativity. If so, and we don ’t bar non-humans from
enjoying the bene ﬁts of a state-sponsored monopoly, singularity agents may well
usurp our roles as inventors, and monopolize the realm of monopolies through thebeneﬁt of the state. This is a purely pragmatic concern, however. More theoretical is
the ethical question regarding whether we could justly, though IP law, preventartiﬁcial agents from reproducing themselves. Let ’s consider this ﬁrst, and then
re-raise the pragmatic concerns to suggest that IP ought to come to an end beforethe singularity is realized,188 D. Koepsell
10.6 Ethics, Patents and Arti ﬁcial Agents
We may well decide not to grant rights of personhood, including the various
freedoms we enjoy as humans, to arti ﬁcial life-forms or agents regardless of
whether they ful ﬁll the de ﬁnition of The Singularity. This might be a good idea,
especially if we don ’t want to compete with them in the marketplace of ideas where
we are used to state-sponsored monopolies giving inventors a leg-up. But is it just?
It is only just if we make an exception for humans when it comes to basic rights,including self-replication or reproduction, or if we exclude these from basic rightsaltogether. We already exclude non-human animals from having basic rights, so theﬁrst choice is already in force, but this may well be due to the cognitive abilities of
non-human animals and not due to their particular material make-up. Recently, as
some have rallied to include the higher apes and dolphins among the ranks of full
rights-holders, it has become clear that the category of a species is not the onlycriterion by which we should consider other being to be potential rights-holders.Rather, capacities appear to be a better arbiter. After all, we measure the rights ofhumans by capacities as well, holding people to differing levels of responsibilityaccording to their capacities at the time of an act. We even hold certain animalsresponsible, at least to the extent of punishment, sometimes capital, for transgres-sions and harms.
So we may well ask, and the patent of ﬁces in both the US and Europe are bound
to ask, whether issuing a patent over a sentient being would violate an ethical dutyor moral code. Morality has been considered to be a measure of patentability due tothe utility clause, and immoral inventions may be denied because they lack properuseful purpose. Thus, while one might invent something immoral, one may notpatent it. But what if the patent itself is immoral? In other words, even if theinvention is not immoral, what if the patenting of the thing invented is itselfimmoral?
We should consider the reasoning behind the prohibition of patents on humans,
and extend that reasoning to its logical conclusion. After the Chakrabarthy deci-
sion, the PTO issued a statement indicating that, while patents on new life-formswould be granted, they would not issue patents for “humans. ”The rationale was
that such patents would con ﬂict with the US Constitution, likely but not explicitly
referring to the 13th Amendment, which prohibited slavery. This self-imposedmoratorium seems also to include human embryos, as well as fully-formed humans.The justi ﬁcation proffered for this limitation on “human ”patents is threefold:
Congress never intended for humans to be patentable, immoral patents fail under
the utility requirement, and the 13th Amendment prohibits owning humans. Notableabout this reasoning, setting aside for the moment the merits of these arguments, isthat none of them will do much to guide patenting arti ﬁcial agents.
An arti ﬁcial agent will be patentable at least as a new, non-obvious, and useful
composition of matter or as a machine. The moral utility doctrine, which is notstatutory but court-made, is not clearly applicable to an arti ﬁcial agent, unless we
either view the making or existence of such agents as somehow immoral, and the10 Can the Singularity Be Patented? … 189
13th Amendment has never been applied to non-human persons. This will require
us to consider the question, already raised in the cases of higher apes and dolphins,of whether we ought to extend to other intelligent creatures some degree of “per-
sonhood ”bringing them under the protection of existing laws, or create some new
category of rights covering other sentient entities.
Another interesting and potentially troubling aspect of patents regarding arti ﬁcial
agents is the problem of software patents. Speci ﬁcally, typically excluded from
patent are “purely mental processes, ”presumably because in order to be patentable
a process must be instantiated in some machine or composition of matter. Oneproblem for an arti ﬁcial agent embodying singularity-type technology will be the
nature of its “operating system ”and whether and the degree to which it may not be
under the control of the agent itself. Are its thought processes and software purely
mental processes? Does this mean they are not patentable by others or by itself?
Even if we grant an agent its liberty by excluding it from patentability as a com-position of matter, based upon some extension of the moral reasoning prohibitingpatenting of humans, will we be able to charge such agents royalties for the use ofcode, upgrades, etc., all of which may be software necessitating or enabling mentalprocesses in the agent.
The basis for the moral prohibition against patents on humans seems clear: a
general acceptance of the liberal notion at least of self-ownership. If a second party
can claim a monopoly over either a part or a whole of another agent, this seems to
cut into a signi ﬁcant part of the claims ordinarily attributed to owners. From this
ﬂows considerable trouble if we begin to accept the notion that agents can be
non-human, and that they may too be rightly endowed with rights previouslyreserved only for humans.
Extending to non-human agents the evolution of rights as they have changed in
liberal democracies, and now embraced by universal treaties, means rethinking thenature of intellectual property because at some point our machines, even if invented
wholly by the use of human creativity, will be constrained in their agency by the
laws which enable patents. We as humans too may wish to carefully reconsider theextent to which we will be willing to allow our machines, where they act creativelyand inventively, to hold the threat of monopoly over us. If the singularity ’s Bill
Gateses, Steve Jobs, or even David Foster Wallaces can produce and monopolisethe objects of their creativity, what chance will humans have under the current lawto compete?
10.7 The Open Alternative
Recently, many working to develop new, groundbreaking technologies have optedto keep them open for all to use, rather than bottle them up behind state-sponsoredmonopolies. Elon Musk ’s Hyperloop concept is one such example as is his decision
to open up the standards behind his Tesla automobiles and not pursue infringement
claims against competitors. Google ’s most valuable code sits not behind the safety190 D. Koepsell
of patents, but rather as a trade secret which could last inde ﬁnitely, as long as no
one reverse engineers or duplicates the code. Many software developers arechoosing to release their code without patents, but using instead open source orother non-exclusive licenses such as that used by Linux. The bene ﬁt of open
innovation is partly realized by fewer lawsuits, and competition and cooperation
seem to be working as motivating and even pro ﬁtable impulses in non-patented
technologies. If applied to products that will move us toward the Singularity, manyof the potential pitfalls discussed above will not materialize, even while we ’ll still
grappled with risk and ethics along the way.
It seems most likely that as the rapid growth of new technologies that edge us
toward the Singularity continues, and keeps challenging the categories that hadalready begun to fail underlying IP law, that those who wish to see its imminent
development, free of the legal costs and risks that other technical developments
have grappled with, the Open alternative will dominate, help prevent legal andeconomic thickets, and similarly help to avoid the dangers presented by IP in aworld in which humans may not be the most inventive creatures around.
References
Ass ’n for Molecular Pathology v. Myriad , 133 S. Ct. 2107, 569 U.S., 186 L. Ed. 2d 124 (2013).
Bell, J. J. (2003). Exploring The “Singularity ”.Futurist ,37(3), 18 –25.
Bilski v. Kappos , 130 S. Ct. 3218, 561 U.S. 593, 177 L. Ed. 2d 792 (2010).
Diamond v. Chakrabarty , 447 U.S. 303, 100 S. Ct. 2204, 65 L. Ed. 2d 144 (1980).
Eckersley, R. (2001). Economic progress, social disquiet: the modern paradox. Australian Journal
of Public Administration ,60(3), 89 –97.
Johnson, L. (2009). Are We Ready for Nanotechnology? How to De ﬁne Humaness In Public
Policy. How to De ﬁne Humaness In Public Policy .
Mayo Collaborative v. Prometheus Labs. , 132 S. Ct. 1289, 566 U.S. 10, 182 L. Ed. 2d 321 (2012).
Patent Act: 35 U.S.C. § 161.
Vinge, V. (1993). The coming technological singularity. Whole Earth Review ,81,8 8–95.
Yampolskiy, R. V. (2013). What to Do with the Singularity Paradox? (pp. 397 –413). Springer
Berlin Heidelberg.10 Can the Singularity Be Patented? … 191


Risks of the Journey to the Singularity

Chapter 2
Risks of the Journey to the Singularity
Kaj Sotala and Roman Yampolskiy
2.1 Introduction1
Many have argued that in the next twenty to one hundred years we will create
artiﬁcial general intelligences [AGIs] (Baum et al. 2011 ; Sandberg and Bostrom
2011 ; Müller and Bostrom 2014 ).2Unlike current “narrow ”AI systems, AGIs
would perform at or above the human level not merely in particular domains (e.g.,chess or arithmetic), but in a wide variety of domains, including novel ones.
3They
would have a robust understanding of natural language and be capable of general
problem solving.
The creation of AGI could pose challenges and risks of varied severity for
society, such as the possibility of AGIs outcompeting humans in the job market(Brynjolfsson and McAfee 2011 ). This article, however, focuses on the suggestion
K. Sotala
Foundational Research Institute, Basel, Switzerland
e-mail: kaj.sotala@foundational-research.org
R. Yampolskiy ( ✉)
University of Louisville, Louisville, USA
e-mail: roman.yampolskiy@louisville.edu1This chapter is based on three earlier publications (Sotala and Yampolskiy 2015 ; Sotala
and Yampolskiy 2013 ; Yampolskiy 2013 ).
2Unlike the term “human-level AI, ”the term “Artiﬁcial General Intelligence ”does not necessarily
presume that the intelligence will be human-like.
3For this paper, we use a binary distinction between narrow AI and AGI. This is merely for the
sake of simplicity we do not assume the actual difference between the two categories to necessarily
be so clean-cut.
© Springer-Verlag GmbH Germany 2017
V. Callaghan et al. (eds.), The Technological Singularity ,
The Frontiers Collection, DOI 10.1007/978-3-662-54033-6\_211
that AGIs may come to act in ways not intended by their creators, and in this way
pose a catastrophic (Bostrom and Ćirkovi ć2008 ) or even an existential (Bostrom
2002 ) risk to humanity.4
2.2 Catastrophic AGI Risk
We begin with a brief sketch of the argument that AGI poses a catastrophic risk to
humanity. At least two separate lines of argument seem to support this conclusion.This argument will be further elaborated on in the following sections.
First, AI has already made it possible to automate many jobs (Brynjolfsson and
McAfee 2011 ), and AGIs, when they are created, should be capable of performing
most jobs better than humans (Hanson 2008 ; Bostrom 2014 ). As humanity grows
increasingly reliant on AGIs, these AGIs will begin to wield more and moreinﬂuence and power. Even if AGIs initially function as subservient tools, an
increasing number of decisions will be made by autonomous AGIs rather than byhumans. Over time it would become ever more dif ﬁcult to replace the AGIs, even if
they no longer remained subservient.
Second, there may be a sudden discontinuity in which AGIs rapidly become far
more numerous or intelligent (Good 1965 ; Chalmers 2010 ; Bostrom 2014 ). This
could happen due to (1) a conceptual breakthrough which makes it easier to runAGIs using far less hardware, (2) AGIs using fast computing hardware to developever-faster hardware, or (3) AGIs crossing a threshold in intelligence that allowsthem to carry out increasingly fast software self-improvement. Even if the AGIswere expensive to develop at ﬁrst, they could be cheaply copied and could thus
spread quickly once created.
Once they become powerful enough, AGIs might be a threat to humanity even if
they are not actively malevolent or hostile. Mere indifference to human values —
including human survival —could be suf ﬁcient for AGIs to pose an existential threat
(Yudkowsky 2008a ,2011 ; Omohundro 2007 ,2008 ; Bostrom 2014 ).
We will now lay out the above reasoning in more detail.
2.2.1 Most Tasks Will Be Automated
Ever since the Industrial Revolution, society has become increasingly automated.Brynjolfsson and McAfee ( 2011 ) argue that the current high unemployment rate in
the United States is partially due to rapid advances in information technology,
4A catastrophic risk is something that might in ﬂict serious damage to human well-being on a
global scale and cause ten million or more fatalities (Bostrom and Ćirkovi ć2008 ). An existential
risk is one that threatens human extinction (Bostrom 2002 ). Many writers argue that AGI might be
a risk of such magnitude (Butler 1863 ; Wiener 1960 ; Good 1965 ; Vinge 1993 ; Joy 2000 ; Yud-
kowsky 2008a ; Bostrom 2014 ).12 K. Sotala and R. Yampolskiy
which has made it possible to replace human workers with computers faster than
human workers can be trained in jobs that computers cannot yet perform. Vendingmachines are replacing shop attendants, automated discovery programs whichlocate relevant legal documents are replacing lawyers and legal aides, and auto-
mated virtual assistants are replacing customer service representatives.
Labor is becoming automated for reasons of cost, ef ﬁciency, and quality. Once a
machine becomes capable of performing a task as well as (or almost as well as) ahuman, the cost of purchasing and maintaining it may be less than the cost ofhaving a salaried human perform the same task. In many cases, machines are alsocapable of doing the same job faster, for longer periods, and with fewer errors. Inaddition to replacing workers entirely, machines may also take over aspects of jobsthat were once the sole domain of highly trained professionals, making the job
easier to perform by less-skilled employees (Whitby 1996 ).
If workers can be affordably replaced by developing more sophisticated AI, there
is a strong economic incentive to do so. This is already happening with narrow AI,which often requires major modi ﬁcations or even a complete redesign in order to be
adapted for new tasks. “A Roadmap for US Robotics ”(Hollerbach et al. 2009 ) calls
for major investments into automation, citing the potential for considerableimprovements in the ﬁelds of manufacturing, logistics, health care, and services.
Similarly, the US Air Force Chief Scientist ’s (Dahm 2010 )“Technology Horizons ”
report mentions “increased use of autonomy and autonomous systems ”as a key
area of research to focus on in the next decade, and also notes that reducing theneed for manpower provides the greatest potential for cutting costs. In 2000, the USCongress instructed the armed forces to have one third of their deep strike forceaircraft be unmanned by 2010, and one third of their ground combat vehicles beunmanned by 2015 (Congress 2000 ).
To the extent that an AGI could learn to do many kinds of tasks —or even any
kind of task —without needing an extensive re-engineering effort, the AGI could
make the replacement of humans by machines much cheaper and more pro ﬁtable.
As more tasks become automated, the bottlenecks for further automation willrequire adaptability and ﬂexibility that narrow-AI systems are incapable of. These
will then make up an increasing portion of the economy, further strengthening theincentive to develop AGI.
Increasingly sophisticated AI may eventually lead to AGI, possibly within the
next several decades (Baum et al. 2011 ; Müller and Bostrom 2014 ). Eventually it
will make economic sense to automate all or nearly all jobs (Hanson 2008 ; Hall
2008 ). As AGIs will possess many advantages over humans (Sotala 2012 ;
Muehlhauser and Salamon 2012a ,b; Bostrom 2014 ), a greater and greater pro-
portion of the workforce will consist of intelligent machines.2 Risks of the Journey to the Singularity 13
2.2.2 AGIs Might Harm Humans
AGIs might bestow overwhelming military, economic, or political power on the
groups that control them (Bostrom 2002 ,2014 ). For example, automation could
lead to an ever-increasing transfer of wealth and power to the owners of the AGIs
(Brynjolfsson and McAfee 2011 ). AGIs could also be used to develop advanced
weapons and plans for military operations or political takeovers (Bostrom 2002 ).
Some of these scenarios could lead to catastrophic risks, depending on the capa-bilities of the AGIs and other factors.
Our focus is on the risk from the possibility that AGIs could behave in unex-
pected and harmful ways, even if the intentions of their owners were benign. Evenmodern-day narrow-AI systems are becoming autonomous and powerful enough
that they sometimes take unanticipated and harmful actions before a human
supervisor has a chance to react. To take one example, rapid automated trading wasfound to have contributed to the 2010 stock market “Flash Crash ”(CFTC and SEC
2010 ).
5Autonomous systems may also cause people dif ﬁculties in more mundane
situations, such as when a credit card is automatically ﬂagged as possibly stolen due
to an unusual usage pattern (Allen et al. 2006 ), or when automatic defense systems
malfunction and cause deaths (Shachtman 2007 ).
As machines become more autonomous, humans will have fewer opportunities
to intervene in time and will be forced to rely on machines making good choices.
This has prompted the creation of the ﬁeld of “machine ethics ”(Wallach and Allen
2009 ; Allen et al. 2006 ; Anderson and Anderson 2011 ), concerned with creating AI
systems designed to make appropriate moral choices. Compared to narrow-AIsystems, AGIs will be even more autonomous and capable, and will thus requireeven more robust solutions for governing their behavior.
6
If some AGIs were both powerful and indifferent to human values, the conse-
quences could be disastrous. At one extreme, powerful AGIs indifferent to human
survival could bring about human extinction. As Yudkowsky ( 2008a ) writes, “The
AI does not hate you, nor does it love you, but you are made out of atoms which itcan use for something else. ”
5On the less serious front, see http://www.michaeleisen.org/blog/?p=358 for an amusing example
of automated trading going awry.
6In practice, there have been two separate communities doing research on automated moral
decision-making (Muehlhauser and Helm 2012a ,b; Allen and Wallach 2012 ; Shulman et al.
2009 ). The “AI risk ”community has concentrated speci ﬁcally on advanced AGIs (e.g. Yud-
kowsky 2008a ; Bostrom 2014 ), while the “machine ethics ”community typically has concentrated
on more immediate applications for current-day AI (e.g. Wallach et al. 2008 ; Anderson and
Anderson 2011 ). In this chapter, we have cited the machine ethics literature only where it seemed
relevant, leaving out papers that seemed to be too focused on narrow-AI systems for our purposes.
In particular, we have left out most discussions of military machine ethics (Arkin 2009 ), which
focus primarily on the constrained special case of creating systems that are safe for battle ﬁeld
usage.14 K. Sotala and R. Yampolskiy
Omohundro ( 2007 ,2008 ) and Bostrom ( 2012 ) argue that standard microeco-
nomic theory prescribes particular instrumental behaviors which are useful for theachievement of almost any set of goals. Furthermore, any agents which do notfollow certain axioms of rational behavior will possess vulnerabilities which some
other agent may exploit to their own bene ﬁt. Thus AGIs which understand these
principles and wish to act ef ﬁciently will modify themselves so that their behavior
more closely resembles rational economic behavior (Omohundro 2012 ). Extra
resources are useful in the pursuit of nearly any set of goals, and self-preservationbehaviors will increase the probability that the agent can continue to further itsgoals. AGI systems which follow rational economic theory will then exhibit ten-dencies toward behaviors such as self-replicating, breaking into other machines, andacquiring resources without regard for anyone else ’s safety. They will also attempt
to improve themselves in order to more effectively achieve these and other goals,
which could lead to rapid improvement even if the designers did not intend theagent to self-improve.
Even AGIs that were explicitly designed to behave ethically might end up acting
at cross-purposes to humanity, because it is dif ﬁcult to precisely capture the
complexity of human values in machine goal systems (Yudkowsky 2011 ; Muehl-
hauser and Helm 2012a ,b; Bostrom 2014 ).
Muehlhauser and Helm ( 2012a ,b) caution that moral philosophy has found no
satisfactory formalization of human values. All moral theories proposed so far
would lead to undesirable consequences if implemented by superintelligentmachines. For example, a machine programmed to maximize the satisfaction ofhuman (or sentient) preferences might simply modify people ’s brains to give them
desires that are maximally easy to satisfy.
Intuitively, one might say that current moral theories are all too simple —even if
they seem correct at ﬁrst glance, they do not actually take into account all the things
that we value, and this leads to a catastrophic outcome. This could be referred to as
thecomplexity of value thesis . Recent psychological and neuroscienti ﬁc experi-
ments con ﬁrm that human values are highly complex (Muehlhauser and Helm
2012a ,b), that the pursuit of pleasure is not the only human value, and that humans
are often unaware of their own values.
Still, perhaps powerful AGIs would have desirable consequences so long as they
were programmed to respect most human values. If so, then our inability to per-
fectly specify human values in AGI designs need not pose a catastrophic risk.Different cultures and generations have historically had very different values from
each other, and it seems likely that over time our values would become consider-
ably different from current-day ones. It could be enough to maintain some small setof core values, though what exactly would constitute a core value is unclear. Forexample, different people may disagree over whether freedom or well-being is amore important value.
Yudkowsky ( 2011 ) argues that, due to the fragility of value, the basic problem
remains. He argues that, even if an AGI implemented most human values, the
outcome might still be unacceptable. For example, an AGI which failed to incor-
porate the value of novelty could create a solar system ﬁlled with countless minds2 Risks of the Journey to the Singularity 15
experiencing one highly optimal and satisfying experience over and over again,
never doing or feeling anything else (Yudkowsky 2009 ).7
In this paper, we will frequently refer to the problem of “AGI safety ”or“safe
AGI,”by which we mean the problem of ensuring that AGIs respect human values,
or perhaps some extrapolation or idealization of human values. We do not seek to
imply that current human values would be the best possible ones, that AGIs couldnot help us in developing our values further, or that the values of other sentientbeings would be irrelevant. Rather, by “human values ”we refer to the kinds of
basic values that nearly all humans would agree upon, such as that AGIs forciblyreprogramming people ’s brains, or destroying humanity, would be a bad outcome.
In cases where proposals related to AGI risk might change human values in somemajor but not as obviously catastrophic way, we will mention the possibility of
these changes but remain agnostic on whether they are desirable or undesirable.
We conclude this section with one frequently forgotten point in order to avoid
catastrophic risks or worse, it is not enough to ensure that only some AGIs are safe.Proposals which seek to solve the issue of catastrophic AGI risk need to alsoprovide some mechanism for ensuring that most (or perhaps even “nearly all ”)
AGIs are either created safe or prevented from doing considerable harm.
2.2.3 AGIs May Become Powerful Quickly
There are several reasons why AGIs may quickly come to wield unprecedentedpower in society. “Wielding power ”may mean having direct decision-making
power, or it may mean carrying out human decisions in a way that makes thedecision maker reliant on the AGI. For example, in a corporate context an AGI couldbe acting as the executive of the company, or it could be carrying out countless
low-level tasks which the corporation needs to perform as part of its daily operations.
Bugaj and Goertzel ( 2007 ) consider three kinds of AGI scenarios: capped
intelligence, soft takeoff, and hard takeoff. In a capped intelligence scenario, all
AGIs are prevented from exceeding a predetermined level of intelligence andremain at a level roughly comparable with humans. In a soft takeoff scenario, AGIs
become far more powerful than humans, but on a timescale which permits ongoinghuman interaction during the ascent. Time is not of the essence, and learning
7Miller ( 2012 ) similarly notes that, despite a common belief to the contrary, it is impossible to
write laws in a manner that would match our stated moral principles without a judge needing to use
a large amount of implicit common-sense knowledge to correctly interpret them. “Laws shouldn ’t
always be interpreted literally because legislators can ’t anticipate all possible contingencies. Also,
humans ’intuitive feel for what constitutes murder goes beyond anything we can commit to paper.
The same applies to friendliness. ”(Miller 2012 ).16 K. Sotala and R. Yampolskiy
proceeds at a relatively human-like pace. In a hard takeoff scenario, an AGI will
undergo an extraordinarily fast increase in power, taking effective control of theworld within a few years or less.
8In this scenario, there is little time for error
correction or a gradual tuning of the AGI ’s goals.
The viability of many proposed approaches depends on the hardness of a takeoff.
The more time there is to react and adapt to developing AGIs, the easier it is tocontrol them. A soft takeoff might allow for an approach of incremental machineethics (Powers 2011 ), which would not require us to have a complete philosophical
theory of ethics and values, but would rather allow us to solve problems in agradual manner. A soft takeoff might however present its own problems, such asthere being a larger number of AGIs distributed throughout the economy, making itharder to contain an eventual takeoff.
Hard takeoff scenarios can be roughly divided into those involving the quantity
of hardware (the hardware overhang scenario), the quality of hardware (the speed
explosion scenario), and the quality of software (the intelligence explosion sce-
nario). Although we discuss them separately, it seems plausible that several of themcould happen simultaneously and feed into each other.
9
2.2.3.1 Hardware Overhang
Hardware progress may outpace AGI software progress. Contemporary super-
computers already rival or even exceed some estimates of the computationalcapacity of the human brain, while no software seems to have both the brain ’s
general learning capacity and its scalability.
10
If such trends continue, then by the time the software for AGI is invented there
may be a computing overhang —an abundance of cheap hardware available for
8Bugaj and Goertzel de ﬁned hard takeoff to refer to a period of months or less. We have chosen a
somewhat longer time period, as even a few years might easily turn out to be too little time for
society to properly react.
9Bostrom ( 2014 , chap. 3) discusses three kinds of superintelligence. A speed superintelligence
“can do all that a human intellect can do, but much faster ”. A collective superintelligence is “a
system composed of large number of smaller intellects such that the system's overall performance
across many very general domains vastly outstrips that of any current cognitive system ”. A quality
superintelligence “is at least as fast as a human mind and vastly qualitatively smarter ”. These can
be seen as roughly corresponding to the different kinds of hard takeoff scenarios. A speedexplosion implies a speed superintelligence, an intelligence explosion a quality superintelligence,
and a hardware overhang may lead to any combination of speed, collective, and quality
superintelligence.
10Bostrom ( 1998 ) estimates that the effective computing capacity of the human brain might be
somewhere around 1017operations per second (OPS), and Moravec ( 1998 ) estimates it at 1014
OPS. As of June 2016, the fastest supercomputer in the world had achieved a top capacity of 1016
ﬂoating-point operations per second (FLOPS) and the ﬁve-hundredth fastest a top capacity of 1014
FLOPS (Top500 2016 ). Note however that OPS and FLOPS are not directly comparable and there
is no reliable way of interconverting the two. Sandberg and Bostrom ( 2008 ) estimate that OPS and
FLOPS grow at a roughly comparable rate.2 Risks of the Journey to the Singularity 17
running thousands or millions of AGIs, possibly with a speed of thought much
faster than that of humans (Yudkowsky 2008b ; Shulman and Sandberg 2010 , Sotala
2012 ).
As increasingly sophisticated AGI software becomes available, it would be
possible to rapidly copy improvements to millions of servers, each new version
being capable of doing more kinds of work or being run with less hardware. Thus,the AGI software could replace an increasingly large fraction of the workforce.
11
The need for AGI systems to be trained for some jobs would slow the rate ofadoption, but powerful computers could allow for fast training. If AGIs end up doingthe vast majority of work in society, humans could become dependent on them.
AGIs could also plausibly take control of Internet-connected machines in order
to harness their computing power (Sotala 2012 ); Internet-connected machines are
regularly compromised.
12
2.2.3.2 Speed Explosion
Another possibility is a speed explosion (Solomonoff 1985 ; Yudkowsky 1996 ;
Chalmers 2010 ), in which intelligent machines design increasingly faster machines.
A hardware overhang might contribute to a speed explosion, but is not required for
it. An AGI running at the pace of a human could develop a second generation of
hardware on which it could run at a rate faster than human thought. It would thenrequire a shorter time to develop a third generation of hardware, allowing it to runfaster than on the previous generation, and so on. At some point, the process wouldhit physical limits and stop, but by that time AGIs might come to accomplish mosttasks at far faster rates than humans, thereby achieving dominance. (In principle, thesame process could also be achieved via improved software.)
11The speed that would allow AGIs to take over most jobs would depend on the cost of the
hardware and the granularity of the software upgrades. A series of upgrades over an extendedperiod, each producing a 1% improvement, would lead to a more gradual transition than a single
upgrade that brought the software from the capability level of a chimpanzee to a rough human
equivalence. Note also that several companies, including Amazon and Google, offer vast amountsof computing power for rent on an hourly basis. An AGI that acquired money and then invested allof it in renting a large amount of computing resources for a brief period could temporarily achieve
a much larger boost than its budget would otherwise suggest.
12Botnets are networks of computers that have been compromised by outside attackers and are
used for illegitimate purposes. Rajab et al. ( 2007 ) review several studies which estimate the sizes
of the largest botnets as being between a few thousand to 350,000 bots. Modern-day malwarecould theoretically infect any susceptible Internet-connected machine within tens of seconds of its
initial release (Staniford et al. 2002 ). The Slammer worm successfully infected more than 90% of
vulnerable hosts within ten minutes, and had infected at least 75,000 machines by the thirty-minutemark (Moore et al. 2003 ). The previous record holder in speed, the Code Red worm, took fourteen
hours to infect more than 359,000 machines (Moore et al. 2002 ).18 K. Sotala and R. Yampolskiy
The extent to which the AGI needs humans in order to produce better hardware
will limit the pace of the speed explosion, so a rapid speed explosion requires theability to automate a large proportion of the hardware manufacturing process.However, this kind of automation may already be achieved by the time that AGI is
developed.
13
2.2.3.3 Intelligence Explosion
Third, there could be an intelligence explosion , in which one AGI ﬁgures out how
to create a qualitatively smarter AGI, and that AGI uses its increased intelligence tocreate still more intelligent AGIs, and so on,
14such that the intelligence of
humankind is quickly left far behind and the machines achieve dominance (Good
1965 ; Chalmers 2010 ; Muehlhauser and Salamon 2012a ,b; Loosemore and
Goertzel 2012 ; Bostrom 2014 ).
Yudkowsky ( 2008a ,b) argues that an intelligence explosion is likely. So far,
natural selection has been improving human intelligence, and human intelligencehas to some extent been able to improve itself. However, the core process by whichnatural selection improves humanity has been essentially unchanged, and humanshave been unable to deeply affect the cognitive algorithms which produce their own
intelligence. Yudkowsky suggests that if a mind became capable of directly editing
itself, this could spark a rapid increase in intelligence, as the actual process causingincreases in intelligence could itself be improved upon. (This requires that thereexist powerful improvements which, when implemented, considerably increase therate at which such minds can improve themselves.)
Hall ( 2008 ) argues that, based on standard economic considerations, it would not
make sense for an AGI to focus its resources on solitary self-improvement. Rather,in order not to be left behind by society at large, it should focus its resources on
doing the things that it is good at and trade for the things it is not good at. However,
once there exists a community of AGIs that can trade with one another, thiscommunity could collectively undergo rapid improvement and leave humansbehind.
13Loosemore and Goertzel ( 2012 ) also suggest that current companies carrying out research and
development are more constrained by a lack of capable researchers than by the ability to carry outphysical experiments.
14Most accounts of this scenario do not give exact de ﬁnitions for “intelligence ”or explain what a
“superintelligent ”AGI would be like, instead using informal characterizations such as “a machine
that can surpass the intellectual activities of any man however clever ”(Good 1965 )o r“an intellect
that is much smarter than the best human brains in practically every ﬁeld, including scienti ﬁc
creativity, general wisdom and social skills ”(Bostrom 1998 ). Yudkowsky ( 2008a )d eﬁnes intel-
ligence in relation to “optimization power, ”the ability to reliably hit small targets in large search
spaces, such as by ﬁnding the a priori exceedingly unlikely organization of atoms which makes up
a car. A more mathematical de ﬁnition of machine intelligence is offered by Legg and Hutter
(2007 ). Sotala ( 2012 ) discusses some of the functional routes to actually achieving
superintelligence.2 Risks of the Journey to the Singularity 19
A number of formal growth models have been developed which are relevant to
predicting the speed of a takeoff; an overview of these can be found in Sandberg(2010 ). Many of them suggest rapid growth. For instance, Hanson ( 1998 ) suggests
that AGI might lead to the economy doubling in months rather than years. How-
ever, Hanson is skeptical about whether this would prove a major risk to humanity,
and considers it mainly an economic transition similar to the Industrial Revolution.
To some extent, the soft/hard takeoff distinction may be a false dichotomy.
A takeoff may be soft for a while, and then become hard. Two of the main factorsinﬂuencing the speed of a takeoff are the pace at which computing hardware is
developed and the ease of modifying minds (Sotala 2012 ). This allows for scenarios
in which AGI is developed and there seems to be a soft takeoff for, say, the initialten years, causing a false sense of security until a breakthrough in hardware
development causes a hard takeoff.
Another factor that might cause a false sense of security is the possibility that
AGIs can be developed by a combination of insights from humans and AGIsthemselves. As AGIs become more intelligent and it becomes possible to automateportions of the development effort, those parts accelerate and the parts requiringhuman effort become bottlenecks. Reducing the amount of human insight requiredcould dramatically accelerate the speed of improvement. Halving the amount ofhuman involvement required might at most double the speed of development,
possibly giving an impression of relative safety, but going from 50% human insight
required to 1% human insight required could cause the development to becomeninety-nine times faster.
15
From a safety viewpoint, the conservative assumption is to presume the worst
(Yudkowsky 2001 ). Yudkowsky argues that the worst outcome would be a hard
takeoff, as it would give us the least time to prepare and correct errors. On the otherhand, it can also be argued that a soft takeoff would be just as bad, as it would allowthe creation of multiple competing AGIs, allowing the AGIs that were the least
burdened with goals such as “respect human values ”to prevail. We would ideally
like a solution, or a combination of solutions, which would work effectively forboth a soft and a hard takeoff.
References
Allen, Colin, and Wendell Wallach. 2012. “Moral Machines: Contradiction in Terms or
Abdication of Human Responsibility. ”In Lin, Abney, and Bekey 2012, 55 –68.
Allen, Colin, Wendell Wallach, and Iva Smit. 2006. “Why Machine Ethics? ”IEEE Intelligent
Systems 21 (4): 12 –17. doi: 10.1109/MIS.2006.83 .
Amdahl, Gene M. 1967. “Validity of the Single Processor Approach to Achieving Large Scale
Computing Capabilities. ”In Proceedings of the April 18 –20, 1967, Spring Joint Computer
15The relationship in question is similar to that described by Amdahl ’s(1967 ) law.20 K. Sotala and R. Yampolskiy
Conference —AFIPS ’67 (Spring), 483 –485. New York: ACM Press. doi: 10.1145/1465482.
1465560 .
Anderson, Michael, and Susan Leigh Anderson, eds. 2011. Machine Ethics. New York:
Cambridge University Press.
Arkin, Ronald C. 2009. Governing Lethal Behavior in Autonomous Robots. Boca Raton, FL: CRC
Press.
Baum, Seth D., Ben Goertzel, and Ted G. Goertzel. 2011. “How Long Until Human-Level AI?
Results from an Expert Assessment. ”Technological Forecasting and Social Change 78 (1):
185–195. doi: 10.1016/j.techfore.2010.09.006 .
Bostrom, Nick. 1998. “How Long Before Superintelligence? ”International Journal of Futures
Studies 2.
Bostrom, Nick. 2002. “Existential Risks: Analyzing Human Extinction Scenarios and Related
Hazards. ”Journal of Evolution and Technology 9. http://www.jetpress.org/volume9/risks.html .
Bostrom, Nick. 2012. “The Superintelligent Will: Motivation and Instrumental Rationality in
Advanced Arti ﬁcial Agents. ”In“Theory and Philosophy of AI, ”edited by Vincent C. Müller.
Special issue, Minds and Machines 22 (2): 71 –85. doi: 10.1007/s11023-012-9281-3 .
Bostrom, Nick. 2014. Superintelligence: Paths, dangers, strategies. Oxford University Press.
Bostrom, Nick, and Milan M. Ćirkovi ć. 2008. “Introduction. ”In Bostrom, Nick, and Milan M.
Ćirkovi ć, eds. Global Catastrophic Risks. New York: Oxford University Press., 1 –30.
Brynjolfsson, Erik, and Andrew McAfee. 2011. Race Against The Machine: How the Digital
Revolution is Accelerating Innovation, Driving Productivity, and Irreversibly TransformingEmployment and the Economy. Lexington, MA: Digital Frontier. Kindle edition.
Bugaj, Stephan Vladimir, and Ben Goertzel. 2007. “Five Ethical Imperatives and Their
Implications for Human-AGI Interaction. ”Dynamical Psychology. http://goertzel.org/
dynapsyc/2007/Five\_Ethical\_Imperatives\_svbedit.htm .
Butler, Samuel [Cellarius, pseud.]. 1863. “Darwin Among the Machines. ”Christchurch Press,
June 13. http://www.nzetc.org/tm/scholarly/tei-ButFir-t1-g1-t1-g1-t4-body.html .
CFTC & SEC (Commodity Futures Trading Commission and Securities & Exchange Commis-
sion). 2010. Findings Regarding the Market Events of May 6, 2010: Report of the Staffs of theCFTC and SEC to the Joint Advisory Committee on Emerging Regulatory Issues. Washington,
DC. http://www.sec.gov/news/studies/2010/marketevents-report.pdf .
Chalmers, David John. 2010. “The Singularity: A Philosophical Analysis. ”Journal of
Consciousness Studies 17 (9 –10): 7 –65.http://www.ingentaconnect.com/content/imp/jcs/
2010/00000017/f0020009/art00001 .
Congress, US. 2000. National Defense Authorization, Fiscal Year 2001, Pub. L. No. 106 –398, 114
Stat. 1654.
Dahm, Werner J. A. 2010. Technology Horizons: A Vision for Air Force Science & Technology
During 2010-2030. AF/ST-TR-10-01-PR. Washington, DC: USAF. http://www.au.af.mil/au/
awc/awcgate/af/tech\_horizons\_vol-1\_may2010.pdf .
Good, Irving John. 1965. “Speculations Concerning the First Ultraintelligent Machine. ”In
Advances in Computers, edited by Franz L. Alt and Morris Rubinoff, 31 –88. Vol. 6. New
York: Academic Press. doi: 10.1016/S0065-2458(08)60418-0 .
Hall, John Storrs 2008. “Engineering Utopia. ”In Wang, Goertzel, and Franklin 2008, 460 –467.
Hanson, Robin. 1998. “Economic Growth Given Machine Intelligence. ”Unpublished manuscript.
Accessed May 15, 2013. http://hanson.gmu.edu/aigrow.pdf .
Hanson, Robin. 2008. “Economics of the Singularity. ”IEEE Spectrum 45 (6): 45 –50. doi: 10.1109/
MSPEC.2008.4531461 .
Hollerbach, John M., Matthew T. Mason, and Henrik I. Christensen. 2009. A Roadmap for US
Robotics: From Internet to Robotics. Snobird, UT: Computing Community Consortium. http://
www.usrobotics.us/reports/CCC%20Report.pdf .
Joy, Bill. 2000. “Why the Future Doesn ’t Need Us. ”Wired, April. http://www.wired.com/wired/
archive/8.04/joy.html .
Legg, Shane, and Marcus Hutter. 2007. “A Collection of De ﬁnitions of Intelligence. ”In Advances
in Arti ﬁcial General Intelligence: Concepts, Architectures and Algorithms —Proceedings of the2 Risks of the Journey to the Singularity 21
AGI Workshop 2006, edited by Ben Goertzel and Pei Wang, 17 –24. Frontiers in Arti ﬁcial
Intelligence and Applications 157. Amsterdam: IOS.
Loosemore, Richard, and Ben Goertzel. 2012. “Why an Intelligence Explosion is Probable. ”In
Eden, Amnon, Johnny Søraker, James H. Moor, and Eric Steinhart, eds. Singularity
Hypotheses: A Scienti ﬁc and Philosophical Assessment. The Frontiers Collection. Berlin:
Springer.
Miller, James D. 2012. Singularity Rising: Surviving and Thriving in a Smarter, Richer, and More
Dangerous World. Dallas, TX: BenBella Books.
Moore, David, Vern Paxson, Stefan Savage, Colleen Shannon, Stuart Staniford, and Nicholas
Weaver. 2003. “Inside the Slammer Worm. ”IEEE Security & Privacy Magazine 1 (4): 33 –39.
doi:10.1109/MSECP.2003.1219056 .
Moore, David, Colleen Shannon, and Jeffery Brown. 2002. “Code-Red: A Case Study on the
Spread and Victims of an Internet Worm. ”In Proceedings of the Second ACM SIGCOMM
Workshop on Internet Measurement (IMW ’02), 273 –284. New York: ACM Press. doi: 10.
1145/637201.637244 .
Moravec, Hans P. 1998. “When Will Computer Hardware Match the Human Brain? ”Journal of
Evolution and Technology 1. http://www.transhumanist.com/volume1/moravec.htm .
Muehlhauser, Luke, and Louie Helm. 2012. “The Singularity and Machine Ethics. ”In Eden,
Amnon, Johnny Søraker, James H. Moor, and Eric Steinhart, eds. Singularity Hypotheses: A
Scienti ﬁc and Philosophical Assessment. The Frontiers Collection. Berlin: Springer.
Muehlhauser, Luke, and Anna Salamon. 2012. “Intelligence Explosion: Evidence and Import. ”In
Eden, Amnon, Johnny Søraker, James H. Moor, and Eric Steinhart, eds. Singularity
Hypotheses: A Scienti ﬁc and Philosophical Assessment. The Frontiers Collection. Berlin:
Springer.
Müller, V. C., and Bostrom, N. 2014. Future progress in arti ﬁcial intelligence: A survey of expert
opinion. Fundamental Issues of Arti ﬁcial Intelligence.
Omohundro, Stephen M. 2007. “The Nature of Self-Improving Arti ﬁcial Intelligence. ”Paper
presented at Singularity Summit 2007, San Francisco, CA, September 8 –9.http://
selfawaresystems.com/2007/10/05/paper-on-the-nature-of-self-improving-arti ﬁcial-
intelligence/ .
Omohundro, Stephen M. 2008. “The Basic AI Drives. ”In Wang, Goertzel, and Franklin 2008,
483–492.
Omohundro, Stephen M. 2012. “Rational Arti ﬁcial Intelligence for the Greater Good. ”In Eden,
Amnon, Johnny Søraker, James H. Moor, and Eric Steinhart, eds. Singularity Hypotheses: A
Scienti ﬁc and Philosophical Assessment. The Frontiers Collection. Berlin: Springer.
Powers, Thomas M. 2011. “Incremental Machine Ethics. ”IEEE Robotics & Automation
Magazine 18 (1): 51 –58. doi: 10.1109/MRA.2010.940152 .
Rajab, Moheeb Abu, Jay Zarfoss, Fabian Monrose, and Andreas Terzis. 2007. “My Botnet is
Bigger than Yours (Maybe, Better than Yours): Why Size Estimates Remain Challenging. ”In
Proceedings of 1st Workshop on Hot Topics in Understanding Botnets (HotBots ’07).Berkeley,
CA: USENIX. http://static.usenix.org/event/hotbots07/tech/full\_papers/rajab/rajab.pdf .
Sandberg, Anders. 2010. “An Overview of Models of Technological Singularity. ”Paper presented
at the Roadmaps to AGI and the Future of AGI Workshop, Lugano, Switzerland, March 8.
http://agi-conf.org/2010/wp-content/uploads/2009/06/agi10singmodels2.pdf .
Sandberg, Anders, and Nick Bostrom. 2008. Whole Brain Emulation: A Roadmap. Technical
Report, 2008-3. Future of Humanity Institute, University of Oxford. http://www.fhi.ox.ac.uk/
wpcontent/uploads/brain-emulation-roadmap-report1.pdf .
Sandberg, Anders, and Nick Bostrom. 2011. Machine Intelligence Survey. Technical Report,
2011-1. Future of Humanity Institute, University of Oxford. http://www.fhi.ox.ac.uk/reports/
2011-1.pdf .
Shachtman, Noah. 2007. “Robot Cannon Kills 9, Wounds 14. ”Wired, October 18. http://www.
wired.com/dangerroom/2007/10/robot-cannon-ki/ .22 K. Sotala and R. Yampolskiy
Shulman, Carl, and Anders Sandberg. 2010. “Implications of a Software-Limited Singularity. ”In
Mainzer, Klaus, ed. ECAP10: VIII European Conference on Computing and Philosophy.
Munich: Dr. Hut.
Shulman, Carl, Henrik Jonsson, and Nick Tarleton. 2009. “Machine Ethics and Superintelligence. ”
In Reynolds, Carson, and Alvaro Cassinelli, eds. AP-CAP 2009: The Fifth Asia-Paci ﬁc
Computing and Philosophy Conference, October 1st-2nd, University of Tokyo, Japan,
Proceedings, 95 –97.
Solomonoff, Ray J. 1985. “The Time Scale of Arti ﬁcial Intelligence: Re ﬂections on Social
Effects. ”Human Systems Management 5:149 –153.
Sotala, Kaj, and Roman V. Yampolskiy. 2013. Responses to catastrophic AGI risk: a survey.
Technical report 2013-2. Berkeley, CA: Machine Intelligence Research Institute.
Sotala, Kaj, and Roman V. Yampolskiy. 2015. Responses to catastrophic AGI risk: a survey.
Physica Scripta, 90(1), 018001.
Sotala, Kaj. 2012. “Advantages of Arti ﬁcial Intelligences, Uploads, and Digital Minds. ”
International Journal of Machine Consciousness 4 (1): 275 –291. doi: 10.1142/
S1793843012400161 .
Staniford, Stuart, Vern Paxson, and Nicholas Weaver. 2002. “How to 0wn the Internet in Your
Spare Time. ”In Proceedings of the 11th USENIX Security Symposium, edited by Dan Boneh,
149–167. Berkeley, CA: USENIX. http://www.icir.org/vern/papers/cdc-usenix-sec02/ .
Top500.org. 2016. Top500 list –June 2016. https://www.top500.org/list/2016/06/ .
Vinge, Vernor. 1993. “The Coming Technological Singularity: How to Survive in the Post-Human
Era.”In Vision-21: Interdisciplinary Science and Engineering in the Era of Cyberspace, 11 –22.
NASA Conference Publication 10129. NASA Lewis Research Center. http://ntrs.nasa.gov/
archive/nasa/casi.ntrs.nasa.gov/19940022855\_1994022855.pdf .
Wallach, Wendell, and Colin Allen. 2009. Moral Machines: Teaching Robots Right from Wrong.
New York: Oxford University Press. doi: 10.1093/acprof:oso/9780195374049.001.0001 .
Wallach, Wendell, Colin Allen, and Iva Smit. 2008. “Machine Morality: Bottom-Up and
Top-Down Approaches for Modelling Human Moral Faculties. ”In“Ethics and Arti ﬁcial
Agents. ”Special issue, AI & Society 22 (4): 565 –582. doi: 10.1007/s00146-007-0099-0 .
Whitby, Blay. 1996. Re ﬂections on Arti ﬁcial Intelligence: The Legal, Moral, and Ethical
Dimensions. Exeter, UK: Intellect Books.
Wiener, Norbert. 1960. “Some Moral and Technical Consequences of Automation. ”Science 131
(3410): 1355 –1358. http://www.jstor.org/stable/1705998 .
Yampolskiy, Roman V. 2013. What to Do with the Singularity Paradox? Studies in Applied
Philosophy, Epistemology and Rational Ethics vol 5, pp. 397 –413. Springer Berlin Heidelberg.
Yudkowsky, Eliezer. 1996. “Staring into the Singularity. ”Unpublished manuscript. Last revised
May 27, 2001. http://yudkowsky.net/obsolete/singularity.html .
Yudkowsky, Eliezer. 2001. Creating Friendly AI 1.0: The Analysis and Design of Benevolent
Goal Architectures. The Singularity Institute, San Francisco, CA, June 15. http://intelligence.
org/ﬁles/CFAI.pdf .
Yudkowsky, Eliezer. 2008a. “Artiﬁcial Intelligence as a Positive and Negative Factor in Global
Risk. ”In Bostrom, Nick, and Milan M. Ćirkovi ć, eds. Global Catastrophic Risks. New York:
Oxford University Press., 308 –345.
Yudkowsky, Eliezer. 2008b. “Hard Takeoff. ”Less Wrong (blog), December 2. http://lesswrong.
com/lw/wf/hard\_takeoff/ .
Yudkowsky, Eliezer. 2009. “Value is Fragile. ”Less Wrong (blog), January 29. http://lesswrong.
com/lw/y3/value\_is\_fragile/ .
Yudkowsky, Eliezer. 2011. Complex Value Systems are Required to Realize Valuable Futures.
The Singularity Institute, San Francisco, CA. http://intelligence.org/ ﬁles/ComplexValues.pdf .2 Risks of the Journey to the Singularity 23