Physica Scripta

INVITED COMMENT
How feasible is the rapid development of artificial superintelligence?
To cite this article: Kaj Sotala 2017 Phys. Scr. 92 113001

View the article online for updates and enhancements.
This content was downloaded from IP address 129.8.242.67 on 25/10/2017 at 15:09
Invited Comment
How feasible is the rapid development of
artiﬁcial superintelligence?
Kaj Sotala
Foundational Research Institute, Germany
E-mail: kaj.sotala@foundational-research.org
Received 22 March 2017, revised 18 August 2017
Accepted for publication 4 October 2017
Published 24 October 2017
Abstract
What kinds of fundamental limits are there in how capable arti ﬁcial intelligence (AI)systems
might become? Two questions in particular are of interest: (1)How much more capable could AI
become relative to humans, and (2)how easily could superhuman capability be acquired? To
answer these questions, we will consider the literature on human expertise and intelligence,discuss its relevance for AI, and consider how AI could improve on humans in two major aspectsof thought and expertise, namely simulation and pattern recognition. We ﬁnd that although there
are very real limits to prediction, it seems like AI could still substantially improve on human
intelligence.
Keywords: arti ﬁcial intelligence, prediction, intelligence, limits of intelligence, arti ﬁcial general
intelligence, superintelligence, technological singularity
Introduction
Since Turing (1950 ), the dream of arti ﬁcial intelligence (AI)
research has been the creation of a ‘machine that could think ’.
While current expert consensus is that the creation of such a
system will take several decades if not more (Müller and
Bostrom 2016 ), recent progress in AI has still raised worries
about the challenges involved with increasingly capable AIsystems (Future of Life Institute 2015 , Amodei et al 2016 ).
In addition to the risks posed by near-term developments,
there is the possibility of AI systems eventually reachingsuperhuman levels of intelligence, eventually breaking out of
human control (Bostrom 2014 ). Various research agendas and
lists of research priorities have been suggested for managingthe challenges that this level of capability would pose tosociety (Soares and Fallenstein 2014 , Russell et al 2015 ,
Amodei et al 2016 , Taylor et al 2016 ).
For managing the challenges presented by increasingly
capable AI systems, one needs to know how capable thosesystems might ultimately become, and how quickly. If AI
systems can rapidly achieve strong capabilities, becoming
powerful enough to take control of the world before any humancan react, then that implies a very different approach than onewhere AI capabilities develop gradually over many decades,never getting substantially past the human level (Sotala and
Yampolskiy 2015 ). We might phrase these questions as:
1. How much more capable can AIs become relative to
humans?
2. How easily (in terms of time and resources required
)
could superhuman capability be acquired?
Views on these questions vary. Authors such as Bostrom
(2014 )and Yudkowsky (2008 )argue for the possibility of a fast
leap in intelligence, with both offering hypothetical examplescenarios where AI rapidly acquires a dominant position overhumanity. On the other hand, Anderson (2010 )and Lawrence
(2016 )appeal to fundamental limits on predictability —and thus
intelligence —posed by the complexity of the environment.
The argument for limits of intelligence (Anderson 2010 ,
Lawrence 2016 )could be summarized as saying that, past a
certain point, increased intelligence is only of limited bene ﬁt,
for the unpredictability of the environment means that youwould have to spend exponentially more resources to evaluatea vastly increasing amount of possibilities.
Noise also accumulates over time, reducing the reliability
of the available models. For many kinds of predictions,increasing the prediction window would require an exponentialincrease in the amount of measurements (Martela 2016 ).F o r|Royal Swedish Academy of Sciences Physica Scripta
Phys. Scr. 92 (2017 )113001 (14pp ) https: //doi.org /10.1088 /1402-4896 /aa90e8
0031-8949 /17/113001 +14$33.00 © 2017 IOP Publishing Ltd Printed in the UK 1
instance, weather models become increasingly uncertain when
projected farther out in time. Forecasters can only access alimited amount of observations relative to the weather system ’s
degrees of freedom, and any initial imprecisions will magnifyover time and cause the accuracy to deteriorate (Buizza 2002 ).
In general, the accuracy of any long-term prediction will be
limited by data uncertainty, model uncertainty, and the available
computational time. Similar considerations would also apply toattempts to predict things such as the behaviour of humansocieties. The advantage that even a superhuman intelligencemight have over humans may be limited.
On the other hand, it is not obvious whether this point of
view really is in con ﬂict with the assumption of AI being able
to quickly grow to become powerful. There being limits to
prediction does not imply that humans would be particularly
close to the limits, nor that it would necessarily take a greatamount of time to move from sub-human to superhumancapability.
This article attempts to consider these questions by
considering what we know about expertise and intelligence.After reviewing the relevant research on human expertise, wewill discuss its relevance for AI, and consider how AI could
improve on humans in two major aspects of thought and
expertise, namely simulation and pattern recognition. Ourcurrent conclusion is that although the limits to prediction arereal, it seems like AI could still substantially improve onhuman intelligence. The possibility of AI developing sig-niﬁcant real-world capabilities in a relatively brief time seems
like one that cannot be ruled out.
Before examining these questions, we need to consider
the de ﬁnition of ‘capability ’in more detail, and justify our
focus on intelligence as prediction ability.
Capability and intelligence as prediction ability
Bostrom (2014 ,p3 9 )deﬁnes a superintelligence as ‘any
intellect that greatly exceeds the cognitive performance of
humans in virtually all domains of interest ’. Additionally,
Bostrom (2014 , chapter 3 )deﬁnes three subcategories of a
superintelligence. A speed superintelligence thinks faster than
humans; a collective superintelligence is composed of many
smaller intellects whose overall performance outstrips that ofexisting cognitive systems; and a quality superintelligence is
one that is at least as fast as a human mind, and vastly qua-
litatively smarter.
In a footnote to his original de ﬁnition, Bostrom notes that
this de ﬁnition of superintelligence can be compared with
Legg (2008 ), who de ﬁnes intelligence as ‘an agent ’s ability to
achieve goals in a wide range of environments ’.
This de ﬁnition, originally from Legg and Hutter (2007a ),
draws on a collection of 70 de ﬁnitions of intelligence (
Legg
and Hutter 2007b )from various professional groups, dic-
tionaries, psychologists, and AI researchers. Legg and Hutter
(2007a )argue that this de ﬁnition summarizes the essential
features in the various surveyed de ﬁnitions, in that theygenerally discuss an individual who is interacting with some
environment that is not fully known, trying to achieve variousgoals in that environment, and learning and exploring duringthat interaction.
Some de ﬁnitions of intelligence list traits which are not
explicitly included in this de ﬁnition; for example, a group
statement signed by 52 psychologists (Gottfredson 1997a )
includes in intelligence ‘the ability to reason, plan, solve pro-
blems, think abstractly, comprehend complex ideas, learn quicklyand learn from experience ’. However, Legg and Hutter (2007a )
argue that all of these abilities are ones that allow humans toachieve goals, so are implicitly included in the Legg and Hutterdeﬁnition. Additionally, Legg and Hutter suggest that their
deﬁnition is more general, as the re could exist intelligences
which did not have all of these speci ﬁc capabilities, but did have
alternative capabilities which allo wed them to achieve their goals.
Legg and Hutter (2007a )offer a formalization of their
deﬁnition, cast in a reinforcement learning framework.
Brieﬂy, the formalization involves an agent which interacts
with an environment in discrete timesteps; on each timestep,
the agent chooses an action and receives both an observation
and a reward. An agent is (universally )intelligent to the
extent that it can maximize its reward over the space of allenvironments drawn from a universal distribution.
This de ﬁnition and formalization is a view of intelligent
performance as a learning and prediction problem: an agent isintelligent to the extent that it can learn to predict, using thesmallest possible set of observations, which of its actions willdeliver the greatest amount of reward in the environment thatit is interacting with.
Out of Bostrom ’s(2014 )superintelligence subtypes, a mind
that was superintelligent under such a view would most likelyfall under the category of a quality superintelligence. Some ofthe examples that Bostrom (2014 )offers to illustrate the concept
of quality intelligence include nonhuman animals that cannot
achieve human cognitive capabilities even when ‘intensely
trained by human instructors ’, as well as human de ﬁcits such as
autism spectrum disorders that may impair e.g. social function-
ing. Implicit in these examp les is the notion that nonhuman
animals and individuals with cognitive de ﬁcits cannot achieve
the same level of performance in various domains as unimpaired
humans do, even when given the same opportunities to observeand learn about the domains in qu estion. They lack the cognitive
capabilities that would allow them to utilize their observations tolearn to predict which kinds of actions would provide thegreatest success in the relevant domains.
Under this view, we can more precisely rephrase our ﬁrst
question, ‘how much more capable can AIs become relative
to humans ’,a s ‘how much better than humans can AIs
become in using small amounts of sense data to learn to
predict which actions most effectively further their goals ’. For
the purposes of this discussion, we will also assume that‘predicting which actions most effectively further one ’s goals ’
is an accurate characterization of what human expertise (in
any given domain )means. As we will discuss in the following
section, the foundation of human expertise lies in acquiring
2Phys. Scr. 92 (2017 )113001 Invited Comment
the necessary knowledge to instantly see, when faced with
some situation, the right course of action for that situation.
The development of human expertise
Ideally, we might turn to theoretical AI research for a precise
theory about acquiring cognitive capabilities. Unfortunately
AI research is not at this point yet. Instead we will considerthe research on human expertise and decision-making.
Expertise as mental representations
There exists a preliminary understanding, if not of the details
of human decision-making, then at least the general outline. A
picture that emerges from this research is that expertise is
about developing the correct mental representations(Klein 1999 , Ericsson and Pool 2016 ).
A mental representation is a very general concept,
roughly corresponding to any mental structure forming thecontent of something that the brain is thinking about (Ericsson
and Pool 2016 ).
Domain-speci ﬁc mental representations are important
because they allow experts to know what something means;know what to expect; know what good performance shouldfeel like; know how to achieve the good performance; knowthe right goals for a given situation; know the steps necessaryfor achieving those goals; mentally simulate how somethingmight happen; learn more detailed mental representations forimproving their skills (Klein 1999 , Ericsson and Pool 2016 ).
Although good decision-making is often thought of as a
careful deliberation of all the possible options, such a type ofthinking tends to be typical of novices (Klein 1999 ). A novice
will have to try to carefully reason their way through to ananswer, and will often do poorly regardless, because they donot know what things are relevant to take into account andwhich ones are not. An expert does not need to —they are
experienced enough to instantly know what to do.
A speci ﬁc model of expertise is the recognition-primed
decision-making model (Klein 1999 ). First, a decision-maker
sees some situation, such as a ﬁre for a ﬁreﬁghter or a design
problem for an architect. The situation may then be recog-nized as familiar, such as a typical garage ﬁre. Recognizing a
familiar situation means understanding what goals make
sense and what should be focused on, which cues to pay
attention to, what to expect next and when a violation of
expectations shows that something is amiss, and knowing
what the typical ways of responding are. Ideally, the expert
will instantly know what to do.
The expectations arising from mental representations also
give rise to intuition . As one example, Klein (1999 )describes
the case of a ﬁreﬁghter lieutenant responding to a kitchen ﬁre
in an ordinary one-story residential house. The lieutenant ’s
crew sprayed water on the ﬁre, but contrary to expectations,
the water seemed to have little impact. Something about the
situation seemed wrong to the lieutenant, who ordered hiscrew out of the house. As soon as they had left the house, theﬂoor where they had been standing collapsed. If the ﬁre-
ﬁghters had not pulled out, they would have fallen down to
theﬁre raging in the basement. The lieutenant, not knowing
what had caused him to give the order to withdraw, initiallyattributed the decision to some form of extra-sensoryperception.
In a later interview, the lieutenant explained that he did
not suspect that the building had a basement, nor that the seatof the ﬁre was under the ﬂoor that he and his crew were
standing on. However, several of his expectations of a typicalkitchen ﬁre were violated by the situation. The lieutenant was
wondering why the ﬁre did not react to water as expected, the
room was much hotter than he would have expected out of asmall kitchen ﬁre, and while a heat that hot should have made
a great deal of noise, it was very quiet. The mismatch between
the expected pattern and the actual situation led to an intuitivefeeling of not knowing what was going on, leading to thedecision to regroup. This is intuition: an automatic compar-ison of the situation against existing mental representations ofsimilar situations, guiding decision-making in ways whosereasons are not always consciously available.
In an unfamiliar situation, the expert may need to con-
struct a mental simulation of what is going on, how things
might have developed to this point, and what effect differentactions would have. Had the ﬂoor mentioned in the previous
example not collapsed, given time the ﬁreﬁghter lieutenant
might have been able to put the pieces together and constructa narrative of a ﬁre starting from the basement to explain the
discrepancies. For a future-oriented example, a ﬁreﬁghter
thinking about how to rescue someone from a dif ﬁcult spot
might mentally simulate where different rescue harnesses
might be attached on the person, and whether that would exertdangerous amounts of force on them.
Mental representations are necessary for a good simula-
tion, as they let the expert know what things to take intoaccount, what things could plausibly be tried, and what effectsthey would have. In the example, the ﬁreﬁghter ’s knowledge
allows him to predict that speci ﬁc ways of attaching the
rescue harness would have dangerous consequences, while
others are safe.
Developing mental representations
Mental representations are developed through practice. Anovice will try out something and see what happens as aresult. This gives them a rough mental representation and a
prediction of what might happen if they try the same thing
again, leading them to try out the same thing again or dosomething else instead.
Just practice is not enough, however —there also needs to
be feedback. Someone may do a practice drill over and overagain and think that they are practicing and thus improving —
but without some sign of how well that is going, they may justkeep repeating the same mistakes over and over (Ericsson and
Pool 2016 ).
The importance of quality feedback is worth emphasiz-
ing. Skills do not develop unless there is feedback that is
3Phys. Scr. 92 (2017 )113001 Invited Comment
conducive to developing better mental representations. In
fact, there are entire ﬁelds in which experienced practitioners
are not much better than novices, because the ﬁeld does not
provide them with enough feedback. Shanteau (1992 )pro-
vides the following breakdown of professions for which thereis agreement on the nature of their performance (Reprinted
from Shanteau, J (1992 ). Competence in experts: The role of
task characteristics. Organizational Behavior and HumanDecision Processes, (53), 252 –266, Copyright (1992 ), with
permission from Elsevier. ):
Good performance Bad performance
Weather forecasters Clinical psychologists
Livestock judges PsychiatristsAstronomers Astrologers
Test pilots Student admissions
Soil judges Court judgesChess masters Behavioural researchersPhysicists Counselors
Mathematicians Personnel selectors
Accountants Parole of ﬁcers
Grain inspectors Polygraph (lie detector )judges
Photo interpreters Intelligence analysts
Insurance analysts Stock brokers
In analysing why some domains enable the development
of genuine expertise and others do not, Shanteau identi ﬁed a
number of considerations that relate to the nature of feedback.
In an occupation like weather forecasting, the criteria you use
for forecasting are always the same; you will always be facingthe same task and can practice it over and over; you get quickand feedback on whether your prediction was correct; you canuse formal tools to analyse what you predicted would happenand why that prediction did or did not happen; and things canbe analysed in objective terms. This allows weather fore-casters to develop powerful mental representations that get
better and better at making the correct prediction.
Contrast this with someone like an intelligence analyst.
The analyst may be called upon to analyse very different cluesand situations; each of the tasks may be unique, making itharder to know which lessons from previous tasks apply; formany of the analyses, one might never know whether theywere right or not; and questions about socio-cultural matterstend to be much more subjective than questions about
weather, making objective analysis impossible. In short, for
much of the work that the analyst does, there is simply nofeedback available to tell whether the analyst has made theright judgment or not. And without feedback, there is no wayto improve one ’s mental representations, and thus expertise.
A slightly different look on expertise is the heuristics and
biases literature, which frequently portrays even experts asbeing easily mistaken. In contrast, the expertise literature that
we have reviewed so far has viewed experts as being typically
capable and as having trustworthy intuition. Kahneman andKlein (2009 )make an attempt to reconcile the two ﬁelds, and
come to agree that:
Expert intuition may be trustworthy, if the intuition
relates to a ‘high-validity ’domain and the expert has had
a chance to learn the regularities in that domain.A domain is ‘high-validity ’if‘there are stable relationships
between objectively identi ﬁable cues and subsequent events
or between cues and the outcomes of possible actions ’.
Medicine and ﬁreﬁghting have fairly high validity,
whereas predictions of the future value of individualstocks and long-term
1forecasts of political events are
domains with practically zero validity.
‘Some [domains ]are both highly valid and substantially
uncertain. Poker and warfare are examples. The bestmoves in such situations reliably increase the potential forsuccess ’.
‘[A domain ]of high validity is a necessary condition for
the development of skilled intuitions. Other necessaryconditions include adequate opportunities for learning the
[domain ](prolonged practice and feedback that is both
rapid and unequivocal ).I f [a domain ]provides valid cues
and good feedback, skill and expert intuition willeventually develop in individuals of suf ﬁcient talent ’.
This consensus is in line with what we have covered so
far, though it also includes the consideration of validity. Onecannot learn mental representations that would predict adomain or dictate the right actions for different situations in adomain, if that domain is simply too complicated or chaotic tobe predicted. Kahneman and Klein (2009 )provide an illus-
trative example of domain being simply too hard to interpret:
the question of how the history of the 20th century would
have been different if the fertilized eggs that became Hitler,Stalin and Mao had been female. It seems clear that thingswould have developed very differently, but how exactly?There seems to be no way to know.
Meanwhile, practice does help in more predictable domains.
A recent meta-analysis (Macnamara et al2014 )on the effects of
practice on skill found that the more predictable an activity was,
the more practice contributed to performance in that activity.
Implications for AI
Having reviewed some necessary background, we will now
ﬁnally get back to the topic of superintelligence capabilities.
Relevance for AI
Similarly to humans, AI systems cannot reach intelligent
conclusions by a mere brute force calculation of every pos-
sibility. Rather, an intelligence needs to learn to exploit pre-dictable regularities in the world in order to develop further.All machine learning based systems are based on this prin-ciple: they learn models of the world that are in this sensesimilar to the mental representations that humans learn.
However, the models employed by current machine
learning systems are much more limited than the mental
representations employed by humans (Lake et al 2016 ).
1Kahneman and Klein do not de ﬁne what they mean by ‘long-term ’, but
geopolitical events up to a year or so away can be predicted with reasonableaccuracy, with the accuracy falling towards chance for events 3 –5 years
away. (Tetlock and Gardner 2015 ,p5 ).
4Phys. Scr. 92 (2017 )113001 Invited Comment
Machine learning systems are also developed for solving
problems ef ﬁciently on existing computing hardware rather
than for being biologically plausible. There is thus reason toexpect even future AI systems to employ models which differin various respects from the mental representations used byhumans. As such, we will use the term ‘mental representa-
tions ’when in the context of humans, and ‘models ’when
discussing the analogous structure in future AI systems.
In a sense, mental representations contain the optimal
solutions to the problems at hand (Klein 1999 ): a human
expert will have learned to identify the smallest set of cuesthat will let them know how to act in a certain situation; theirmental representations encode information about how tochoose the correct actions using the least amount of thought.In other words, an expert pays attention exactly to the featuresin the data which are relevant for making the decision, andacts accordingly. An AI ’s models could use more data and
become larger than human mental representations, and iden-tify features which humans might have missed. There ishowever no advantage in using more data than necessary formaking the correct decision, so at least a subset of the AI ’s
models is likely to be similar to mental representations in thatthey encode the smallest amount of features of the environ-ment which allow for rapid and correct decision-making in agiven context and for a given goal.
It is possible that AIs would also come to have models
for which this characterization was a poor ﬁt and which were
tailored for taking better advantage of e.g. an AI ’s ability to
process more data at a time. We will not examine this morespeculative possibility, as for our argument it is unnecessaryto consider hypothetical models which are better than human
mental representations; we are focused on establishing thepossibility that roughly human-like models would already beenough to enable superhuman capability
2.
Like with human experts, machine learning also tries to
focus its analysis on exactly the right number of cues that willprovide the right predictions, ignoring any irrelevant infor-mation. Traditional machine learning approaches have reliedextensively on feature engineering , a labor-intensive process
where humans determine which cues in the data are worthpaying attention to.
A major reason behind the recent success of deep
learning models is their capability for feature learning or
representation learning : being able to independently discover
high-level features in the data which are worth payingattention to, without (as much )external guidance (Bengio
et al 2012 ). Being able to identify and extract the most
important features of the data allows the system to make itsdecisions based on the smallest amount of cues that allows itto reach the right judgment —just as human experts learn to
identify the most relevant cues in the situations that theyencounter.Finally, the aspect of increasingly detailed mental
representations giving an expert a yardstick to compare theirperformance against (Ericsson and Pool 2016 )has an analo-
gue in reinforcement learning methods. In deep reinforcementlearning, a deep learning model learns to estimate howvaluable a speci ﬁc state of the world is, after which the sys-
tem takes actions to move the world towards that state (Mnih
et al 2015 ). Similarly, a human expert comes to learn that
speciﬁc states (e.g. a certain feeling in the body when diving )
are valuable, and can then increasingly orient their behaviourso as to achieve this state.
In summary, human experts use mental representations as
the building blocks of their expertise, with the modelsemployed by current state-of-the-art AI systems having a
number of key similarities. As there have been no serious
alternative accounts presented of how expertise might work,we will assume that the capabilities of hypothetical super-intelligences will depend, at least in part, on them developingthe correct models to represent key features of the environ-ment in a similar way as human mental representations do.
This paper set out to consider two main questions:
1. How much more capable can AIs become relative to
humans?
2. How easily (in terms of time and resources required )
could superhuman capability be acquired?
Let us now return to these.
The argument for AI ’s predictive capabilities being lim-
ited was that there are limits to prediction, and that predictingevents an ever-increasing amount forward in time requires
exponential reasoning power as well as measurement points,
quickly becoming intractable. How capable could AI becomedespite these two points?
The components of human expertise might be roughly
divided into two: building up a battery of accurate mentalrepresentations, and being able to use them for mentalsimulations. Similarly, approaches to arti ﬁcial intelligence
can roughly be divided into pattern recognition and model-
building (Lake et al 2016 ), depending on whether patterns in
data or models of the world are treated as the primary unit ofthought.
As this kind of a distinction seems to emerge both from
psychology and AI research, we will assume that AI ’s
expertise will also involve acquiring models (or equivalently,
doing pattern recognition )as well as accurately using them in
simulations. We will consider these two separately.
Simulation
Potential capability . An interesting look at the potential
beneﬁts offered by improved simulation ability come from
looking at Philip Tetlock ’s Good Judgement Project (GJP ),
popularized in the book Superforecasting (Tetlock and
Gardner 2015 )3. Participating in a contest to forecast the2The reader may note that the AI possibly using many different kinds of
models, some of them human-like and some more advanced, has a parallel inthe heterogeneity hypothesis of concepts (Machery 2009 ,2010 ), according to
which the mental representations of humans do not form a natural kind andactually consist of many different kinds of mental structures that are used indifferent situations and for different purposes.3Except for when citations to other content are explicitly included, all the
discussion about superforecasters and the Good Judgment Project usesSuperforecasting as its source.
5Phys. Scr. 92 (2017 )113001 Invited Comment
probability of various events, the best GJP participants —the
so-called ‘superforecasters ’—managed to make predictions
whose accuracy outperformed those of professional
intelligence analysts working with access to classi ﬁed data4.
This is particularly interesting as the superforecasters had noparticular domain expertise in answering most of thequestions, with sample questions including ones such as
Will North Korea launch a new multistage missile before
10 May 2014?
Will Russian armed forces enter Kharkiv, Ukraine, by 10
May 2014?
Will there be a signi ﬁcant attack on Israeli territory before
10 May 2014?
Will Robert Mugabe cease to be President of Zimbabwe
by 30 September 2011?
Will Greece remain a member of the EU through 1
June 2012?
Tetlock and Gardner report the superforecasters ’accur-
acy in terms of Brier score, which is a scale between 0 and 2,with 0.5 indicating random guessing
5. On this scale, super-
forecasters had a score of 0.25 at the end of GJP ’sﬁrst year,
compared to 0.37 of the other forecasters participating in theproject. By the end of the second year, superforecasters hadimproved their Brier score to 0.07 (Mellers et al 2014 ).
Superforecasters could also project further out in time: their
accuracy at making predictions 300 days out was better as theother forecasters ’accuracy at making predictions 100 days
out. In terms of being on the right side of 50 /50, GJP ’s best
wisdom-of-the-crowd algorithms (deriving an overall predic-
tion from the different forecasters ’predictions )delivered a
correct prediction on 86% of all daily forecasts (Tetlock
et al 2014 ).
The superforecasters ’success relied on a number of
techniques, but a central one was the ability to consider andjudge the relevance of a number of factors that might cause aprediction to become true or false. Tetlock and Gardnerillustrate this technique by discussing how a superforecaster,Bill Flack, approached the question of whether an invest-
igation of Yasser Arafat ’s remains would reveal traces of
polonium, suggestive of Arafat having been poisoned byIsrael.
Flack started by considering what it would take for the
investigation to reach a particular outcome, and realized thathe did not know what the chances were of polonium tracessurviving in a body for several years. He started byinvestigating how polonium testing worked, and concludedthat enough polonium could in fact survive for it to be found
in the testing.Next, Flack considered what could cause polonium to
end up in the body. Israel poisoning Arafat could have doneit, but so could an Palestinian enemy that Arafat had. Therewas also the probability of the body being intentionallycontaminated after Arafat ’s death, by some faction trying to
frame Israel for the death. Each possibility made a positive
test result more probable, based on how probable those
individual possibilities were. Next Flack moved on toinvestigate what it would take for any of the possibilities tobe true. For the case of Israel poisoning Arafat, it requiredIsrael having access to polonium; Israel being willing to takethe risk of intentionally poisoning him; and Israel having themeans to poison Arafat with the polonium. These possibilitiesserved as starting points for researching the probability of the
‘Israel poisoned Arafat ’hypothesis, after which Flack would
break down and investigate what it would take for the otherhypotheses to be true.
Tetlock does not go into detail about the prerequisites for
being able to carry out such analysis —other than noting that it
is slow and effortful —but there are some considerations that
seem like plausible prerequisites. First, a person needs to haveenough general knowledge to generate different possibilities
for how an event could have come true. Next, they need the
ability to analyse and investigate those possibilities further,either personally acquiring the relevant domain knowledgefor evaluating their plausibility, or ﬁnding a relevant subject
matter expert. In this example, Flack familiarized himselfwith the science of polonium testing until he was satis ﬁed that
it would be possible to detect polonium traces from a longtime ago.
This suggests a general procedure which AI could also
follow in order to predict the possibility of something inwhich it does not yet have expertise. An AI that was trying topredict the outcome of some speci ﬁc question could work tap
into its existing general knowledge in an attempt to identifyrelevant causal factors; if it failed to generate them, it couldlook into existing disciplines which seemed relevant for thequestion. For each identi ﬁed possibility, it could branch off a
new subprocess to do research into that particular direction,
sharing information as necessary with a main process whosepurpose was to integrate the insights derived from all therelevant searches.
Such a capability for several parallel streams of attention
could provide a major advantage. A human researcher orforecaster who branches off to do research on a subquestionwill need to make sure that they do not lose track of the big
picture, and needs to have an idea of whether they are making
meaningful progress on that subquestion and whether itwould be better to devote attention to something else instead.To the extent that there can be several parallel streams ofattention, these issues can be alleviated, with a main streamfocusing on the overall question and substreams on speci ﬁc
subpossibilities.
How much could this improve on human forecasters?
Forecasters performed better when they were placed on teams
where they shared information between each other, whichsimilarly allowed an extent of parallelism in prediction-making,in that different forecasters could pursue their own angles and
4Though this claim needs to be treated with some caution, as no of ﬁcial
information about the intelligence analysts ’performance has been published.
The claim is based on Washington Post editor David Ignatius writing that ‘a
participant in the project ’had told him that superforecasters had ‘performed
about 30% better than the average for intelligence community analysts whocould read intercepts and other secret data ’(Ignatius 2013 ). The intelligence
community has neither con ﬁrmed nor denied this statement, and Philip
Tetlock has stated that he believes it to be true.
5A version of the scale which ranges between 0 and 1 is also commonly used.
6Phys. Scr. 92 (2017 )113001 Invited Comment
directions in exploring the problem. The differences between
individual forecasters and teams of forecasters with comparablelevels of training ranged between 0.05 and 0.10 Brier points atthe end of the ﬁrst year, and between 0.02 and 0.08 Brier points
at the end of the second year (Mellers et al 2014 ).I nh u m a n s
however, it seems likely that the extent of parallelism was
constrained by the fact that each forecaster had to indepen-
dently familiarize themselves with much of the same material,and that their ability to share knowledge between each otherwas limited by the speed of writing and reading. This suggestsa possibility for further improvement.
In general, accurate forecasting requires an ability to carry
out sophisticated causal modelling about a variety ofinteracting factors. Tetlock and Gardner emphasize the extent
to which superforecaster forums discuss many different ‘on the
one hand ’/‘on the other hand ’possibilities. In a discussion of
whether Saudi Arabia might agree to OPEC production cuts inNovember 2014, one superforecaster noted that Saudi Arabiahad large ﬁnancial reserves so could afford to let oil prices run
low. On the other hand, he noted, Saudi Arabia needed to raisetheir social spending to bolster the support for the monarchy,but yet again, Saudi Arabian rulers might view the act of trying
to control oil prices as futile. The superforecaster in question
concluded that the question ‘felt no-ish, 80% ’.(Saudis ended
up not supporting production cuts. )
This suggests that AI with suf ﬁcient hardware capability
could achieve considerable prediction ability by its capabilityto explore many different perspectives and causal factors atonce. The simulations of humans tend to be limited to aroundthree causal factors and six transition states (Klein 1999 ). The
discussion of the superforecasters clearly brought up many
more possibilities, and their accuracy suggests moderateability to integrate all those factors together. Yet commentssuch as ‘feels no-ish ’suggests that they still could not
construct a full-blown simulation in which the various causalfactors would have in ﬂuenced each other based on principled
rules which could be inspected, evaluated, and revised basedon feedback and accuracy. This seems especially plausible
given that Klein speculates the limits in the size of human
simulations to come from working memory limitations.
AI systems with larger working memory capacities might
be able to construct much more detailed simulations.Contemporary computer models can involve simulations withthousands or tens of thousands variables, though ﬂexibly
incorporating diverse models into a single simulation willprobably take considerably more memory and computing
power than what is used in today ’s models.
Example: parallel streams of attention with a LIDA-like
architecture
How could different streams of attention within AI share information
between each other? Recall that we have de ﬁned the development
of expertise as the ability to accumulate patterns which are used to
identify relevant cues and to indicate what predictions should be
derived out of those. A computational model for attention andconsciousness is Global Workspace Theory (Baars 2002 ,2005 ),o f
which a particular AI implementation is the LIDA model (Franklin
and Patterson 2006 , Franklin et al2014 , Madl et al2016 ). LIDA is(Continued. )
a model of the mind that is inspired by psychological and neu-
roscienti ﬁc research and attempts to capture its main mechanisms.
We can use LIDA to get a rough example of what having several
‘streams of attention ’would mean, and how information could be
exchanged between them. The purpose of this example is not to
suggest that an AI would necessarily work by this mechanism, butmerely to make the speculation about streams of attention slightlymore grounded in existing theories of how a general intelligence
(the human mind )might work. Thus, to the extent that LIDA is
correct as a model of human intelligence, and to the extent that theexample in this box is correct about LIDA allowing for there to be
several attentional streams at the same time, this provides some
information about it being possible to have several such streams inminds in general, and how that might concretely work.
LIDA works by means of an understand-attend-act cycle. In each
cycle, low-level sensory information is initially interpreted so as toassociate it with higher-level concepts to form a ‘percept ’, which
is then sent to a workspace. In the workspace, the percept activates
further associations in other memory systems, which are combinedwith the percept to create a Current Situational Model, an under-standing of what is going on at this moment.
The entirety of the Current Situational Model is likely to be too
complex for the agent to process, so it needs to select a part of it toelevate to the level of conscious attention to be acted upon. This is
carried out using ‘attention codelets ’, small pieces of code that
attempt to train attention on some particular piece of information,each with their own set of concerns of what is important. Attention
codelets with matching concerns form coalitions of what to attend,
competing against other coalitions. Whichever coalition ends upwinning the competition will have its chosen part of the CurrentSituational Model ‘become conscious ’, broadcast to the rest of the
system, and particularly Procedural Memory.
The Procedural Memory holds schemes, or templates of different
actions that can be taken in different contexts. Schemes which
include a context or an action that matches the contents of the
conscious broadcast become available as candidates for possibleactions. They are copied to the Action Selection mechanism,
which chooses a single action to perform. The selected action is
further sent to Sensory-Motor Memory, which contains informa-tion of how exactly to perform the action. The outcome of taking
this action manifests itself as new sensory information, beginning
the cognitive cycle anew.
Here is a description of how this process —or something like it —
might be applied in the case of AI seeking to predict the outcome
of a speci ﬁc question, such as the ‘will Saudi Arabia agree to oil
production cuts ’question discussed above. The decision to
consider this question has been made in an earlier cognitive
cycle, and information relevant to it is now available in the inner
environment and the Current Situational Model. The concepts ofSaudi Arabia and oil production trigger several associations in
the AI ’s memory systems, such as the fact that oil prices will
affect Saudi Arabia ’sﬁnancial situation, and that oil prices are
also in ﬂuenced by other factors such as global demand. Two
coalitions of attention codelets might form, one focusing on the
current ﬁnancial situation and another on in ﬂuences on oil prices.
In LIDA, these codelets would normally compete, and one of them
would win and trigger a speci ﬁc action, such as a deeper invest-
igation of Saudi Arabia ’sﬁnancial situation. In our hypothetical
AI however, it might be enough that both coalitions manage toexceed some threshold level of success, indicating them both to be
potentially relevant. In that case, new instances of the Procedural
7Phys. Scr. 92 (2017 )113001 Invited Comment
(Continued. )
Memory, Action Selection and Sensory-Motor Memory mechan-
isms might be initialized, with one coalition sending its contents to
theﬁrst set of instances and the other to another. These streams
could then independently carry out searches of the information
that was deemed relevant, also having their own local Situational
Models and Workspaces focusing on content relevant for thissearch. As they worked, these streams would update the variousmemory subsystems with the results of their learning, making new
associations and attention codelets available to all attentional
streams. Their functioning could be supervised by a general high-level attention stream, whose task was to evaluate the performance
of the various lower-level streams and allocate resources between
them accordingly.
These simulations do not necessarily need to incorporate
an exponentially increasing number of variables in order toachieve better prediction accuracy. As previously noted,superforecasters were more accurate at making predictions
300 days out than the rest of the forecasters in GJP were at
making predictions 100 days out. Given that at least some ofthe superforecasters only used a few hours a day on makingtheir predictions, and that they had many predictions to rate,they probably did not consider a vastly larger amount of
factors than the rest of the forecasters.
Klein (1999 )offers an example of a professor who used
three causal factors (the rate of in ﬂation, the rate of
unemployment, and the rate of foreign exchange )and a few
transitions to relatively accurately simulate how the Polisheconomy would develop in response to the decision to
convert from socialism to a market economy. In contrast, less
sophisticated experts could only name two variables (inﬂation
and unemployment )and not develop any simulations at
all, basing their predictions mostly on their ideologicalleanings.
Having large explicit models also allows for the models
to be adjusted in response to feedback. The professor ’s
estimate was in many extents correct, but failed to predict the
government being less ruthless and more cautious than it had
said it would be closing down unproductive plants. Thegovernment ’s caution could thus be added as an additional
variable to be considered for the next model. The addition ofthis variable alone might then considerably increase theaccuracy of the simulation.
Tetlock and Gardner report that the superforecasters used
highly granular probability estimates —carefully thinking about
whether the probability of an event was 3% as opposed to 4%,
for instance —and that the granularity actually contributed to
accuracy, with the predictions getting less accurate if they wererounded to the closest 5%. Given that such granularity wasachieved by integrating various possibilities and considera-tions, it seems like an ability to consider and integrate an evenlarger amount of possibilities might provide even increasedgranularity, and thus a prediction edge.
In summary, AI could be able to run vastly larger
simulations than humans could, with this possibility beingsubject to computing power limitations; given this, itssimulations could also be explicit, allowing it to adjust andcorrect them in response to feedback to provide improved
prediction accuracy; and it could have several streams ofattention running concurrently and sharing informationbetween each other. Existing evidence from human expertssuggests that large increases to prediction capability might notnecessarily need a large increase in the number of variables
considered, and that even small increases can provide
considerable additional gains.
The amount of predictive edge that this could give to an
AI as compared to a human or a group of humans is unclear,but humans do tend to prefer simple stories and explanationsthat are compact enough that all of the important details canbe kept in mind at once. Simple hypotheses often turn out tobe insuf ﬁcient because the world is more complicated than a
simple hypothesis allows for. Even in domains such as
engineering, where there exist formal ways of modellingthe entire domain, a task such as the design of a modernairplane or operating system contains too much complexityfor a single person to comprehend. While the impact ofuncertainty can never be eliminated, being able to take moreof the world ’s underlying complexity into account than
humans do, may provide an AI with a predictive edge at least
in some domains.
Rate of capability growth . How fast could AI develop
the ability to run comprehensive and large simulations?6
Creating larger simulations than humans have access to
seems to require extensive computational resources, either
from hardware or optimized software. As an additionalconsideration, we have previously mentioned limited workingmemory restricting the capabilities of humans, but humanworking memory is notthe same thing as RAM in computer
systems. If one were running a simulation of the human brainin a computer, one could not increase the brain ’s available
working memory simply by increasing the amount of RAM
the simulation had access to. Rather, it has been hypothesized
that working memory differences between individuals mayreﬂect things such as the ability to discriminate between
relevant and irrelevant information (Unsworth and
Engle 2007 ), which could be related to things like brain
network structure and thus be more of a software than ahardware issue
7. Yudkowsky (2013 )notes that if increased
intelligence would be a simple matter of scaling up the brain,
the road from chimpanzees to humans would likely have been
much shorter, as simple factors such as brain size can respondrapidly to evolutionary selection pressure.
Thus, advances in simulation size depend on progress in
both hardware and algorithms. Hardware progress in hard topredict, but advances in algorithmic capabilities seem doableusing mostly theoretical and mathematical research. This
6This section does not consider how fast the AI could develop the necessary
mental representations to be used in the simulations. That question will bediscussed in the next section.
7Though it is worth noting that gdoes correlate to some extent with brain
size, with a mean correlation of 0.4 in measurements that are obtained usingbrain imaging as opposed to external measurements of brain size (Rushton
and Ankney 2009 ). This would seem to suggest that the raw number of
neurons and thus ‘general hardware capacity ’would also be relevant.
8Phys. Scr. 92 (2017 )113001 Invited Comment
would require the development of expertise in mathematics,
programming, and theoretical computer science.
Much of mathematical problem-solving is about having a
library of procedures, reformulations, and heuristics that onecan try (Polya 1990 ), as well as developing a familiarity and
understanding of many kinds of mathematical results, which
one may then later on recognize as relevant. This seems like
the kind of task that relies strongly on pattern-matchingabilities, and might in principle be in reach by an advanceddeep reinforcement learning system that was fed a suf ﬁciently
large library of heuristics and worked proofs to let it developsuperhuman mathematical intuition
8. Modern-day theorem
provers often know what kinds of steps are valid, but notwhich steps are worth taking; merging them with the
‘artiﬁcial intuition ’of deep reinforcement learning systems
might eventually produce systems with superhuman mathe-matical ability.
Progress in this ﬁeld could allow AI systems to achieve
superhuman abilities in math research, considerably increas-ing their ability to develop more optimized software to takefull advantage of the available hardware. To the extent thatrelatively small increases in the number of variables
considered in a high-level simulation would allow for
dramatically increased prediction ability (as is suggested by
e.g. the superforecasters being better predictors with thrice theprediction horizon of less accurate forecasters ), moderate
increases in the size of the AI ’s simulations could translate to
drastic increases in terms of real-world capability.
Yudkowsky (2013 )notes that although the evolutionary
record strongly suggests that algorithmic improvements were
needed for taking us from chimpanzees to humans, the record
rules out exponentially increasing hardware always beingneeded for linear cognitive gains: the size of the human brainis only four times that of the chimpanzee brain. This furthersuggests that relatively limited improvements could allow fordrastic increases in intelligence.
Pattern recognition
The capability to run large simulations is not enough by itself.The AI also needs to acquire a suf ﬁciently large number of
patterns to be included in the simulations, to predict howdifferent pieces in the simulation behave.
Potential capability . When it comes to well-de ﬁned tasks,
current AI systems excel at pattern recognition, being able to
analyse vast amounts of data and build them into an overall
model, ﬁnding regularities that human experts never would
have. For instance, human experts would likely have beenunable to anticipate that men who ‘like’the Facebook page
‘Being Confused After Waking Up From Naps ’are more
likely to be heterosexual (Kosinski et al 2013 ). Similarly, the
Go-playing AI AlphaGo, whose good performance againstthe expert player Lee Sedol could to a large extent be
attributed to its built-up understanding of the kinds of board
patterns that predict victory, managed to make moves that Goprofessionals watching the game considered creative and
novel.
The ability to ﬁnd subtle patterns in data suggests that
AI systems might be able to make predictions in domains
which humans currently consider impossible to predict. Wepreviously discussed the issue of the (predictive )validity of
a domain, with domains being said to have higher validity if
‘there are stable relationships between objectively identi ﬁ-
able cues and subsequent events or between cues and the
outcomes of possible actions ’(Kahneman and Klein 2009 ).
Aﬁeld could also be valid despite being substantially
uncertain, with warfare and poker being listed as examplesofﬁelds that were valid (letting a skilled actor improve their
average performance )despite also being highly uncertain
(with good performance not being guaranteed even for a
skilled actor ).
We already know that the validity of a ﬁeld also depends
on an actor ’s cognitive and technological abilities. For
example, weather forecasting used to be a ﬁeld in which
almost no objectively identi ﬁable cues were available, relying
mostly on guesswork and intuition, but the development of
modern meteorological theory made it a much more valid
ﬁeld (Shanteau 1992 ). Thus, even ﬁelds which have low
validity to humans with modern-day capabilities, could
become more valid for more advanced actors.
A possible example of a domain that is currently
relatively low-validity, but which could become substantiallymore valid, is that of predicting the behaviour of individual
humans. Machine learning tools can already generate
personality pro ﬁles harvested from people ’s Facebook
‘likes ’
that are slightly more accurate than the pro ﬁles made by
people ’s human friends (Youyou et al 2015 ), and can be used
to predict private traits such as sexual orientation (Kosinski
et al 2013 ). This has been achieved using a relatively limited
amount of data and not much intelligence; a more
sophisticated modelling process could probably make even
better predictions from the same data.
Taleb (2007 )has argued for history being strongly
driven by ‘black swan ’events, events with such a low
probability that they are unanticipated and unprepared for,
but which have an enormous impact on the world. To theextent that this is accurate, it suggests limits on the validityof prediction. However, Tetlock and Gardner (2015 )argue
that while the black swans themselves may be unanticipated,
once the event has happened its consequences may be mucheasier to predict. Although superforecasters have shown no
ability to predict black swans such as the 9 /11 terrorist
attacks, they could predict the answers to questions like‘Will the United States threate n military action if the Taliban
do not hand over Osama bin Laden? ’and ‘Will the Taliban
comply? ’.
Thus, even though AI might be unable to predict some
very rare events, once those events have happened, it couldutilize its built-up knowledge of how people typically react to
different events in order to predict the consequences better
than anyone else.
8See Whalen (2016 )for preliminary work in this direction.
9Phys. Scr. 92 (2017 )113001 Invited Comment
Rates of capability growth
How quickly could AI acquire more detailed models? Here again
opinions differ. Hibbard (2016 )argues, based on Mahoney ’s
(2008 )argument for intelligence being a function of both
resources and knowledge, that explosive growth is unlikely.Benthall (2017 )makes a similar argument. On the other hand,
authors such as Bostrom (2014 )and Yudkowsky (2008 )suggest
the possibility for fast increases.
How to improve learning speed? We know that among humans,
there are considerable differences in the extent to which peoplelearn. Human cognitive differe nces have a strong neural and
genetic basis (Deary et al 2010 ), and strongly predict academic
performance (Deary et al 2007 ), socio-economic outcomes
(Strenze 2007 ), and job performance and the effectiveness of on-
the-job learning and experience (Gottfredson 1997b ).T h e r ea l s o
exist child prodigies who before adolescence achieve a level ofperformance comparable to an adult professional, without havingbeen able to spend comparable amounts of time training(Ruthsatz et al 2013 ). In general, some people are able to learn
faster from the same experiences, notice relevant patterns faster,
and continue learning from experience even past the point whereothers cease to achieve additional gains
9.
While there is so far no clear consensus on why some
people learn faster than others, there are some clear clues.Individual differences in cognitive abilities may be a result ofdifferences in a combination of factors, such as workingmemory capacity, attention control, and long-term memory
(Unsworth et al2014 ). Ruthsatz et al (2013 ), in turn, note that
‘child prodigies ’skills are highly dependent on a few features
of their cognitive pro ﬁles, including elevated general IQs,
exceptional working memories, and elevated attention todetail ’.
Many tasks require paying attention to many things at
once, with a risk of overloading the learner ’s working
memory before some of the performance has been automated.For an example, McPherson and Renwick (2001 )consider
children who are learning to play instruments, and note thatchildren who had previously learned to play anotherinstrument were faster learners. They suggest this to be inpart because the act of reading musical notation had becomeautomated for these children, saving them from the need to
process notation in working memory and allowing them to
focus entirely on learning the actual instrument.
This general phenomenon has been recognized in
education research. Complex activities that require multiplesubskills can be hard to master even if the students havemoderate competence in each individual subskill, as usingseveral of them at the same time can produce an over-whelming cognitive load (Ambrose et al 2010 , chapter 4 ).
Recommended strategies for dealing with this include
reducing the scope of the problem at ﬁrst and then building
up to increasingly complex scopes. For instance, ‘a piano
teacher might ask students to practice only the right handpart of a piece, and then only the left hand part, beforecombining them ’(ibid ).
An increased working memory capacity, which is
empirically associated with faster learning capabilities, could
theoretically assist in learning in allowing more things to be
comprehended simultaneously without them overwhelmingthe learner. Thus, AI with a large working memory couldlearn and master at once much more complicated wholes thanhumans.
Additionally, we have seen that a key part of ef ﬁcient
learning is the ability to monitor one ’s own performance and
to notice errors which need correcting; this seems in line with
cognitive abilities correlating with attentional control and
elevated attention to detail. McPherson and Renwick (2001 )
also remark on the ability of some students to play through apiece with considerably fewer errors on their second run-through than the ﬁrst one, suggesting that this indicates ‘an
outstanding ability to retain a mental representation of [K]
performance between run-throughs, and to use this as a basisfor learning from [K]errors ’. In contrast, children who
learned more slowly seemed to either not notice their
mistakes, or alternatively to not remember them when theyplayed the piece again.
Whatever the AI analogues of working and long-term
memory, attentional control, and attention to detail are, itseems at least plausible that these could be improved upon bydrawing exclusively on relatively theoretical research and in-house experiments. This might enable AI to both absorb vast
datasets, as current-day deep learning systems do, andalso
learn from superhumanly small amounts of data.
Limits of learning speed . How much can the human learning
speed be improved upon? This remains an open question.There are likely to be sharply diminishing returns at somepoint, but we do not know whether they are near the human
level. Human intelligence seems constrained by a number of
biological and physical factors that are unrelated to gainsfrom intelligence. Plausible constraints include the size of thebirth canal limiting the volume of human brains, the brain ’s9Readers who are familiar with the ‘deliberate practice ’literature may
wonder if that literature might not contradict these claims about the impact ofintelligence. After all, the deliberate practice research suggests that talent isirrelevant, and that deliberate, well-supervised training is the only thing thatmatters. However, as noted by the ﬁeld’s inventor, deliberate practice is a
concept that is applicable to some very speci ﬁc—one might even say arti ﬁcial
—domains. Deliberate practice can only be applied in ﬁelds in which there
are objective metrics, highly developed objectively-measurable expertise, andactive competition to improve the existing practices. Areas that do not qualifyare “anything in which there is little or no direct competition, such as
gardening and other hobbies, for instance, and many of the jobs in today ’s
workplace —business manager, teacher, electrician, engineer, consultant,
and so on ”, as there are no objective criteria for performance (Ericsson and
Pool 2016 ). Fields that have well-de ﬁned, objective criteria for good
performance are ones which are the easiest to master using even current-dayAI methods —in fact, they ’re basically the only ones that can be truly
mastered using current-day AI methods. A somewhat cheeky way to
summarize these results would be by saying that, in the kinds of ﬁelds that
could be mastered by AI methods that exhibit no general intelligence, generalintelligence isn ’t the most important thing. This even seems to be Ericsson ’s
own theoretical stance: that in these ﬁelds, general intelligence eventually
ceases to matter because the expert will have developed specialized mentalrepresentations that they can just rely on in every situation. So these resultsare not very interesting to those of us who are interested in domains that do
require general intelligence.
10Phys. Scr. 92 (2017 )113001 Invited Comment
extensive energy requirements limiting the overall amount of
cells, limits to the speed of signalling in neurons, anincreasing proportion of the brain ’s volume being spent on
wiring and connections (rather than actual computation )as
the number of neurons grows, and inherent unreliabilities inthe operation of ion channels (Fox 2011 ). There does not
seem to be any obvious reason for why the threshold for
diminishing gains from intelligence to learning speed wouldjust happen to coincide with the level of intelligence allowedby our current biology. Alternatively, there could have beendiminishing returns all along, but ones which still made itworthwhile for evolution to keep investing in additionalintelligence.
The available evidence also seems to suggest that within
the human range at least, increased intelligence continues to
contribute to additional gains. The Study of MathematicallyPrecocious Youth is a 50 year longitudinal study involvingover 5000 exceptionally talented individuals identi ﬁed between
1972 and 1997. Despite its name, many its participants aremore verbally than mathematically talented. The study has ledto several publications; among others, Wai et al (2005 )and
Lubinski and Benbow (2006 )examine the question of whether
ability differences within the top 1% of the human population
make a difference in life.
Comparing the top (Q4 )and bottom (Q1 )quartiles of two
cohorts within this study shows both to signi ﬁcantly differ
from the ordinary population, as well as from each other. Outof the general population, about 1% will obtain a doctoraldegree, whereas 20% of Q1 and 32% of Q4 did. 0.4% of Q1achieved tenure at a top-50 US university, as did 3% of Q4.
Looking at a 1 to 10 000 cohort, 19% had earned patents, as
compared to 7.5% of the Q4 group, 3.8% of the Q1 group, or1% of the general population.
It is important to emphasize that the evidence we have
reviewed so far does not merely mean that AI couldpotentially learn faster in terms of time: it also suggests thatthe AI could potentially learn faster in terms of training data .
The smaller datasets AI needs in order to develop accurate
models, the faster it can adapt to new situations.
Besides the considerations we have already discussed,
there seems to be potential for accelerated learning throughmore detailed analysis of experiences. For example, chessplayers improve most effectively by studying the games ofgrandmasters, and trying to predict what moves the grand-masters would have made in any situation. When thegrandmaster play deviates from the move that the student
would have made, the student goes back to try to see what
they missed (Ericsson and Pool 2016 ). This kind of detailed
study is effortful however, and can only be sustained forlimited amounts at a time. With enough computationalresources, the AI could routinely run this kind of analysison all sense data it received, constantly attempting to buildincreasingly detailed models that would correctly predictthe data.
How much interaction is needed? Some commentators,
such as Hibbard (2016 )argue that knowledge requiresinteraction with the world, so the AI would be forced to
learn over an extended period of time as the interaction
takes time.
From our previous review, we know that feedback is
needed for the development of expertise. However, one mayalso get feedback from studying static materials. As we noted
before, chess players spend more time studying published
matches and trying to predict the grandmaster moves —and
then getting feedback when they look up the next move and
have their prediction con ﬁrmed or falsi ﬁed—than they do
actually playing matches against live opponents (Ericsson and
Pool 2016 ). The Go-playing AlphaGo system did not achieve
its skill by spending large amounts of time playing human
opponents, but rather studying the games of humans andplaying games against itself (Silver et al 2016 ). And while
any individual human can only study a single game at a time,
AI systems could study a vast number of games in paralleland learn from all of them
10.
An important difference is that domains such as chess
and Go are formally speci ﬁed domains, which AI can
perfectly simulate. For a domain such as social interaction,the AI ’s ability to accurately simulate the behaviour of
humans is limited by its current competence in the domain.
While it can run a simulation based on its existing model ofhuman behaviour, predicting how humans would behave
based on that model, it needs external data in order to ﬁnd out
how accurate its prediction was.
This is not necessarily a problem however, given the vast
(and ever-increasing )amount of recorded social interaction
happening online. YouTube, e-mail lists, forums, blogs, andsocial media services all provide rich records of various kinds
of social interaction, for AI to test its predictive models
against without needing to engage in interaction of its own.Scienti ﬁc papers —increasingly available on an open access
basis —on topics such as psychology and sociology offer
additional information for the AI to supplement its under-standing with, as do various guides to social skills. All of this
information could be acquired simply by downloading it, with
the main constraints being the time needed to ﬁnd, download,
and process the data, rather than time needed for social
interactions.
As noted earlier, relatively crude statistical methods can
already extract relatively accurate psychological pro ﬁles out
of data such as people ’s Facebook ‘likes ’(Kosinski
et al 2013 , Youyou et al 2015 ), giving reason to suspect
that a general AI could develop very accurate predictiveabilities given the kind of a process described above.
Several other domains, such as software security and
mathematics seem similarly amenable to being masteredlargely without needing to interact with the world outside the
AI, other than searching for relevant materials. Some domains
such as physics would probably need novel experiments, butAI focusing on the domains that were the easiest and fastest
for it to master might ﬁnd suf ﬁcient sources of capability from
those alone.
10See Mnih et al (2016 )for a discussion of how incorporating parallel
learning improves upon on modern deep learning systems.
11Phys. Scr. 92 (2017 )113001 Invited Comment
Given the above considerations, it does not seem like
AI’s speed of learning would necessarily be strongly
interaction-constrained.
Conclusions
We set out to consider the fundamental practical limits of
intelligence, and the limits to how quickly an AI system couldacquire very high levels of capability.
Fictional representations of high intelligence often depict
a picture of geniuses as masterminds who have an almost
godlike prediction ability, laying out intricate multi-step plans
where every contingency is planned for in advance(TVTropes 2017a ). When discussing ‘superintelligent ’AI
systems, one might easily think that the discussion was pos-tulating something along the lines of those ﬁctional examples,
and rightly reject it as unrealistic.
Given what we know about the limits of prediction, for
AI to make a single plan which takes into account every
possibility is surely impossible. However, having reviewed
the science of human expertise, we have found that expertswho are good at their domains tend to develop powerfulmental representations which let them react to various situa-tions as they arise, and to simulate different plans and out-comes in their heads.
Looking from humans to AIs, we have found that AI
might be able to run much more sophisticated mental simu-
lations than humans could. Given human intelligence differ-
ences and empirical and theoretical considerations aboutworking memory being a major constraint for intelligence, theempirical ﬁnding that increased intelligence continues to
beneﬁt people throughout the whole human range, and the
observation that it would be unlikely for the theoretical limitsof intelligence to coincide with the biological and physicalconstraints that human intelligence currently faces, it seems
like AIs could come to learn considerably faster from data
than humans do. It also seems like in many domains, thiscould be achieved by using existing materials as a source offeedback for predictions, without necessarily being con-strained by time taken for interacting with the external world.
Thus, it looks that even though an AI system could not
make a single superplan for world conquest right from thebeginning, it could still have a superhuman ability to adapt
and learn from changing and novel situations, and react to
those faster than its human adversaries. As an analogy,experts playing most games can not precompute a winningstrategy right from the ﬁrst move either, but they can still
react and adapt to the game ’s evolving situation better than a
novice can, enabling them to win
11.
Many of the hypothetical advantages —such as a larger
working memory, the ability to consider more possibilities at
once, and the ability to practice on many training instances inparallel —that AI might have seem to depend on available
computing power. Thus the amount of hardware the AI had atits disposal could limit its capabilities, but there exists thepossibility of developing better-optimized algorithms byinitially specializing in ﬁelds such as programming and
theoretical computer science, which the AI might become
very good at.
One consideration which we have not yet properly
addressed is the technology landscape at the time when the AIarrives (Tomasik 2017 , section 7 ). If a general AI can be
developed, then various forms of sophisticated narrow AI willalso be in existence. Some of them could be used to detectand react to a general AI, and tools such as sophisticatedpersonal pro ﬁling for purposes of social manipulation will
likely already be in existence. Considering how these in ﬂu-
ence the considerations discussed here is an important ques-tion, but one which is outside the scope of this article.
In summary, even if AI could not create a complete
master plan from scratch, there seems to be a reasonablechance that could still come to substantially outperformhumans in many domains, developing and using superiorexpertise than what humans were capable of. How fast AI
systems could develop to such a level would depend on the
speed at which algorithmic and hardware improvementsbecame available. They could potentially be very fast, if e.g.the required algorithmic insights were more on the level ofscaling up the size of the AI ’s simulations and number of
attentional streams, rather than requiring any genuinely newideas compared to what allowed the AI to achieve a roughhuman level in the ﬁrst place.
Acknowledgments
Thank you to David Althaus, Stuart Armstrong, Gwern
Branwen, Bill Hibbard, David Krueger, Josh Marlow, CarlShulman, Brian Tomasik, and two anonymous reviewers onhelpful comments on this paper.
ORCID iDs
Kaj Sotala
https: //orcid.org /0000-0003-2140-826X
References
Anderson M 2010 Problem Solved: Unfriendly AI http://
hplusmagazine.com /2010/12/15/problem-solved-unfriendly-ai /
(Accessed: 27 September 2016 )
Ambrose S A, Bridges M W, DiPietro M, Lovett M C,
Norman M K and Mayer R E 2010 How Learning Works:
Seven Research-Based Principles for Smart Teaching Jossey-
Bass. http://teaching.temple.edu /sites/tlc/ﬁles/resource /
pdf/What%20Factors%20Motivate%20Students%20to%
20Learn\_.pdf
Amodei D, Olah C, Steinhardt J, Christiano P, Schulman J and
Mané D 2016 Concrete Problems in AI safety arXiv: 1606.
06565 [cs.AI ]11This is to say, while we concluded that the ﬁctional trope of a ‘Xanatos
Gambit ’(TVTropes 2017a )is unrealistic, a much more accurate description
of how a superintelligent AI actually acted could be the one of ‘Xanatos
Speed Chess ’, in which complex plans are constantly revised as the situation
progresses (TVTropes 2017b ).
12Phys. Scr. 92 (2017 )113001 Invited Comment
Baars B J 2002 The conscious access hypothesis: origins and recent
evidence Trends Cogn. Sci. 647–52
Baars B J 2005 Global workspace theory of consciousness: toward a
cognitive neuroscience of human experience Progress Brain
Res. 15045–53
Bengio Y, Courville A and Vincent P 2012 Representation learning:
a review and new perspectives arXiv: 1206.5538 [cs.LG ]
Benthall S 2017 Don ’t fear the reaper: refuting Bostrom ’s
superintelligence argument. arXiv: 1702.08495 [cs.AI ]
Bostrom N 2014 Superintelligence: Paths, Dangers, Strategies
(Oxford: Oxford University Press )
Buizza R 2002 Chaos and weather prediction, ECMWF http://
ecmwf.int /sites/default /ﬁles/Chaos%20and%20weather%
20prediction.pdf
Deary I J, Penke L and Johnson W 2010 The neuroscience of human
intelligence differences Nat. Rev. Neurosci. 11201–11
Deary I J, Strand S, Smith P and Fernandes C 2007 Intelligence and
educational achievement Intelligence 3513
Ericsson A and Pool R 2016 Peak: Secrets from the New Science of
Expertise (New York: Houghton Mif ﬂin Harcourt )
Fox D 2011 The Limits of Intelligence Sci. Am. (http://cs.virginia.
edu/~robins /The\_Limits\_of\_Intelligence.pdf )(July 2011 36 –43)
Franklin S, Madl T, D ’Mello S and Snaider J 2014 LIDA: a systems-
level Architecture for cognition, emotion, and learning IEEE
Trans. Auton. Mental Dev. 619–41
Franklin S and Patterson F G Jr 2006 The LIDA architecture:
Adding new modes of learning to an intelligent, autonomoussoftware agent. Presented at the Integrated Design and ProcessTechnology, IDPT-2006 http://ccrg.cs.memphis.edu
/assets /
papers /zo-1010-lida-060403.pdf
Future of Life Institute 2015 An Open Letter: Research Priorities for
Robust and Bene ﬁcial Arti ﬁcial Intelligence. https: //
futureo ﬂife.org /ai-open-letter /
Gottfredson L S 1997a Mainstream science on intelligence: an
editorial with 52 signatories, history, and bibliographyIntelligence 2413–23
Gottfredson L S 1997b Why g matters: the complexity of everyday
lifeIntelligence 2479–132
Hibbard B 2016 A defense of humans for transparency in arti ﬁcial
intelligence. http://ssec.wisc.edu /~billh/g/transparency\_
defense.html (Accessed: 28 September 2016 )
Ignatius D 2013 David Ignatius: more chatter than needed. The
Washington Post .https: //washingtonpost.com /opinions /
david-ignatius-more-chatter-than-needed /2013/11/01/
1194a984-425a-11e3-a624-41d661b0bb78\_story.html
Kahneman D and Klein G 2009 Conditions for intuitive expertise. A
failure to disagree Am. Psychol. 64515–26
Klein G 1999 Sources of Power: How People Make Decisions
(Cambridge, MA: MIT Press )(https: //books.google.de /
books?id =nn1kGwL4hRgC )
Kosinski M, Stillwell D and Graepel T 2013 Private traits and
attributes are predictable from digital records of human
behavior Proc. Natl Acad. Sci. USA 1105802 –5
Lake B M, Ullman T D, Tenenbaum J B and Gershman S J 2016
Building Machines That Learn and Think Like People .
arXiv:
1604.00289 [cs.AI ]
Lawrence N 2016 Future of AI 6. Discussion of ‘Superintelligence:
Paths, Dangers, Strategies. ’http://inverseprobability.com /
2016/05/09/machine-learning-futures-6 (Accessed: 27
September 2016 )
Legg S 2008 Machine super intelligence Doctoral Dissertation
Università della Svizzera italiana
Legg S and Hutter M 2007a Universal intelligence: a de ﬁnition of
machine intelligence Minds Mach. 17391–444
Legg S and Hutter M 2007b A collection of de ﬁnitions of
intelligence Frontiers Artif. Intell. Appl. 15717–24
Lubinski D and Benbow C P 2006 Study of mathematically
precocious youth after 35 years: uncovering antecedents for thedevelopment of math-science expertise Perspectives Psychol.
Sci.: J. Assoc. Psychol. Sci. 1316–45
Machery E 2009 Doing Without Concepts (Oxford: Oxford
University Press )
Machery E 2010 Précis of doing without concepts Behav. Brain Sci.
33195–206
Macnamara B N, Hambrick D Z and Oswald F L 2014 Deliberate
practice and performance in music, games, sports, education,
and professions: a meta-analysis Psychol. Sci. 251608 –18
Madl T, Franklin S, Chen K, Montaldi D and Trappl R 2016
Towards real-world capable spatial memory in the LIDA
cognitive architecture Biol. Inspired Cogn. Archit. 16
87–104
Mahoney M 2008 A model for recursively self improving programs
http://mattmahoney.net /rsi.pdf
Martela F 2016 Törmääkö tekoäly älykkyyden ylärajaan? Tivi.ﬁ.
http://tivi.ﬁ/blogit /tormaako-tekoaly-alykkyyden-ylarajaan-
6584349
McPherson G E and Renwick J M 2001 A longitudinal study of self-
regulation in children ’s musical practice Music Educ. Res. 3
169–86
Mellers B, Ungar L, Baron J, Ramos J, Gurcay B, Fincher K and
Tetlock P E 2014 Psychological strategies for winning ageopolitical forecasting tournament Psychol. Sci. 25
1106 –15
Mnih V, Badia A P, Mirza M, Graves A, Lillicrap T P, Harley T and
Kavukcuoglu K 2016 Asynchronous methods for deepreinforcement learning Int. Conf. on Machine Learning
(http://jmlr.org /proceedings /papers /v48/mniha16.pdf )
Mnih V, Kavukcuoglu K, Silver D, Rusu A A, Veness J,
Bellemare M G and Hassabis D 2015 Human-level controlthrough deep reinforcement learning Nature 518529–33
Müller V C and Bostrom N 2016 Future progress in arti ﬁcial
intelligence: a survey of expert opinion Fundamental Issues of
Artiﬁcial Intelligence (Switzerland: Springer International
Publishing )pp 553 –70
Polya G 1990 How to Solve It: A New Aspect of Mathematical
Method (London: Penguin Books Ltd )(https: //amazon.com /
How-Solve-Mathematical-Penguin-Science /dp/0140124993 )
Rushton J P and Ankney C D 2009 Whole brain size and general
mental ability: a review Int. Neurosci. 119691–731
Russell S, Dewey D and Tegmark M 2015 Research priorities for
robust and bene ﬁcial arti ﬁcial intelligence AI Mag. 36
105–14
Ruthsatz J, Ruthsatz K and Stephens K R 2013 Putting practice into
perspective: child prodigies as evidence of innate talentIntelligence 4560–5
Shanteau J 1992 Competence in experts: the role of task
characteristics Organ. Behav. Human Decision Process. 53
252–66
(http://citeseerx.ist.psu.edu /viewdoc /download?
doi=10.1.1.625.1063&rep =rep1&type =pdf )
Silver D, Huang A, Maddison C J, Guez A, Sifre L,
Van Den Driessche G and Dieleman S 2016 Mastering thegame of Go with deep neural networks and tree search Nature
529484–9
Soares N and Fallenstein B 2014 Agent Foundations for Aligning
Machine Intelligence with Human Interests: A TechnicalResearch Agenda. Machine Intelligence Research Institute
https: //intelligence.org /ﬁles/TechnicalAgenda.pdf
Sotala K and Yampolskiy R V 2015 Responses to catastrophic AGI
risk: a survey Phys. Scr. 90018001
Strenze T 2007 Intelligence and socioeconomic success: a meta-
analytic review of longitudinal research Intelligence 35
401–26
Taleb N N 2007 Black swans and the domains of statistics Am.
Statistician 61198–200
Taylor J, Yudkowsky E, LaVictoire P and Critch A 2016 Alignment
for advanced machine learning systems. Machine Intelligence
13Phys. Scr. 92 (2017 )113001 Invited Comment
Research Institute https: //intelligence.org /ﬁles/
AlignmentMachineLearning.pdf
Tetlock P E, Mellers B A and Rohrbaugh N 2014 Forecasting
Tournaments Tools for Increasing Transparency and
Improving the Quality of Debate Current Directions http://
cdp.sagepub.com /content /23/4/290.short
Tetlock P and Gardner D 2015 Superforecasting: The Art and
Science of Prediction (New York: Crown )
Tomasik B 2017 Arti ﬁcial Intelligence and Its Implications for
Future Suffering. https: //foundational-research.org /artiﬁcial-
intelligence-and-its-implications-for-future-suffering #reply-
to-bostroms-arguments-for-a-hard-takeoff (Accessed: 16
October 2017 )
Turing A M 1950 Computing machinery and intelligence Mind 59
433–60
TVTropes 2017a Xanatos Gambit. http://tvtropes.org /pmwiki /
pmwiki.php /Main /XanatosGambit
TVTropes 2017b Xanatos Speed Chess. http://tvtropes.org /
pmwiki /pmwiki.php /Main /XanatosSpeedChess
Unsworth N and Engle R W 2007 The nature of individual
differences in working memory capacity: active maintenancein primary memory and controlled search from secondary
memory Psychol. Rev. 114104–32
Unsworth N, Fukuda K, Awh E and Vogel E K 2014 Working
memory and ﬂuid intelligence: capacity, attention control, and
secondary memory retrieval Cogn. Psychol. 711–26
Wai J, Lubinski D and Benbow C P 2005 Creativity and
occupational accomplishments among intellectually precocious
youths: an age 13 to Age 33 longitudinal study J. Educ.
Psychol. 97484–92
Whalen D 2016 Holophrasm: a neural Automated Theorem Prover
for higher-order logic. arXiv: 1608.02644 [cs.AI ]
Youyou W, Kosinski M and Stillwell D 2015 Computer-based
personality judgments are more accurate than those made byhumans Proc. Natl Acad. Sci. USA 1121036 –40
Yudkowsky E 2008 Arti ﬁcial intelligence as a positive and
negative factor in global risk Global Catastrophic Risks ed
MMĆNick Bostrom (Oxford: Oxford University Press )
pp 308 –45 (https: //intelligence.org /ﬁles/AIPosNegFactor.pdf )
Yudkowsky E 2013 Intelligence Explosion Microeconomics (No.
2013-1 ). Machine Intelligence Research Institute https: //
intelligence.org /ﬁles/IEM.pdf
14Phys. Scr. 92 (2017 )113001 Invited Comment


It's not too soon to be wary of AI: We need to act now to protect humanity from future superintelligent machines

JAMIE CHUNG/TRUNK ARCHIVE; HAND: JUSTIN METZ
editor’s note: This article is based on a chapter of the author’s newly released book , Human Compatible: Artificial Intelligence
and the Problem of Control, published by Viking, an imprint of Penguin Publishing Group, a division of Penguin Random House.It’s Not Too Soon to Be Wary of AI
We need to act now to protect
humanity from future
superintelligent machines
by stuart russell
SPECTRUM.IEEE.ORG | OCT 2019 | 47
AI research is making great strides toward its long-term
goal of human-level or superhuman intelligent machines.
If it succeeds in its current form, however, that could well
be catastrophic for the human race. The reason is that the
“standard model” of AI requires machines to pursue a fixed
objective specified by humans. We are unable to specify the
objective completely and correctly, nor can we anticipate or prevent the harms that machines pursuing an incorrect objective will create when operating on a global scale with
superhuman capabilities. Already, we see examples such as social-media algorithms that learn to optimize click-through
by manipulating human preferences, with disastrous conse -
quences for democratic systems.
Nick Bostrom’s 2014 book Superintelligence: Paths,
 D
angers,
Strategies presented a detailed case for taking the risk seri-
ously. In what most would consider a classic example of Brit -
ish understatement, The Economist magazine’s review of
Bostrom’s book ended with: “The implications of introduc-
ing a second intelligent species onto Earth are far-reaching
enough to deserve hard thinking.”
Surely, with so much at stake, the great minds of today
are already doing this hard thinking—engaging in serious
debate, weighing up the risks and benefits, seeking solu-
tions, ferreting out loopholes in
solutions, and so on. Not yet, as
far as I am aware. Instead, a great
deal of effort has gone into various
forms of denial.
Some well-known AI research-
ers have resorted to arguments that
hardly merit refutation. Here are just
a few of the dozens that I have read
in articles or heard at conferences:
Electronic calculators are
superhuman at arithmetic.
Calculators didn’t take over the
world; therefore, there is no reason
to worry about superhuman AI.
Historically, there are zero
examples of machines killing
millions of humans, so, by induction,
it cannot happen in the future.
No physical quantity in the universe
can be infinite, and that includes
intelligence, so concerns about
superintelligence are overblown.
Perhaps the most common
response among AI researchers is to say that “we can always just
switch it off.” Alan Turing himself
raised this possibility, although he
did not put much faith in it:If a machine can think, it might think more intelligently than
we do, and then where should we be? Even if we could
keep the machines in a subservient position, for instance by
turning off the power at strategic moments, we should, as a
species, feel greatly humbled.... This new danger...is certainly something which can give us anxiety.
Switching the machine off won’t work for the simple rea-
son that a superintelligent entity will already have thought
of that possibility and taken steps to prevent it. And it will do
that not because it “wants to stay alive” but because it is pur -
suing whatever objective we gave it and knows that it will fail
if it is switched off. We can no more “just switch it off” than
we can beat AlphaGo (the world-champion Go-playing pro -
gram) just by putting stones on the right squares.
Other forms of denial appeal to more sophisticated
ideas, such as the notion that intelligence is multifaceted.
For example, one person might have more spatial intel -
ligence than another but less social intelligence, so we
cannot line up all humans in strict order of intelligence.
This is even more true of machines: Comparing the “intel-
ligence” of AlphaGo with that of the Google search engine
is quite meaningless.
48 | OCT 2019 | SPECTRUM.IEEE.ORG
Kevin Kelly, founding editor of Wired mag -
azine and a remarkably perceptive technol-
ogy commentator, takes this argument one
step further. In “The Myth of a Superhuman
AI,” he writes, “Intelligence is not a single
dimension, so ‘smarter than humans’ is a
meaningless concept.” In a single stroke,
all concerns about superintelligence are
wiped away.
Now, one obvious response is that a
machine could exceed human capabilities
in all relevant dimensions of intelligence. In
that case, even by Kelly’s strict standards, the machine would be smarter than a human.
But this rather strong assumption is not nec-
essary to refute Kelly’s argument.
Consider the chimpanzee. Chimpanzees
probably have better short-term memory
than humans, even on human-oriented tasks such as recalling sequences of dig -
its. Short-term memory is an important
dimension of intelligence. By Kelly’s argu-
ment, then, humans are not smarter than
chimpanzees; indeed, he would claim that
“smarter than a chimpanzee” is a meaning -
less concept.
This is cold comfort to the chimpanzees and other spe -
cies that survive only because we deign to allow it, and to
all those species that we have already wiped out. It’s also
cold comfort to humans who might be worried about being
wiped out by machines.
The risks of superintelligence can also be dismissed by argu-
ing that superintelligence cannot be achieved. These claims
are not new, but it is surprising now to see AI researchers
themselves claiming that such AI is impossible. For exam-
ple, a major report from the AI100 organization, “ Artificial
Intelligence and Life in 2030,” includes the following claim:
“Unlike in the movies, there is no race of superhuman robots
on the horizon or probably even possible.”
To my knowledge, this is the first time that serious AI
researchers have publicly espoused the view that human-
level or superhuman AI is impossible—and this in the mid-
dle of a period of extremely rapid progress in AI research,
when barrier after barrier is being breached. It’s as if a group
of leading cancer biologists announced that they had been fooling us all along: They’ve always known that there will
never be a cure for cancer.
What could have motivated such a volte-face? The report
provides no arguments or evidence whatever. (Indeed, what
evidence could there be that no physically possible arrange -
ment of atoms outperforms the human brain?) I suspect that
the main reason is tribalism—the instinct to circle the wagons
against what are perceived to be “attacks” on AI. It seems
odd, however, to perceive the claim that superintelligent AI
is possible as an attack on AI, and even odder to defend AI
by saying that AI will never succeed in its goals. We cannot insure against future catastrophe simply by betting against
human ingenuity.
If superhuman AI is not strictly impossible, perhaps it’s
too far off to worry about? This is the gist of Andrew Ng’s
assertion that it’s like worrying about “overpopulation on
the planet Mars.” Unfortunately, a long-term risk can still be cause for immediate concern. The right time to worry about
a potentially serious problem for humanity depends not just
on when the problem will occur but also on how long it will take to prepare and implement a solution.
For example, if we were to detect a large asteroid on course
to collide with Earth in 2069, would we wait until 2068 to
start working on a solution? Far from it! There would be a
worldwide emergency project to develop the means to coun-
ter the threat, because we can’t say in advance how much time is needed.
Ng’s argument also appeals to one’s intuition that it’s
extremely unlikely we’d even try to move billions of humans to Mars in the first place. The analogy is a false one, how -
ever. We are already devoting huge scientific and technical
resources to creating ever more capable AI systems, with
very little thought devoted to what happens if we succeed. A
more apt analogy, then, would be a plan to move the human race to Mars with no consideration for what we might breathe,
drink, or eat once we arrive. Some might call this plan unwise.
Another way to avoid the underlying issue is to assert that
concerns about risk arise from ignorance. For example, here’s
Oren Etzioni, CEO of the Allen Institute for AI, accusing Elon
Musk and Stephen Hawking of Luddism because of their calls
to recognize the threat AI could pose:Switching the
machine off won’t work for the simple reason that a superintelligent entity will already have thought of that possibility and taken steps to prevent it.
SPECTRUM.IEEE.ORG | OCT 2019 | 49 ILLUSTRATION BY Justin Metz
The notion that anyone mentioning risks is “against
AI” seems bizarre. (Are nuclear safety engineers “against
electricity”?) But more importantly, the entire argument
is precisely backwards, for two reasons. First, if there
were no potential benefits, there would be no impetus
for AI research and no danger of ever achieving human-
level AI. We simply wouldn’t be having this discussion at
all. Second, if the risks are not successfully mitigated, there
will be no benefits.
The potential benefits of nuclear power have been greatly
reduced because of the catastrophic events at Three Mile
Island in 1979, Chernobyl in 1986, and Fukushima in 2011.
Those disasters severely curtailed the growth of the nuclear
industry. Italy abandoned nuclear power in 1990, and
 B
elgium, Germany, Spain, and Switzerland have announced
plans to do so. The net new capacity per year added from
1991 to 2010 was about a tenth of what it was in the years
immediately before Chernobyl.
Strangely, in light of these events, the renowned cog -
nitive scientist Steven Pinker has argued that it is inap -
propriate to call attention to the risks of AI because the
“culture of safety in advanced societies” will ensure that
all serious risks from AI will be eliminated. Even if we
disregard the fact that our advanced culture of safety has
produced Chernobyl, Fukushima, and runaway global
warming, Pinker’s argument entirely misses the point.
The culture of safety—when it works—consists precisely
of people pointing to possible failure modes and finding
ways to prevent them. And with AI, the standard model
is the failure mode.
Pinker also argues that problematic AI behaviors arise
from putting in specific kinds of objectives; if these are left
out, everything will be fine:
AI dystopias project a parochial
alpha-male psychology onto the concept
of intelligence. They assume that
superhumanly intelligent robots would
develop goals like deposing their masters
or taking over the world.
Yann LeCun, a pioneer of deep learning
and director of AI research at Facebook,
often cites the same idea when downplay -
ing the risk from AI:
There is no reason for AIs to have
self-
 pr
eservation instincts, jealousy, etc....
AIs will not have these destructive “emotions”
unless we build these emotions into them.
Unfortunately, it doesn’t matter whether
we build in “emotions” or “desires” such
as self-preservation, resource acquisition,
knowledge discovery, or, in the extreme case, Those who argue
the risk is negligible
have failed to explain
why superintelligent
AI will necessarily
remain under
human control.At the rise of every technology innovation, people have
been scared. From the weavers throwing their shoes in
the mechanical looms at the beginning of the industrial
era to today’s fear of killer robots, our response has
been driven by not knowing what impact the new
technology will have on our sense of self and our
livelihoods. And when we don’t know, our fearful minds
fill in the details.
Even if we take this classic ad hominem argument at face
value, it doesn’t hold water. Hawking was no stranger to sci-
entific reasoning, and Musk has supervised and invested in
many AI research projects. And it would be even less plau-
sible to argue that Bill Gates, I.J. Good, Marvin Minsky, Alan
Turing, and Norbert Wiener, all of whom raised concerns,
are unqualified to discuss AI.
The accusation of Luddism is also completely misdirected.
It is as if one were to accuse nuclear engineers of Luddism
when they point out the need for control of the fission reac-
tion. Another version of the accusation is to claim that men-
tioning risks means denying the potential benefits of AI. For example, here again is Oren Etzioni:
Doom-and-gloom predictions often fail to consider the
potential benefits of AI in preventing medical errors, reducing car accidents, and more.
And here is Mark Zuckerberg, CEO of Facebook, in a recent
media-fueled exchange with Elon Musk:
If you’re arguing against AI, then you’re arguing against safer
cars that aren’t going to have accidents. And you’re arguing against being able to better diagnose people when they’re sick.
50 | OCT 2019 | SPECTRUM.IEEE.ORG
taking over the world. The machine is going to have those emo -
tions anyway, as subgoals of any objective we do build in—and
regardless of its gender. As we saw with the “just switch it off”
argument, for a machine, death isn’t bad per se. Death is to
be avoided, nonetheless, because it’s hard to achieve objec-
tives if you’re dead.
A common variant on the “avoid putting in objectives”
idea is the notion that a sufficiently intelligent system will
necessarily, as a consequence of its intelligence, develop the
“right” goals on its own. The 18th-century philosopher David
Hume refuted this idea in A Treatise of Human Nature. Nick Bostrom, in Superintelligence, presents Hume’s position as
an orthogonality thesis:
Intelligence and final goals are orthogonal: more or less
any level of intelligence could in principle be combined
with more or less any final goal.
For example, a self-driving car can be given any particu-
lar address as its destination; making the car a better driver
doesn’t mean that it will spontaneously start refusing to go
to addresses that are divisible by 17.
By the same token, it is easy to imagine that a general-
 purpose

intelligent system could be given more or less any objective to pursue—including maximiz-
ing the number of paper clips
or the number of known dig -
its of pi. This is just how rein -
forcement learning systems
and other kinds of reward opti-
mizers work: The algorithms
are completely general and accept any reward signal. For
engineers and computer scien-tists operating within the stan-
dard model, the orthogonality
thesis is just a given.
The most explicit critique
of Bostrom’s orthogonality thesis comes from the noted
roboticist Rodney Brooks, who
asserts that it’s impossible for a
program to be “smart enough
that it would be able to invent
ways to subvert human society
to achieve goals set for it by
humans, without understand-
ing the ways in which it was
causing problems for those
same humans.”
Unfortunately, it’s not
only possible for a program to behave like this; it is, in fact, inevitable, given the
way Brooks defines the issue.
Brooks posits that the optimal plan for a machine to “achieve
goals set for it by humans” is causing problems for humans. It
follows that those problems reflect things of value to humans that were omitted from the goals set for it by humans. The optimal plan being carried out by the machine may well cause
problems for humans, and the machine may well be aware
of this. But, by definition, the machine will not recognize those problems as problematic. They are none of its concern.
In summary, the “skeptics”—those who argue that the risk
from AI is negligible—have failed to explain why superintelli-
gent AI systems will necessarily remain under human control;
and they have not even tried to explain why superintelligent
AI systems will never be developed.
Rather than continue the descent into tribal name-calling
and repeated exhumation of discredited arguments, the AI
community must own the risks and work to mitigate them.
The risks, to the extent that we understand them, are neither
minimal nor insuperable. The first step is to realize that the
standard model—the AI system optimizing a fixed objective—
must be replaced. It is simply bad engineering. We need to
do a substantial amount of work to reshape and rebuild the
foundations of AI.
n
↗ POST YOUR COMMENTS at https:/ /spectrum.ieee.org/risksofai1019
SPECTRUM.IEEE.ORG | OCT 2019 | 51 ILLUSTRATION BY Justin Metz