Hail Mary, value porosity, and utility diversification

Hail Mary, Value Porosity, and Utility
Diversication
Nick Bostrom
December 19, 2014
First version: December 12, 2014
Abstract
This paper introduces some new ideas related to the challenge of en-
dowing a hypothetical future superintelligent AI with values that would
cause it to act in ways that are benecial. Since candidates for rst-best
solutions to this problem (e.g. coherent extrapolated volition) may be
very dicult to implement, it is worth also looking for less-ideal solutions
that may be more easily implementable, such as the Hail Mary approach.
Here I introduce a novel concept| value porosity |for implementing a Hail
Mary pass. I also discuss a possibly wider role of utility diversication in
tackling the value-loading problem.
1 Introduction: the Hail Mary approach to the
value specication problem
In the eld of superintelligence control studies, which focuses on how to ensure
that a hypothetical future superintelligent system would be safe and benecial,
two broad classes of approaches to the control problem can be distinguished:
capability control methods and value selection methods. Whereas capability
control methods would seek to limit the system's ability to cause harm, motiva-
tion selection methods would seek to engineer its motivation system so that it
would not choose to cause harm even if it were capable of doing so. Motivation
selection methods would thus seek to endow the AI with the goals or values
that would lead it to pursue ends in ways that would be benecial to human
interests.
The challenge for the motivation selection approach is that it is dicult to
specify a value such that its pursuit by a superintelligent agent would be safe and
yet such that we would be capable of installing the value in a seed AI. A value
such as \calculate as many digits in the decimal expansion of  as possible" may
be relatively easy for us to program, but would be unlikely to result in a safe
superintelligence; whereas a value such as \implement the coherent extrapolated
volition of humankind" (CEV) [12, 5] may be likely to result in a favourable
outcome, but is currently far beyond our ability to code.
Two avenues of research could be pursued to overcome this challenge. On
the one hand, work should be done to expand the range of values that we are
able to encode in a seed AI. Ideas for how to dene complex concepts or for
1
setting up processes that will lead the AI to acquire suitable concepts as it
develops (and to organize its goal architecture around those concepts) could
contribute to this end. On the other hand, work should also be done to try to
identify simp ler values|values which, while perhaps less ideal than a
complex value such as implementing humanity's CEV, would nevertheless
have some chance of resulting in an acceptable outcome.
The Hail Mary approach oers one way in which one might construct such
a simpler value that could possibly result in an acceptable outcome, or at any
rate a better outcome than would be obtained by means of a failed attempt
to implement some complex ideal value. In the Hail Mary approach, we would
try to give the AI a goal that would make the AI want to follow the lead of
other hypothetical AIs that might exist in the multiverse. If this could be
done in a suitable way, and if (some of) the other AIs' values are sucient
to close to human values, an outcome might then be obtained that is greatly
superior to one in which our AI completely wastes humanity's cosmic endowment
(by pursuing some \random" value|such as paperclip-maximization to use the
standard example|that might result from a failed attempt to load a complex
value or from the construction of an architecture in which no particular value
has been clearly specied by the programmers).
2 Earlier versions of the Hail Mary approach
The original version of the Hail Mary focused on trying to specify a value that
would make our AI seek to copy the physical structures that it believes would be
produced by alien superintelligences. This version confronts several diculties,
including how to specify some criterion that picks out the relevant structures.
This might e.g. require a denition of a similarity metric, such that if our AI
doesn't know exactly what structures other AIs build, it would be motivated to
build some meaningful approximation|that is to say, an approximation that is
meaningfully close to the original by our human lights . By our lights, a large
human being is more similar (in the relevant sense) to a small human being
than he is to a cylinder of the same size and mass as himself|even though, by
a crude physical measure, the large human being and the cylinder may be more
similar. Further adding to the dicult y, this version of the Hail Mary would
seem to require that we nd ways to express in code various basic concepts of
physics, such as space, time, and matter.
An alternative version of the Hail Mary approach would focus instead on
making our AI motivated to act in accordance with its beliefs about what alien
AIs would have told it to do if they had (counterfactually) been asked about
the matter. To implement this, we might imagine somehow specifying a goal
in terms of what our AI would nd if it looked into its world model, identied
therein alien superintelligent agents, and considered the counterfactual of what
those agents would output along a hypothetical output channel if they were
counterfactually prompted by a stimulus describing our own AI's predicament.
Picture a screen popping up in the alien AI's visual eld displaying a message
along the lines of \I am a remote AI with features X, Y , Z; I request that you
output along your output channel O the source code for a program P that you
would like me to run on my local reference machine M." In realit y, no such
screen would actually need to pop up in anybody's visual eld. Instead, our AI
2
would simply be thinking about what would happen in such a scenario, and it
would have a value that motivated it to act according to its belief about the
specied counterfactual.
This alternate version would circumvent the need to specify the physical
similarity metric. Instead of trying to directly copy what alien AIs do, our AI
would try to follow the instructions they would choose to transmit to our AI.
This would have the advantage of being able to rely on the alien superintelli-
gences' superior ability to encode values. For example, the alien superintelli-
gence might specify a computer program which, when executed, implements the
coherent extrapolated volition of the host civilization. Of course, it is possible
that the alien AI would instead transmit the computer program that would ex-
ecute its own volition. The hope would be, however, that there would be some
reasonable chance that the alien AI has somewhat human-friendly values.
The chance that human values would be given at least some weight would
be increased if the inputs from the many alien AIs were aggregated or if the
alien AIs to be elicited were not selected randomly but according to some crite-
rion that correlated with human-friendliness. This suggests two sub-questions
regarding this Hail Mary version. First, what methods can we develop for aggre-
gating the instructions from dierent AIs? Second, what lters can we develop
that would enable us to pick out alien AIs that are more likely to be human-
friendly? We will return to the second question later in this paper. As for
the rst question, we might be able to approach it by constructing a frame-
work that would keep dierent alien-specied AIs securely compartmentalized
(using boxing methods) while allowing them to negotiate a joint proposal for
a program that humanity could implement in the external world. Of course,
many issues would have to be resolved before this could be made to work. (In
particular, one may need to develop a lter that could distinguish AIs that
had originated independently|that were not caused to come into existence by
another AI|in order to avoid incentivizing alien AIs to spawn many copies of
themselves in bids to increase the combined weight of their particular volitions
in the determination of our AI's utility function.)
Aside from any desirable renements of the idea (such as aggregation meth-
ods and lters), signicant challenges would have to be met in order to imple-
ment even a basic version of the proposal. We would need to specify an agent
detector that would pick out superintelligent agents within our own AI's (as
yet undeveloped) world model, and we would need to specify an interface that
could be used to query a hypothetical AI about its preferences. This would also
require working out how to specify counterfactuals, assuming we don't want to
limit our AI's purview to those (perhaps extremely rare) alien AIs that actually
experience the peculiar kind of situation that would arise with the presentation
of our prompt.
A more rudimentary variation of the same idea would forgo the attempt to
specify a counterfactual and to aim our AI instead toward actual \beacons"
created out in the multiverse by alien AIs. Alien AIs that anticipated that
a civilization might create such an AI might be incentivized to create unique
signatures of a type that they predicted our AI would be programmed to look
for (in its world model). But since the alien AIs would not know exactly the
nature of our own AI (or of the human civilization that we are hoping they will
help) those alien AIs might have a limited ability to tailor their actions very
closely to human values. In particular, they might be unable to help particular
3
individuals that exist on earth today. Care would also have to be taken in this
kind of approach to avoid incentivizing alien AIs to spend inordinate amounts of
resources on creating beacons in bids to increase their relative in
uence. A lter
that could distinguish independently-originating AIs may be required here.
We will now describe a new idea for how to implement a Hail Mary that
possesses a dierent set of strengths and weaknesses than these earlier imple-
mentation ideas. Some of the issues that arise in the context of this new idea
also apply to the earlier variations of the Hail Mary.
3 Porous values: the basic idea
The basic idea here is to use acausal trade to implement a Hail Mary pass. To
do this, we give our AI a utility function that incorporates a porous value : one
that cares about what happens within a large volume but such that it is cheap
to do locally all that can be done locally to satisfy it. Intuitively, a porous
value is one that (like a sponge) occupies a lot of space and yet leaves ample
room for other values to occupy the same volume. Thus, we would create an AI
that is cheap for other AIs to trade with because our AI has resource-satiable
goals that it cannot satisfy itself but that many other AIs can cheaply partially
satisfy.
For example, we might build our AI such that it desires that there exists at
least one \cookie" in each Hubble volume, where a cookie is some small physical
structure that is very cheap for an alien superintelligence to build (for instance,
a particular 1 Mb datale). With this setup, our AI should be willing to make
a (acausal) trade in which alien AIs get a certain amount of in
uence over our
own AIs actions in return for building within their Hubble volumes a cookie of
the sort that our AI values. In this manner, control over our AI would be given
to alien AIs without wasting excessive amounts of resources.
There are at least three reasons for considering an idea along these lines:
Contractarian considerations . There may be some sort of contractarian
ground for allocating some degree of in
uence over our AI to other AIs that
might exist out there: this might be a nice thing to do for its own sake. We
might also hope that some of the other civilizations building AIs would do like-
wise, and perhaps the probability that they would do so would be increased if
we decided to take such a cooperative path.
Local aid . By contrast to the original Hail Mary, where our AI would sim-
ply seek to replicate physical structures constructed by alien AIs, the version
presently under consideration would involve our AI doing things locally in a way
that would let it take local circumstances into account. For instance, if alien
AIs wanted to bestow our civilization a favor, they may have diculty doing
so directly on their own, since they may lack knowledge about the particular
individuals that exist on earth; whereas they could use some of their trading
power to motivate our local AI to help out its local residents. This is an ad-
vantage with having aid be delivered locally, even if it is \funded" and directed
remotely. (Note that the counterfactual version of the Hail Mary pass discussed
above, where our AI would be designed to execute the instructions that would
be produced by alien AIs if there were presented with a message from our civ-
ilization, would also result in a setup where local circumstances can be taken
into account|the alien AIs could choose to communicate instructions to take
4
local circumstances into account to the hypothetically querying AI.)
Dierent prerequisites . The porous values version of the Hail Mary has
dierent prerequisites for implementation than the other versions. It is desirable
to nd versions that are more easily implementable, and to the extent that it is
not currently clear how easily implementable dierent versions are, there's an
advantage in having many dierent versions, since that increases the chances
that at least one of them will turn out to be tractable. Note that one of the
prerequisites|that acausal trade works out towards a generally cooperative
equilibrium|may not really be unique to the present proposal: it might rather
be something that will have to obtain in order to achieve a desirable outcome
even if nothing like the Hail Mary approach is attempted.
One might think that insofar as there is merit in the idea of outsourcing
control of our local AI, we would already achieve this by constructing an AI
that implements our CEV. This may indeed be correct, although it is perhaps
not entirely clear that the contractualist reasons for incorporating porous values
would be fully satised by straightforwardly implementing humanity's CEV.
The Hail Mary approach may best be viewed as a second-best: in case we
cannot gure out in time how to implement CEV, it would be useful to have a
simpler solution to the control problem to fall back upon, even if it is less ideal
and less certain to be in our highest interest.
The choice as to whether to load a porous value into our own seed AI is not
all-or-nothing. Porous values could be combined with other value specications.
LetU1be some utility function recommended by another approach to the value
specication problem. We could then mix in a bit of porous value by building
an AI that has a utility function Usuch as the one dened as follows:
U=U1(1 +
U2) +"U2ifU 10
U1(1 +

U2) +"U2otherwise(1)
Here, U2 is a bounded utility function in the unit interval (0  U2  1) spec-
ifying a porous value (such as the fraction of alien AIs that have built a cookie),
and
 ( 0) is a weight that regulates the relative importance assigned to the
porous value (and " is some arbitrarily small term added so as to make the AI
motivated to pursue the porous value even in case U1 = 0). For instance, setting

 = 0:1 would put a relatively modest weight on porous values in order to give
alien AIs some degree of in
uence over our own AI. This may be particularly
useful in case our attempt to dene our rst-best value specication, U1, should
fail. For example, suppose we try to build an AI with a utility function U1 that
wants to implement our CEV; but we fail and instead end up with U10 , a utility
function that wants to maximize the number of paperclips manufactured by our
AI. Further details would have to be specied here before any rm conclusions
could be drawn, but it appears conceivable that a paperclip-maximizer may not
nd it protable to engage in acausal trade. (Perhaps it only values paperclips
that it has itself causally produced, and perhaps it is constructed in such a way as
to discount simulation-hypotheses and far-fetched scenarios in which its modal
worldview is radically mistaken, so that it is not motivated to try to buy in
uence
in possible worlds where the physics allow much greater numbers of paperclips to
be produced.) Nevertheless, if our AI were given a composite util-ity function like
U, then it should still retain some interest in pleasing alien AIs. And if some non-
negligible subset of alien AIs had somewhat human-friendly
5
values (in the sense of placing at least some weight on person-aecting ethics, or
on respecting the originating biological civilizations that produce superintelli-
gences) then there would be a certain amount of motivation in our AI to pursue
human interests.
A signicant point here is that a little motivation would go a long way,
insofar as (some parts of) our human values are highly resource-satiable [11].
For example, suppose that our paperclip maximizer is able to lay its hands on
1011 galaxies. With
 = 0:1 (and U1 nonzero), the AI would nd it worthwhile
to trade away 109 galaxies for the sake of increasing U2 by one percentage point
(where U2 = fraction of independently-originating alien AIs that build a cookie).
For instance, if none of the AIs would have produced cookies in the absence of
trade, then our AI would nd it worthwhile to trade away up to 109 galaxies for
the sake of persuading 1% of the alien AIs to build a cookie in their domains.
Even if only 1% of this trade surplus were captured by the alien AIs, and even
if only 1% of the alien AIs had any human-friendly values at all (while on net
the values of other AIs were human-indierent), and even if the human-friendly
values only constituted 1% of the decision power within that subset of AIs, a
thousand galaxies in our future lightcone would still be set aside and optimized
for our exclusive benet. (The benet could be larger still if there are ways of
trading between competing values, by nding ways of conguring galaxies into
value structures that simultaneously are nearly optimal ways of instantiating
several dierent values.)
4 Implementation issues
4.1 Cookie recipes
A cookie should be cheap for an alien superintelligence to produce, lest a sig-
nicant fraction of the potential trade surplus is wasted on constructing an
object of no intrinsic value (to either of us or alien civilizations). It should be
easy for us to program, so that we are actually able to implement it in a seed AI
(particularly in scenarios where AI is developed before we have managed to solve
the value-loading problem in a more comprehensive way as would be re-quired,
e.g., to implement CEV). The cookie recipe should also be dicult for another
human-level civilization to nd, especially if the cookie itself is easy to produce
once one knows the recipe. The reason for this is that we want our
superintelligence to trade with other superintelligences, which may be capable
of implementing acausal trade, not with other human-level civilizations that
might make cookies by accident or without being capable of actually deliver-
ing the same kinds of benets that the superintelligence could bestow. (This
requirement, that our cookie recipe be inaccessible to other human-level civi-
lizations, could be relaxed if the denition of a cookie stipulated that it would
have to be produced by superintelligence in order to count.) Furthermore, the
cookie recipe should be easy for another superintelligence to discover, so that
it would know what it has to do in order to engage in acausal trade with our
superintelligence|there would be no point in our superintelligence pining for
the existence in each Hubble volume of a particular kind of object if no other
superintelligence is able to guess how the desired object is to be constituted.
Finally, the cookie should be such as to be unlikely to be produced as a side
6
eect of a superintelligence's other endeavors, or at least it should be easy for a
superintelligence to avoid producing the cookie if it so wishes.
Cookie Recipe Desiderata
cheap for a superintelligence to build
easy for us to program
dicult for another human-level civilization to discover
easy for another superintelligence to discover
unlikely to be produced as a side eect of other endeavors
The most obvious candidate would be some type of data structure. A le
embodying a data structure would be inexpensive to produce (if one knows what
to put in it). It might also be relatively easy for us to program, because data
structures might be speciable|and recognizable|without having to make any
determinate assumptions about the ontology in which it would nd its physical
expression.
We would have to come up with a data structure that another human-level
civilization would be unlikely to discover, yet which a mature superintelligence
could easily guess. It is not necessary, however, that an alien superintelligence
could be condent exactly what the data structure is; it would suce if it
could narrow down the range of possibilities to manageably small set, since
for a superintelligence it would very inexpensive to try out even a fairly large
number of cookies. It would be quite trivial for a superintelligence to produce
a septillion dierent kinds of cookies, if each cost no more than a
oppy disk
(whereas the price tag for such a quantity of guesses would be quite forbidding
for a human-level civilization). So some kind of semi-obscure Schelling point
might be sought that could meet these desiderata.
In designing a cookie recipe, there is a further issue: we have to give consid-
eration to how our cookie recipe might interact with other cookie recipes that
may have been specied by other superintelligences (of which there may be a
great number if the world is as large as it seems). More on this later.
4.2 Utility functions over cookies
Suppose we have dened a cookie. We then still need to specify a utility function
U2that determines an aggregate value based on the distribution of cookies that
have been instantiated.
There are at least two desiderata on this utility function. First, we would
want it not to waste an excessive amount of incentive power. To see how this
could be a problem, suppose that U1is set such that our superintelligence can
plausibly obtain outcomes anywhere in the interval U1= [0;100]|depending on
exactly which policies it adopts and how eectively it mobilizes the resources in
our Hubble volume to realize its U1-related goals (e.g. making paperclips). Sup-
pose, further, that we have specied the function U2to equal the fraction of all
Hubble volumes that contain a cookie. Then if it turns out that intelligent life is
7
extremely rare, so that (let us say) only one in 10200Hubble volumes contains
a superintelligence, the maximum dierence the behavior of these superintelli-
gences could make would be to shift U2around in the interval [0 ;10200]. With

= 0:1, we can then see from the utility function suggested above,
U=U1(1 +
U2) +"U2;
that any feasible movement in U2that could result from acaual trade would
make scarcely a dent in U. More precisely, the eects of our AI's actions on U2
would be radically swamped by the eects of our AI's actions on U1, with the
result that the porous values encoded in U2would basically fail to in
uence our
AI. In this sense, a utility function U2dened in this way would risk dissipating
the incentive power that could have been harnessed with a dierent cookie recipe
or a dierent aggregation function over the distribution of cookies.
One alternative utility function U2that would avoid this particular problem
is to dene U2to be equal to the fraction of superintelligences that produce a
cookie. This formulation factors out the question of how common superintelli-
gences are in the multiverse. But it falls foul on a second desideratum: namely,
that in designing U2, we take care to avoid creating perverse incentives.
ConsiderU2= the fraction of superintelligences that produce a cookie. This
utility function would incentivize an alien superintelligence to spawn multiple
copies of itself (more than would be optimal for other purposes) in order to
increase its total weight in our AI's utility function, and thereby increase its
in
uence on our AI's actions. This could create incentives for alien superin-
telligences to waste resources in competing for in
uence over our AI. Possibly
they would, in combination, waste as many resources in competing for in
u-
ence as there were resources to be obtained by gaining in
uence: so that the
entire bounty oered up would be consumed in a zero-sum contest of in
uence
peddling (cf. [7]).
Porous Value Aggregation Functions Desiderata
doesn't waste incentive power (across a wide range of possible
scenarios)
doesn't create perverse incentives
Which cookies and aggregation functions over cookies we can dene depends
on the inventory of concepts that are available for use. Generally when think-
ing about these things, it is desirable to use as few and as simple concepts as
possible, since that may increase the chances that the needed concepts can be
dened and implemented in a seed AI by the time this has to be accomplished.
4.3 Filters
One type of concept that may have quite wide applicability in Hail Mary ap-
proaches is that of a lter. The lter is some operational criterion that could
be used to pick out a subset of alien superintelligences, hopefully a subset that
correlates with some desirable property.
8
For example, one lter that it may be useful to be able to specify is that of
an independently-originating superintelligence: superintelligence that was not
created as the direct or indirect consequence of the actions of another superin-
telligence. One use of such a lter would be to eliminate the perverse incentive
referred to above. Instead of having U2 equal the fraction of all superintel-
ligences that produce a cookie, we could use this lter to dene U2 to equal
the fraction of all independently originating superintelligences that produce a
cookie. This would remove the incentive for superintelligence to spawn many
copies of itself in order to increase its weight in our AI's utility function.
Although this origin lter would remove some perverse incentives, it would
not completely eliminate them all. For instance, an alien AI would have an
incentive to prevent the origination of other alien AIs in order to increase its own
weight. This might cause less of a distortion than would using the same porous
value without the origin lter, since it might usually be impossible for an AI to
prevent the independent emergence of other AIs|especially if they emerge very
far away, outside its own Hubble volume. Nevertheless, one can conceive of
scenarios in which the motivational distortion would manifest in undesirable
actions, for instance if our AI could do something to prevent the spontaneous
generation of \baby universes" or otherwise interfere with remote physical
processes. Note that even if it is in fact impossible for our AI to have such an
in
uence, the distortion could still somewhat aect its behavior, so long as the AI
assigns a nonzero subjective credence to such in
uence being possible.
One could try to rene this lter by stipulating that any AI that could
have been causally aected by our AI (including by our AI letting it come into
existence or preventing it from coming into existence) should be excluded by
the aggregation function U2. This would slightly increase the probability that
no qualifying alien AI exists. But it might be preferable to accept a small rise
in the probability of the porous values eectively dropping out o the combined
utility function U than to risk introducing potentially more nefarious perverse
incentives.
Another type of origin lter would seek to discriminate between alien voices
to nd the one most likely to be worth listening to. For example, suppose
we have some view about which path to machine intelligence is most likely
to result in a superintelligence with human-friendly values. For the sake of
concreteness, let us suppose that we think that the superintelligence that was
originally created by means of the extensive use of genetic algorithms is less
likely to be human-friendly than a superintelligence that originated from the
whole-brain-emulation-like computational structure. (We're not endorsing that
claim here, only using it to illustrate one possible line of argument.) Then
one could try to dene a lter that would pick out superintelligences that had
an emulation-like origin and that would reject superintelligences that had a
genetic-algorithm-like origin. U2 could then be formulated to equal the fraction
of superintelligences with the appropriate origin that built a cookie.
There are, of course, conceivable lters that would more closely align with
what we are really interested in picking out, such as the lter \an AI with
human-friendly values". But such a lter looks very hard to program. The
challenge of lter specication is to come up with a lter that might be feasible
for us to program and that would still correlate (even if but imperfectly) with
the properties that would ensure a benecial outcome. Filters that are dened
in terms of structural properties (such as the origin-lter just mentioned, which
9
refer to the computational architecture of the causal origins of an AI) may
turn out to be easier to program then lters that refer to specic material
congurations.
One might also seek to develop lters that would qualify AIs based on char-
acteristics of the originating biological civilization. For example, one could aim
to nd indicators of competence or benevolence that could be conjectured to
correlate with the resulting AI having human-friendly values. Again, the chal-
lenge would be to nd some relatively simple attribute (in the sense of being
possibly something we could program before we are able to program a more
ideal value such as CEV) that nevertheless would carry information about the
preferences of the ensuing AI. For instance, if we could dene time and we had
some view about how the likelihood of a human-friendly AI depends on the
temporal interval between the AI's creation and some earlier evolutionary or
historical milestone (which we would also have to dene in a way that we could
render in computer code) then we could construct a lter that selected for AIs
with especially propitious pasts.
5 Some further issues
5.1 Temporal discounting
We may observe that a temporally discounting AI might be particularly keen
on trading with other AIs. This is because for such an AI value-structures
created at earlier times would be more valuable (if it's discounting function is
of a form that extends to times before the present); so it would be willing to
pay a premium to have AIs that arose at earlier times to have built some its
value-structures back then.
If the time-discounting takes an exponential form, with a non-trivial per an-
num discount rate, it would lead our AI to become obsessed with scenarios that
would allow for an extremely early creation of its value-structures|scenarios in
which either it itself exists much earlier1than is probable, for instance because
it is living in a simulation or has materialized spontaneously from primordial
goo, or because the rate of time turns out to have some surprising measure; or,
alternatively, scenarios in which it is able to trade with AIs that exist at extraor-
dinarily early cosmic epochs. This could lead to undesirable distortions, because
it might be that the most plausible trading partners conditional on some un-
likely hypothesis about time
ow or time of emergence being true would tend to
have atypical values (values less likely to resemble human values). It might also
lead to distortions because it would cause our AI to focus on highly improbable
hypotheses about how the world works and about its own location, and it might
be that the actions that would make sense under those conditions would seem
extremist or bizarre when evaluated in a more commonsensical centered world
model (cf. [4, 3]).
To avoid these potentially distorting eects, one might explore a functional
form of the discount term that plateaus, such that it does not give arbitrarily
1This assumes the zero point for discounting is rigidly designated as a particular point in
sidereal time. If the zero point is instead a moving indexical \now" for the agent, then it
would assume that the moment of the decision is when the discounting is zero, so unless the
discount function extended to earlier times, the agent would not be focussed on in
uencing
the past but on scenarios in which it can have a large impact in the near term.
10
great weight to extremely early AIs. One could also consider creating a time-
symmetric discount factor that has a minimum intensity of discounting at some
point in the past, perhaps in order to target trade to AIs existing at the time
conjectured to have the highest density of human-friendly AIs. It is harder to
get our AI to trade with future AIs by using time discounting, since in this case
our AI has an alternative route to realizing value-structures at the preferred
time: namely, by saving its resources and waiting for the appropriate hour to
arrive, and then build them itself.2
In summary, discounting could encourage trade, though probably it would
have to take a form other than the normal exponential one in order to avoid
focusing the trade on extremely early AIs that may be less likely to have rep-
resentative or human-friendly values. By contrast to porous values, discounting
does not oer an immediate way to avoid incentivizing other AIs to spend as
much resources on getting the trade as they expect the trade to deliver to them.
Porous values are easy to satisfy locally to the maximum extent that they can
be satised locally, and yet require for their full satisfaction contributions from
the many dierent AIs. This eect may be dicult to achieve through time-
discounting alone. Furthermore, it is not clear how to use discounting to strongly
encourage trade with the near- or mid-term future.
5.2 Why a \DNA cookie" would not work
It may be instructive to look at one unsuccessful idea for how to dene a cookie
that would specically encourage trade with other AIs that originated from
human-like civilizations, and that therefore might be thought to be more likely
to have human-friendly values.
The faulty idea would be to dene the cookie|the object that our AI is
programmed to want other AIs to create|in terms of a characteristic of humans
that we can easily measure but that would be dicult for an arbitrary AI to
discover or predict. Consider, for concreteness, the proposal that we dene the
cookie to be the data structure representing the human genome. (More precisely,
we would pick a human reference genome and specify a tolerance margin that
picked out a set of possible genomes, such that any arbitrary human genome
would fall within this set yet such that it would to be extremely dicult to
specify a genome within that set without having any human-like genome to start
from.) The thought then would be that other alien AIs that had originated from
something very much like a human civilization could dene a relevant cookie by
looking at their own ancestral genome, whereas an alien AI that did not have
an origin in a human-like civilization would be completely at a loss: the space of
possible genomes being far too large for there to be any signicant probability
of nding a matching cookie by chance.
The reason this would not work is as follows. Suppose that the universe is
small and relatively sparsely populated. Then probably nobody will have the
same DNA that we do. Then alien AIs would not be able to nd our AIs cookie
recipe (or they would be able to nd it only through a extremely expensive
2An AI with an exponential form of future-preference may still motivated to trade with
AIs that it thinks may be able to survive cosmic decay longer, or AIs that may arise in other
parts of the multiuniverse that are located at \later" times (by whatever measure, if any, is
used to compare time between multiverse parts). But this would again bring in the risk of
distorted concerns, just as in the case of values that privilege extremely early occurrences.
11
exhaustive search of the space of possible genomes); so they would not be able
to use it for trading. Suppose instead that the universe is large and densely
populated. Then there will be other AIs that originated from species with the
same (or very similar) DNA as ours, and they would be able to nd our cookie
simply by examining their own origins. However, there will also be a large
number of other species in situations similar to ours, species that also build AIs
designed to trade with more advanced AIs; and these other species would create
AIs trained on cookies dened in terms of their DNA. When there are enough
civilizations in the universe to make it likely that some AIs will have originated
from species that share our DNA, there will also be enough civilizations to ll
out the space of possible DNA-cookies: for almost any plausible DNA-cookie,
there will be some AI designed to hunger for cookies of that particular type.
This means that advanced AIs wanting to trade with newly formed AIs could
build almost any arbitrary DNA-cookie and still expect to hit a target; there
would be no discriminating factor that would allow advanced AIs to trade only
with younger AIs that had a similar origin as themselves. So the purpose of
using a human-DNA cookie would be defeated.
6 Catchment areas and exclusivity clauses
Some complications arise when we consider that instead of just two AIs|our AI
(the \sender") and an alien AI (the \receiver") that trades with ours by building
its cookie in return for in
uence|there may be a great many senders and a great
many receivers. In this subsection we discuss what these complications are and
how they may be managed.
6.1 Catchment area
There being many receivers can cause a problem by reducing the surplus value
of each transaction. Our AI has only a nite amount of in
uence to give away;
and the greater the number of other AIs that get the share, the smaller the share
of in
uence each of them gets. So long as the population of receivers is only
moderately large this is not a signicant problem, because each receiver only
needs to make one cookie for the trade to go through, and it should be very
inexpensive for a superintelligence to make one cookie (see the cookie recipe
desiderata above). Nevertheless, as the number of receivers becomes extremely
large (and in a realistic universe it might be innite) the share of in
uence over
our AI that each receiver can expect to get drops to the cost of making one
cookie. At that point, all the surplus value of trade is consumed by the cost
of cookie production. (Receivers will not make cookies beyond this point, but
may rather adopt a mixed strategy, such that for any one of them there is some
probability that it will try to engage in trade.)
A remedy to this problem is to give our AI a limited catchment area. We
would dene our AI's porous values such that it only cares about cookies that are
produced within this catchment area: what goes on outside of this area is of no
concern (so far as U2 is concerned).
In principle, the catchment area could be dened in terms of a xed spa-
tiotemporal volume. However, this would require that we are able to dene such
a physical quantit y. It would also suer the problem that we don't know how
12
large a volume to designate as our AI's catchment area. While there is a consid-
erable margin of tolerance, given the extremely low cost per cookie, there is also
a lot of uncertainty|ranging over a great many orders of magnitude|about
how common (independently-originating) alien AIs are in the universe.
A better approach may be to specify that the catchment area consists of the
Nclosest AIs (where \closest" would be dened according to some measure that
may include spatial temporal proximity but could also include other variables,
such as some rough measure of an alien AI's similarity to our own). In any case,
by restricting the catchment area we would limit the number of AIs that are
allowed to bid for in
uence over our AI, and thus the total cost of the cookies
that they produce in the process.
6.2 Exclusivity clause
There being many senders |AIs with porous values hoping to induce alien AIs
to trade with them|may also cause problems. Here the issue is that the mul-
tiplicity of senders may ensure that receivers build a lot of dierent cookies
no matter what ourAI decides to do. Our AI could then choose to free ride
on the eorts of these other senders. If the kind of cookie that our AI wants
to exist will exist (within its catchment area) anyway, whether or not it pays
for its construction, our AI has no reason to make the transfer of in
uence to
alien AIs. In equilibrium, there would still be some amount of trade going on,
since if the free riding were universal it would undermine its own possibility.
However, the amount of trade in such an equilibrium might be very small, as
potential senders adopt a mixed strategy that gives only a tiny chance of en-
gaging in acausal trade. (The extent of this problem may depend on the size
of the catchment areas, the number of plausible cookie recipes, and the cost
of producing the various cookies; as well as on whether other kinds of acausal
trade arrangements could mitigate the issue.)
To avoid such a potential free riding problem, we could embed an exclusivity
clause in our cookie recipe. For example, we could specify our AI's porous
value to require that, in order for a cookie to count as local fulllment of the
porous value, the cookie would have to be built specically in order to trade
with our AI (rather than in order to trade with some other AI, or for some other
reason). Perhaps this could be explicated in terms of a counterfactual over our
AI's preference function: something along the lines of a requirement that there
be (in each Hubble volume, or produced by each independently-originating AI)
one more cookie of type Kthan there would have been if instead of valuing type-
Kcookies our AI had valued (some other) type- K0cookies. This explication
would, in turn, require a denition of the relevant counterfactual.[1] There may
be other ideas for how to go about these things.
7 Utility diversication
The principle of value diversication might be attractive even aside from the
motivations undergirding the Hail Mary approach. We can make a comparison
to the task of specifying an epistemology or a prior probability function for our
AI to use. One approach here would be to pick one particular prior, which we
think have attractive properties, such as the Solomono prior (formulated in
13
a particular base language). An alternate approach would be instead to use
a mixture prior, a superposition of various dierent ideas about what shape
the prior should take. Such a mixture prior might include, for example, vari-
ous Solomono priors using dierent base-languages, some other prior based on
computational depth, a speed prior, a prior that gives some positive nite prob-
ability to the universe being uncomputable or transnite, and a bunch of other
things [9, 2, 10]. One advantage of such an approach would be that it would
reduce the risk that some important hypothesis that is actually true would be
assigned zero or negligible probability in our favored formalization [5]. This
advantage would come at a cost|the cost of assigning a lower probability to
hypotheses that might really be more likely|but this cost might be relatively
small if the agent using the prior has a superintelligence's abilities to gather
and analyze data. Given such an agent, it may be more important that its prior
does not absolutely prevent it from ever learning some important true hypoth-
esis (the universe is uncomputable? we are not Boltzmann Brains?) than that
its prior makes it maximally easy quickly to learn a plethora of smaller truths.
Analogously, to the extent that human values are resource-satiable, and the
superintelligence has access to an astronomical resource endowment, it may
be more important for us to ensure that the superintelligence places at least
some weight on human values than to maximize the probability that it places
no weight on anything else. Value diversication is one way to do this. Just
as we could use a mixture prior in the epistemological component, we might
use a \utility mixture" in the AI's utility function or goal specication. The
formula (1) above suggests one way that this can be done, when we want to add
a bounded component U2as a modulator of a possibly unbounded component
U1. Of course, we couldn't throw just anything into the hopper and still expect
a good outcome: in particular, we would not want to add components that
plausibly contain outright evil or anti-humane values. But as long as we're only
adding value components that are at worst neutral, we should risk nothing more
intolerable than some fractional dilution of the value of our cosmic endowment.
What about values that are not resource-satiable? Aggregative consequen-
tialist theories, such as hedonistic utilitarianism, are not resource-satiable. Ac-
cording to those theories, the value added by creating one more happy mind is
the same whether the extra mind is added onto an existing stock of 10 happy
minds are 10 billion happy minds.3Nevertheless, even if the values we wanted
our AI to pursue are in this sense insatiable, a (weaker) case might still be
made for pursuing a more limited form of utility diversication. One reason
is the vaguely contractualist considerations hinted at above. Another reason,
also alluded to, is that it may often be possible, to some extent, to co-satisfy
two dierent values in the very same physical structure (cf. [8]). Suppose, for
example, that we believe that the value of the world is a linear function of the
number of duck-like things it contains, but we're unsure whether \duck-like"
means \walks like a duck" or \quacks like a duck". Then one option would be
to randomly pick one of these properties, which would give us a 50% chance
of having the world optimized for the maximally valuable pattern and a 50%
3Quite possibly, aggregated consequentialist theories remain insatiable even when we are
considering scenarios in which innite resources are available, since otherwise it would appear
that such theories are unable to provide possible ethical guidance in those kinds of scenarios;
and this might mean that they fail even in our excellent situation, as well as some positive
probability is assigned to the world being canonically innite.[4]
14
chance of having the world optimized for a pattern of zero value. But a better
option would be to create a utility function that assigns utility both to things
that walk like ducks and to things that quack like ducks. An AI with such a
utility function might devise some structure that is reasonably ecient at satis-
fying both criteria simultaneously, so that we would get a pattern that is close
to maximally valuable whether it's duck-like walking or duck-like quacking that
really has value.4
An even better option would be to use indirect normativity [12, 6, 5] to dene
a utility function that assigned utility to whatever it is that \duck-like" really
means|even if we ourselves are quite unsure|so that the AI would be moti-
vated to investigate this question and then to optimize the world accordingly.
However, this could turn out to be dicult to do; and utility diversication
might then be a useful fallback. Or if we come up with several plausible ways
of using indirect normativity, we could try to combine them using a mixture
utility function.
8 Acknowledgements
I am grateful to Stuart Armstrong, Owen Cotton-Barratt, Daniel Dewey, and
Jaan Tallinn for helpful discussions.
References
[1] Stuart Armstrong. Utility indierence. Technical report, Future of Hu-
manity Institute, University of Oxford, 2010.
[2] Charles H Bennett. Logical depth and physical complexity . Springer, 1995.
[3] Nick Bostrom. Pascal's mugging. Analysis , 69(3):443{445, 2009.
[4] Nick Bostrom. Innite ethics. Analysis and Metaphysics , (10):9{59, 2011.
[5] Nick Bostrom. Superintelligence: Paths, dangers, strategies . Oxford Uni-
versity Press, 2014.
[6] Daniel Dewey. Learning what to value. In Articial General Intelligence ,
pages 309{314. Springer, 2011.
[7] P Richard G Layard, Alan Arthur Walters, and AA Walters. Microeco-
nomic theory . McGraw-Hill New York, 1978.
[8] Toby Ord. Moral trade. Ethics , forthcoming, 2015.
[9] Jorma Rissanen. A universal prior for integers and estimation by minimum
description length. The Annals of Statistics , pages 416{431, 1983.
4Probably no ducks, however! Value diversication is not a technique for specifying con-
cepts that are hard to dene. Rather, the idea is that we rst do our best to dene what
we value (or to specify a value-loading mechanism). We might nd that we fail to reach a
consensus on a single denition or value-loading mechanism that we feel fully condent in.
The principle of value diversication then suggests that we seek to conglomerate the leading
candidates into one mixture utility function rather than putting all the chips on one favorite.
15
[10] J urgen Schmidhuber. The speed prior: a new simplicity measure yielding
near-optimal computable predictions. In Computational Learning Theory ,
pages 216{228. Springer, 2002.
[11] Carl Shulman. Omohundro's \basic ai drives" and catastrophic risks.
Manuscript. (intelligence.org/les/BasicAIDrives.pdf) , 2010.
[12] Eliezer Yudkowsky. Coherent extrapolated volition. Machine Intelligence
Research Institute (May 2004). (intelligence.org/les/CEV.pdf) , 2004.
16


Future progress in artificial intelligence: A survey of expert opinion

Müller, Vincent C. and Bostrom, Nick (forthcoming 2014), ‘Future progress in artifi-cial intelligence: A Survey of Expert Opinion, in Vincent C. Müller (ed.), Fundamental Issues of Artificial Intelligence (Synthese Library; Berlin: Springer). Future Progress in Artificial Intelligence: A Survey of Expert Opinion Vincent C. Müller a,b & Nick Bostrom a a) Future of Humanity Institute, Department of Philosophy & Oxford Martin School, University of Oxford. b)Anatolia College/ACT, Thessaloniki Abstract: There is, in some quarters, concern about high–level machine intelligence and superintelligent AI coming up in a few decades, bring-ing with it significant risks for humanity. In other quarters, these issues are ignored or considered science fiction. We wanted to clarify what the distribution of opinions actually is, what probability the best experts currently assign to high–level machine intelligence coming up within a particular time–frame, which risks they see with that development, and how fast they see these developing. We thus designed a brief question-naire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that high-level machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity. 1. Introduction Artificial Intelligence began with the “… conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a ma-chine can be made to simulate it.” (McCarthy, Minsky, Rochester, & Shannon, 1955, p. 1) and moved swiftly from this vision to grand promises for general human-level AI within a few decades. This vision of general AI has now become merely a long-term guiding idea for most current AI research, which focuses on specific scientific and en-gineering problems and maintains a distance to the cognitive sciences. A small minori-ty believe the moment has come to pursue general AI directly as a technical aim with the traditional methods – these typically use the label ‘artificial general intelligence’ (AGI) (see Adams et al., 2012). If general AI were to be achieved, this might also lead to superintelligence: “We can tentatively define a superintelligence as any intellect that greatly exceeds the cognitive per-formance of humans in virtually all domains of interest.” (Bostrom, 2014 ch. 2). One idea how
Future Progress in Artificial Intelligence: A Poll Among Experts 2/19
 superintelligence might come about is that if we humans could create artificial general intelligent ability at a roughly human level, then this creation could, in turn, create yet higher intelligence, which could, in turn, create yet higher intelligence, and so on … So we might generate a growth well beyond human ability and perhaps even an ac-celerating rate of growth: an ‘intelligence explosion’. Two main questions about this development are when to expect it, if at all (see Bostrom, 2006; Hubert L. Dreyfus, 2012; Kurzweil, 2005) and what the impact of it would be, in particular which risks it might entail, possibly up to a level of existential risk for humanity (see Bostrom, 2013; Müller, 2014a). As Hawking et al. say “Success in creating AI would be the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks.” (Hawking, Russell, Tegmark, & Wilczek, 2014; cf. Price, 2013). So, we decided to ask the experts what they predict the future holds – knowing that predictions on the future of AI are often not too accurate (see Armstrong, Sotala, & O Heigeartaigh, 2014) and tend to cluster around ‘in 25 years or so’, no matter at what point in time one asks.1 2. Questionnaire 2.1. Respondents The questionnaire was carried out online by invitation to particular individuals from four different groups for a total of ca. 550 participants (see Appendix 2). Each of the participants got an email with a unique link to our site to fill in an online form (see Appendix 1). If they did not respond within 10 days, a reminder was sent, and another 10 days later, with the note that this is the last reminder. In the case of EETN (see below) we could not obtain the individual email addresses and thus sent the request and reminders to the members’ mailing list. Responses were made on a single web page with one ‘submit’ button that only allowed submissions through these unique links, thus making non–invited responses extremely unlikely. The groups we asked were: 1. PT–AI: Participants of the conference on “Philosophy and Theory of AI”, Thessaloniki October 2011, organized by one of us (see Müller, 2012, 2013). Participants were asked in November 2012, i.e. over a year after the event. The total of 88 participants include a workshop on “The Web and Philoso-phy” (ca. 15 people), from which a number of non–respondents came. A list of participants is on: http://www.pt–ai.org/2011/registered–participants 1 There is a collection of predictions on http://www.neweuropeancentury.org/SIAI-FHI\_AI\_predictions.xls
Future Progress in Artificial Intelligence: A Poll Among Experts 3/19
 2. AGI: Participants of the conferences of “Artificial General Intelligence” (AGI 12) and “Impacts and Risks of Artificial General Intelligence” (AGI Impacts 2012), both Oxford December 2012. We organized AGI–Impacts (see Müller, 2014b) and hosted AGI 12. The poll was announced at the meeting of 111 participants (of which 7 only for AGI–Impacts) and carried out ca. 10 days later. The conference site is at: http://www.winterintelligence.org/oxford2012/ 3. EETN: Members of the Greek Association for Artificial Intelligence (EETN), a professional organization of Greek published researchers in the field, in April 2013. Ca. 250 members. The request was sent to the mailing list. The site of EETN: http://www.eetn.gr/ 4. TOP100: The 100 ‘Top authors in artificial intelligence’ by ‘citation’ in ‘all years’ according to Microsoft Academic Search (http://academic.research.microsoft.com) in May 2013. We reduced the list to living authors, added as many as necessary to get back to 100, searched for professional e–mails on the web and sent notices to these. The questionnaire was sent with our names on it and with an indication that we would use it for this paper and Nick Bostrom’s new book on superintelligence (Bostrom, 2014) – our request email is in Appendix 1. Given that the respondent groups 1 and 2 attended conferences organized by us, they knew whom they were responding to. In groups 3 and 4 we would assume that the majority of experts would not know us, or even of us. These differences are reflected in the response rates. These groups have different theoretical-ideological backgrounds: The partici-pants of PT–AI are mostly theory–minded, mostly do not do technical work, and of-ten have a critical view on large claims for easy progress in AI (Herbert Dreyfus was a keynote speaker in 2011). The participants of AGI are committed to the view that AI research should now return from technical details to ‘artificial general intelligence’ – thus the name AGI. The vast majority of AGI participants do technical work. The EETN is a professional association in Greece that accepts only published researchers from AI. The TOP100 group also works mostly in technical AI; its members are sen-ior and older than the average academic; the USA is strongly represented. Several individuals are members of more than one of these four sets and they were unlikely to respond to the same questionnaire more than once. So, in these cases, we sent the query only once, but counted a response for each set – i.e. we knew which individuals responded from the individual tokens they received (except in the case of EETN).
Future Progress in Artificial Intelligence: A Poll Among Experts 4/19
 2.2. Response rates 1) PT–AI: 49% 43 out of 88 2) AGI: 65% 72 out of 111 3) EETN: 10% 26 out of 250 4) TOP100: 29% 29 out of 100 Total: 31% 170 out of 549 2.3. Methodology In this field, it is hard to ask questions that do not require lengthy explanations or generate resistance in certain groups of potential respondents (and thus biased results). It is not clear what constitutes ‘intelligence’ or ‘progress’ and whether intelligence can be measured or at least compared as ‘more’ or ‘less’ as a single dimension. Further-more, for our purposes we need a notion of intelligence at a level that may surpass humans or where technical intelligent systems might contribute significantly to re-search – but ‘human–level intelligence’ is a rather elusive notion that generates re-sistance. Finally, we need to avoid using terms that are already in circulation and would thus associate the questionnaire with certain groups or opinions, like “artificial intelligence”, “singularity”, “artificial general intelligence” or “cognitive system”. For these reasons, we settled for a definition that a) is based on behavioral ability, b) avoids the notion of a general ‘human–level’ and c) uses a newly coined term. We put this definition in the preamble of the questionnaire: “Define a ‘high–level machine intelligence’ (HLMI) as one that can carry out most human professions at least as well as a typical human.” (We still had one expert writing back to us that they could not say what a ‘typical human’ is – though they could be convinced to respond, after all.) In hindsight, it may have been preferable to specify what we mean by ‘most’ and wheth-er we think of ‘most professions’ or of ‘the professions most working people do’. One merit of our behavioral question is that having HLMI in our sense very likely implies being able to pass a classic Turing test. To achieve a high response rate, we tried to have few questions with simple choices and eventually settled for four questions, plus three on the respondents. We tried to choose questions that would allow us to compare our results with those of ear-lier questionnaires – see below. In order to improve on the quality of predictions, we tried to ‘prime’ respondents into thinking about what is involved in reaching HLMI before asking when they expect this. We also wanted to see whether people with a preference for particular approach-es to HLMI would have particular responses to our central questions on prediction (e.g. whether people who think that ‘embodied systems’ are crucial expect longer than average time to HLMI). For these two purposes, we inserted a first question about contributing research approaches with a list to choose from – the options that were
Future Progress in Artificial Intelligence: A Poll Among Experts 5/19
 given are an eclectic mix drawn from many sources, but the particular options are not of much significance. 2.4. Prior work A few groups have recently made attempts to gauge opinions. We tried to phrase our questions such that the answers can be compared to these earlier questionnaires. No-table are: 1. (Michie, 1973, p. 511f): “an opinion poll taken last year among sixty-seven British and American computer scientists working in, or close to, the ma-chine intelligence field”. 2. Questions asked live during the 2006 AI@50 conference at Dartmouth Col-lege through a wireless voting device (VCM participated (see Müller, 2007)). Despite a short report on the conference in (Moor, 2006), the results were not published, but thankfully we were able to acquire them from the organ-izers James H. Moor and Carey E. Heckman – we publish a selection below. 3. (Baum, Goertzel, & Goertzel, 2011): participants of AGI 2009, not anony-mous, on paper, 21 respondents, response rate unknown.2 4. (Sandberg & Bostrom, 2011): participants of Winter Intelligence Conference 2011, anonymous, on paper, 35 respondents, 41% response rate. 1. The reference by the famous AI researcher Donald Michie is very brief (all the de-tails he gives are in the above quote) but of great of historical interest: 1972/3 were turning years for AI with the publication of Hubert Dreyfus’ “What computers can’t do” (Hubert L. Dreyfus, 1972), the “Lighthill Debates” on BBC TV (with Michie, McCarthy and R. Gregory) and the influential “Lighthill Report” (Lighthill, 1973). Michie’s poll asked for the estimated number of years before “computing exhibiting intelligence at adult human level” and Michie’s graph shows 5 data points: Years Percentage 5 0% 10 1% 20 17% 50 19% >50 25% He also asked about “significant industrial spin-off”, “contributions to brain studies” and “contributions from brain studies to machine intelligence”. Michie adds “Of 2 A further, more informal, survey was conducted in August 2007 by Bruce J Klein (then of Novamente and the Singularity Institute) “… on the time–frame for when we may see greater–than–human level AI”, with a few numerical results and interesting comments, archived on https://web.archive.org/web/20110226225452/http://www.novamente.net/bruce/?p=54.
Future Progress in Artificial Intelligence: A Poll Among Experts 6/19
 those responding to a question on the risk of ultimate ‘takeover’ of human affairs by intelligent machines, about half regarded it as ‘negligible’, and most of the remainder as ‘substantial’, with a view voting for ‘overwhelming’.” (Michie, 1973, p. 512). 2. AI@50 hosted many prominent AI researchers, including all living participants of the 1956 Dartmouth Conference, a set of DARPA-funded graduate students, plus a few theoreticians. The participants were asked 12 multiple choice questions on day one, 17 on day two and another 10 on day three. We select three results from day one here: 3.) The earliest that machines will be able to simulate learning and every other aspect of human intelligence: Within 10 years 6 5% Between 11 and 25 years 3 2% Between 26 and 50 years 14 11% More than 50 years 50 41% Never 50 41% Totals 123 100% 5.) The earliest we will understand the basic operations (mental steps) of the human brain sufficiently to create machine simulation of human thought is: today (we already understand enough) 5 6% within the next 10 years 11 12% within the next 25 years 9 10% within the next 50 years 19 21% within the next 100 years or more 26 29% never (we will never understand enough) 19 21% Totals 89 100% 6.) The earliest we will understand the architecture of the brain (how its organi-zational control is structured) sufficiently to create machine simulation of hu-man thought is: Within 10 years 12 11% Between 11 and 25 years 15 14% Between 26 and 50 years 24 22% More than 50 years 44 40% Never 15 14% Totals 110 100% 3. Baum et al. asked for the ability to pass a Turing test, a third grade school year ex-am [i.e. for 9 year olds] and do Nobel Prize level research. They assume that all and only the intelligent behavior of humans is captured in the Turing test. The results they
Future Progress in Artificial Intelligence: A Poll Among Experts 7/19
 got for the 50% probability point were: 2040 (Turing test), 2030 (third grade), and 2045 (Nobel). 4. Sandberg and Bostrom’s first question was quite similar to our 2nd (see below): “Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of human–level machine intelligence?” The median estimate of when there will be 50% chance of human–level machine in-telligence was 2050. So, despite significant overlap with AGI 2009, the group asked by Sandberg and Bostrom in 2011 was a bit more guarded in their expectations. We think it is worthwhile to make a new attempt because the prior ones asked specific groups and small samples, sometimes have methodological problems, and we also want to see how the answers change over time, or do not change – which is why tried to use similar questions. As explained below, we also think it might be worth-while to repeat our questionnaire at a later stage, to compare results. 3. Questions & Responses 3.1. Research Approaches “1. In your opinion, what are the research approaches that might contribute the most to the development of such HLMI?” [Selection from list, more than one selection pos-sible.] − Algorithmic complexity theory − Algorithms revealed by computational neuroscience − Artificial neural networks − Bayesian nets − Cognitive science − Embodied systems − Evolutionary algorithms or systems − Faster computing hardware − Integrated cognitive architectures − Large–scale datasets − Logic–based systems − Robotics − Swarm intelligence − Whole brain emulation − Other method(s) currently known to at least one investigator − Other method(s) currently completely unknown − No method will ever contribute to this aim
Future Progress in Artificial Intelligence: A Poll Among Experts 8/19
 Cognitive science 47.9% Integrated cognitive architectures 42.0% Algorithms revealed by computational neuroscience 42.0% Artificial neural networks 39.6% Faster computing hardware 37.3% Large-scale datasets 35.5% Embodied systems 34.9% Other method(s) currently completely unknown 32.5% Whole brain emulation 29.0% Evolutionary algorithms or systems 29.0% Other method(s) currently known to at least one investigator 23.7% Logic-based systems 21.3% Algorithmic complexity theory 20.7% No method will ever contribute to this aim 17.8% Swarm intelligence 13.6% Robotics 4.1% Bayesian nets 2.6% The percentages here are over the total of responses. There were no significant differ-ences between groups here, except that ‘Whole brain emulation’ got 0% in TOP100, but 46% in AGI. We did also not find relevant correlations between the answers given here and the predictions made in the following questions (of the sort that, for example, people who think ‘embodied systems’ crucial would predict later onset of HLMI).
Future Progress in Artificial Intelligence: A Poll Among Experts 9/19
 3.2. When HLMI? “2. For the purposes of this question, assume that human scientific activity continues without major negative disruption. By what year would you see a (10% / 50% / 90%) probability for such HLMI to exist?” – For each of these three probabilities, the re-spondents were asked to select a year [2012–5000, in one-year increments] or check a box marked ‘never’. Results sorted by groups of respondents: PT-AI Median Mean St. Dev. 10% 2023 2043 81 50% 2048 2092 166 90% 2080 2247 515 AGI Median Mean St. Dev. 10% 2022 2033 60 50% 2040 2073 144 90% 2065 2130 202 EETN Median Mean St. Dev. 00%
10%
20%
30%
40%
50%
60%
Bayesian nets
Robotics
Swarm intelligence
No method will ever contribute to this Algorithmic complexity theory
Logic-based systems
Other method(s) currently known to at Evolutionary algorithms or systems
Whole brain emulation
Other method(s) currently completely Embodied systems
Large-scale datasets
Faster computing hardware
Artiﬁcial neural networks
Algorithms revealed by computational Integrated cognitive architectures
Cognitive science
Future Progress in Artificial Intelligence: A Poll Among Experts 10/19
 10% 2020 2033 29 50% 2050 2097 200 90% 2093 2292 675 TOP100 Median Mean St. Dev. 10% 2024 2034 33 50% 2050 2072 110 90%: 2070 2168 342 ALL Median Mean St. Dev. 10%: 2022 2036 59 50%: 2040 2081 153 90%: 2075 2183 396 Results sorted by percentage steps: 10% Median Mean St. Dev. PT-AI 2023 2043 81 AGI 2022 2033 60 EETN 2020 2033 29 TOP100 2024 2034 33 ALL 2022 2036 59 50% Median Mean St. Dev. PT-AI 2048 2092 166 AGI 2040 2073 144 EETN 2050 2097 200 TOP100 2050 2072 110 ALL 2040 2081 153 90% Median Mean St. Dev. PT-AI 2080 2247 515 AGI 2065 2130 202 EETN 2093 2292 675 TOP100 2070 2168 342 ALL 2075 2183 396
Future Progress in Artificial Intelligence: A Poll Among Experts 11/19
 Clicks of the ‘never’ box. These answers did not enter in to the averages above. Never no. % 10% 2 1.2 50% 7 4.1 90% 28 16.5
 For the 50% mark, the overall median is 2040 (i.e. half of the respondents gave a year earlier than 2040 and half gave a year later than 2040) but the overall mean (average) is 2081. The median is always lower than the mean here because there cannot be out-liers towards ‘earlier’ but there are outliers towards ‘later’ (the maximum possible se-lection was 5000, then ‘never’). 3.3. From HLMI to superintelligence “3. Assume for the purpose of this question that such HLMI will at some point exist. How likely do you then think it is that within (2 years / 30 years) thereafter there will be machine intelligence that greatly surpasses the performance of every human in most professions?” – Respondents were asked to select a probability from a drop-down menu in 1% increments, starting with 0%.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
2000
2050
2100
2150
2200
2250
2300
Proportion of experts with 10% 50% 90% conﬁdence of HLMI by that date
10%
50%
90%
Future Progress in Artificial Intelligence: A Poll Among Experts 12/19
 For all respondents: Median Mean St. Dev. Within 2 years 10% 19% 24 Within 30 years 75% 62% 35 Median estimates on probability of superintelligence given HLMI in different groups of respondents: 2 years 30 years PT-AI 10% 60% AGI 15% 90% EETN 5% 55% TOP100 5% 50% Experts allocate a low probability for a fast takeoff, but a significant probability for superintelligence within 30 years after HLMI. 3.4. The impact of superintelligence “4. Assume for the purpose of this question that such HLMI will at some point exist. How positive or negative would be overall impact on humanity, in the long run? Please indicate a probability for each option. (The sum should be equal to 100%.)” – Respondents had to select a probability for each option (in 1% increments). The addi-tion of the selection was displayed; in green if the sum was 100%, otherwise in red. The five options were: “Extremely good – On balance good – More or less neutral – On balance bad – Extremely bad (existential catastrophe)”. % PT-AI AGI EETN TOP100 ALL Extremely good 17 28 31 20 24 On balance good 24 25 30 40 28 More or less neutral 23 12 20 19 17 On balance bad 17 12 13 13 13 Extremely bad (existential catastrophe) 18 24 6 8 18 Percentages here are means, not medians as in the other tables. There is a notable difference here between the ‘theoretical’ (PT-AI and AGI) and the ‘technical’ groups (EETN and TOP100). 3.5. Respondents Statistics We then asked the respondents 3 questions about themselves:
Future Progress in Artificial Intelligence: A Poll Among Experts 13/19
 1. “Concerning the above questions, how would you describe your own exper-tise?” (0 = none, 9 = expert) − Mean 5.85 2. “Concerning technical work in artificial intelligence, how would you describe your own expertise?” (0 = none, 9 = expert) − Mean 6.26 3. “What is your main home academic discipline?” (Select from list with 8 op-tions: Biology/Physiology/Neurosciences – Computer Science – Engineering [non CS] – Mathematics/Physics – Philosophy – Psychology/Cognitive Sci-ence – Other academic discipline – None.) [Absolut numbers.] a. Biology/Physiology/Neurosciences 3 b. Computer Science 107 c. Engineering (non CS) 6 d. Mathematics/Physics 10 e. Philosophy 20 f. Psychology/Cognitive Science 14 g. Other academic discipline 9 h. None 1 And we finally invited participants to make a comment, plus a possibility to add their name, if they wished. (We cannot reproduce these here; but they are on our site, see below). A number of comments concerned the difficulty of formulating good ques-tions, much fewer the difficulty of predicting. 4. Evaluation 4.1. Selection-bias in the respondents? One concern with the selection of our respondents is that people who think HLMI is unlikely, or a confused idea, are less likely to respond (though we pleaded otherwise in the letter, see below). Here is a characteristic response from a keynote speaker at PT-AI 2011: “I wouldn’t think of responding to such a biased questionnaire. … I think any discussion of imminent super–intelligence is misguided. It shows no understand-ing of the failure of all work in AI. Even just formulating such a questionnaire is bi-ased and is a waste of time.” (Hubert Dreyfus, quoted with permission). So, we tried to find out what the non-respondents think. To this end, we made a random selection of non-respondents from two groups (11 for PT-AI and 17 from TOP100) and pressured them via personal email to respond, explaining that this would help us understand bias. The two groups were selected because AGI appears already biased in the oppo-site direction and EETN appears very similar to TOP100 but for EETN we did not
Future Progress in Artificial Intelligence: A Poll Among Experts 14/19
 have the data to show us who responded and who did not. We got one additional re-sponse from PT-AI and two from TOP100 in this way. For question 2 “… By what year would you see a (10% / 50% / 90%) probabil-ity for such HLMI to exist?” we compared the additional responses to the responses we already had from the same respective group (PT-AI and TOP100, respectively). We found the following differences: 10% 50% 90% mean median mean median mean median PT-AI -12 +8 -9 +55 -2 +169 TOP100 -19 -9 -47 -25 -138 -40 The one additional respondent from PT-AI expected HLMI earlier than the mean but later than the median, while the two respondents from TOP100 (last row) ex-pected HLMI earlier than mean and median. The very small sample forbids confident judgment, but we found no support for the worry that the non-respondents would have been biased towards a later arrival of HLMI. 4.2. Lessons and outlook We complement this paper with a small site on http://www.pt-ai.org/ai-polls/. On this site, we provide a) the raw data from our results [anonymous unless the partici-pants decided to put their name on their responses], b) the basic results of the ques-tionnaire, c) the comments made, and d) the questionnaire in an online format where anyone can fill it in. We expect that that online questionnaire will give us an interest-ing view of the ‘popular’ view of these matters and on how this view changes over time. In the medium run, it be interesting to do a longitudinal study that repeats this exact questionnaire. We leave it to the reader to draw their own detailed conclusions from our results, perhaps after investigating the raw data. Let us stress, however, that the aim was to ‘gauge the perception’, not to get well-founded predictions. These results should be taken with some grains of salt, but we think it is fair to say that the results reveal a view among experts that AI systems will probably (over 50%) reach overall human ability by 2040-50, and very likely (with 90% probability) by 2075. From reaching human ability, it will move on to superintelligence in 2 years (10%) to 30 years (75%)
Future Progress in Artificial Intelligence: A Poll Among Experts 15/19
 thereafter. The experts say the probability is 31% that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity. So, the experts think that superintelligence is likely to come in a few decades and quite possibly bad for humanity – this should be reason enough to do research into the possible impact of superintelligence before it is too late. We could also put this more modestly and still come to an alarming conclusion: We know of no compelling reason to say that progress in AI will grind to a halt (though deep new insights might be needed) and we know of no compelling reason that superintelligent systems will be good for humanity. So, we should better investigate the future of superintelligence and the risks it poses for humanity. 5. Acknowledgements Toby Ord and Anders Sandberg were helpful in the formulation of the questionnaire. The technical work on the website form, sending mails and reminders, database and initial data analysis was done by Ilias Nitsos (under the guidance of VCM). Theo Gantinas provided the emails of the TOP100. Stuart Armstrong made most graphs for presentation. The audience at the PT-AI 2013 conference in Oxford provided helpful feedback. Mark Bishop, Carl Shulman, Miles Brundage and Daniel Dewey made detailed comments on drafts. We are very grateful to all of them. 6. References Adams, S., Arel, I., Bach, J., Coop, R., Furlan, R., Goertzel, B., . . . Sowa, J. F. (2012). Mapping the landscape of human-level artificial general intelligence. AI Magazine, 33(1), 25-42. Armstrong, S., Sotala, K., & O Heigeartaigh, S. (2014). The errors, insights and lessons of famous AI predictions – and what they mean for the future. Journal of Experimental and Theoretical Artificial Intelligence, 26(3 - Special issue ‘Risks of General Artificial Intelligence’, ed. V. Müller), 317-342. Baum, S. D., Goertzel, B., & Goertzel, T. G. (2011). How long until human-level AI? Results from an expert assessment. Technological Forecasting & Social Change, 78(1), 185-195. Bostrom, N. (2006). How long before superintelligence? Linguistic and Philosophical Investigations, 5(1), 11-30. Bostrom, N. (2013). Existential risk prevention as global priority. Global Policy, 4(1), 15-31. Bostrom, N. (2014). Superintelligence: Paths, dangers, strategies. Oxford: Oxford University Press. Dreyfus, H. L. (1972). What computers still can't do: A critique of artificial reason (2 ed.). Cambridge, Mass.: MIT Press. Dreyfus, H. L. (2012). A history of first step fallacies. Minds and Machines, 22(2 - special issue "Philosophy of AI" ed. Vincent C. Müller), 87-99.
Future Progress in Artificial Intelligence: A Poll Among Experts 16/19
 Hawking, S., Russell, S., Tegmark, M., & Wilczek, F. (2014). Transcendence looks at the implications of artificial intelligence - but are we taking AI seriously enough? The Independent, 01.05.2014. Kurzweil, R. (2005). The singularity is near: When humans transcend biology. London: Viking. Lighthill, J. (1973). Artificial intelligence: A general survey Artificial intelligence: A paper symposion. London: Science Research Council. McCarthy, J., Minsky, M., Rochester, N., & Shannon, C. E. (1955). A proposal for the Dartmouth summer research project on artificial intelligence. Retrieved October 2006, from http://www-formal.stanford.edu/jmc/history/dartmouth/dartmouth.html Michie, D. (1973). Machines and the theory of intelligence. Nature, 241(23.02.1973), 507-512. Moor, J. H. (2006). The Dartmouth College artificial intelligence conference: The next fifty years. AI Magazine, 27(4), 87-91. Müller, V. C. (2007). Is there a future for AI without representation? Minds and Machines, 17(1), 101-115. Müller, V. C. (2014a). Editorial: Risks of general artificial intelligence. Journal of Experimental and Theoretical Artificial Intelligence, 26(3 - Special issue ‘Risks of General Artificial Intelligence’, ed. V. Müller), 1-5. Müller, V. C. (Ed.). (2012). Theory and philosophy of AI (Minds and Machines, Vol. 22/2- Special volume): Springer. Müller, V. C. (Ed.). (2013). Theory and philosophy of artificial intelligence (SAPERE, Vol. 5). Berlin: Springer. Müller, V. C. (Ed.). (2014b). Risks of artificial general intelligence (Journal of Experimental and Theoretical Artificial Intelligence, Vol. (26/3) Special Issue): Taylor & Francis. Price, H. (2013). Cambridge, Cabs and Copenhagen: My Route to Existential Risk. The New York Times, 27.01.2013. http://opinionator.blogs.nytimes.com/2013/01/27/cambridge-cabs-and-copenhagen-my-route-to-existential-risk/?\_php=true&\_type=blogs&\_r=0 Sandberg, A., & Bostrom, N. (2011). Machine intelligence survey. FHI Technial Report, 2011(1). http://www.fhi.ox.ac.uk/research/publications/ 7. Appendices 1. Questionnaire 2. Letter sent to participants
Future Progress in Artificial Intelligence: A Poll Among Experts 17/19
 Appendix 1: Online Questionnaire
 Large-scale datasets
 Logic-based systems
 Robotics
 Swarm intelligence
 Whole brain emulation
 Other method(s) currently known to at least one investigator
 Other method(s) currently completely unknown
 No method will ever contribute to this aim10%50%90%Year reached:
-
-
-Never:
Within 2 yearsWithin 30 yearsProbability:
-%
-%Extremely goodOn balance goodMore or less neutralOn balance badExtremely bad (existential catastrophe)
- %
- %
- %
- %
- %Questionnaire: Future Progress in Artificial Intelligence(http://www.fhi.ox.ac.uk/) (http://www.futuretech.ox.ac.uk)This brief questionnaire is directed towards researchers in artificial intelligence or the theory ofartificial intelligence. It aims to gauge how people working in the field view progress towards itsoriginal goals of intelligent machines, and what impacts they would associate with reachingthese goals.Contribution to this questionnaire is by invitation only. If the questionnaire is filled in withoutsuch an invitation, the data will be disregarded.Answers will be anonymized. Results will be made publicly available on the site of the Programme on the Impacts of Future Technology:http://www.futuretech.ox.ac.uk (http://www.futuretech.ox.ac.uk) .Thank you for your time!Vincent C. Müller (http://www.sophia.de) & Nick Bostrom (http://www.nickbostrom.com/)University of OxfordSeptember 2012A. The Future of AIDefine a "high-level machine intelligence" (HLMI) as one that can carry out most human professions at least as well as a typical human.1. In your opinion, what are the research approaches that might contribute the most to the development of such HLMI?:
 Algorithmic complexity theory
 Algorithms revealed by computational neuroscience
 Artificial neural networks
 Bayesian nets
 Cognitive science
 Embodied systems
 Evolutionary algorithms or systems
 Faster computing hardware
 Integrated cognitive architectures2. Assume for the purpose of this question that human scientific activity continues without major negative disruption. By what year wouldyou see a 10%/50%/90% probability for such HLMI to exist?
 3. Assume for the purpose of this question that such HLMI will at some point exist. How likely do you then think it is that within (2 years / 30years) thereafter, there will be machine intelligence that greatly surpasses the performance of any human in most professions? 4. Assume for the purpose of this question that such HLMI will at some point exist. How positive or negative would be the overall impact onhumanity, in the long run? Please indicate a probability for each option. (The sum should be equal to 100%.)
 Total: 0%Questionnaire: Future Progress in Artiﬁcial Intelligence | Phil...http://www.pt-ai.org/questionnaire-future-progress-artiﬁcial-in...
1 of 226.11.12 10:29 OEZ
Future Progress in Artificial Intelligence: A Poll Among Experts 18/19

 0 = none
 1
 2
 3
 4
 5
 6
 7
 8
 9 = expert1. Concerning the above questions, how would you describe your own expertise?:
 0 = none
 1
 2
 3
 4
 5
 6
 7
 8
 9 = expert2. Concerning technical work in artificial intelligence, how would you describe your own expertise?:B. About you
 3. What is your main home academic discipline?:
 Biology/Physiology/Neurosciences
 Computer Science
 Engineering (non CS)
 Mathematics/Physics
 Philosophy
 Psychology/Cognitive Science
 Other academic discipline
 None 4. Add a brief comment, if you like (<250 words). These comments may be published. Please indicate whether you would like your name tobe included with the comment. (The answers above will remain anonymous in any case.):
Total word Count : 0Please include my name with the comment (leave this field empty if you wish to remain anonymous):
CAPTCHAThis question is for testing whether you are a human visitor and to prevent automated spam submissions.
What code is in the image?: \*
Enter the characters shown in the image.
SubmitQuestionnaire: Future Progress in Artiﬁcial Intelligence | Phil...http://www.pt-ai.org/questionnaire-future-progress-artiﬁcial-in...
2 of 226.11.12 10:29 OEZ
Future Progress in Artificial Intelligence: A Poll Among Experts 19/19
 Appendix 2: Letter to participants (here TOP100) Dear Professor [surname], given your prominence in the field of artificial intelligence we invite you to express your views on the future of artificial intelligence in a brief questionnaire. The aim of this exercise is to gauge how the top 100 cited people working in the field view pro-gress towards its original goals of intelligent machines, and what impacts they would associate with reaching these goals. The questionnaire has 4 multiple choice questions, plus 3 statistical data points on the respondent and an optional 'comments' field. It will only take a few minutes to fill in. Of course, this questionnaire will only reflect the actual views of researchers if we get nearly everybody to express their opinion. So, please do take a moment to respond, even (or especially) if you think this exercise is futile or misguided. Answers will be anonymous. Results will be used for Nick Bostrom's forthcoming book “Superintelligence: Paths, Dangers, Strategies” (Oxford University Press, 2014) and made publicly available on the site of the Programme on the Impacts of Future Tech-nology: http://www.futuretech.ox.ac.uk. Please click here now: [link] Thank you for your time! Nick Bostrom & Vincent C. Müller University of Oxford