How we’re predicting AI–or failing to

MIRI
MACHINE INTELLIGENCE
RESEARCH INSTITUTEHow We’re Predicting AI—or Failing To
Stuart Armstrong
Future of Humanity Institute
Kaj Sotala
Machine Intelligence Research Institute
Abstract
This paper will look at the various predictions that have been made about AI and pro-
pose decomposition schemas for analyzing them. It will propose a variety of theoretical
tools for analyzing, judging, and improving these predictions. Focusing specifically on
timeline predictions (dates given by which we should expect the creation of AI), it will
show that there are strong theoretical grounds to expect predictions to be quite poor in
this area. Using a database of 95 AI timeline predictions, it will show that these expec-
tations are borne out in practice: expert predictions contradict each other considerably,
and are indistinguishable from non-expert predictions and past failed predictions. Pre-
dictions that AI lie 15 to 25 years in the future are the most common, from experts and
non-experts alike.
NOTE: The findings in this paper are based on a dataset error. For details, see
https://aiimpacts.org/error-in-armstrong-and-sotala-2012/ .
Armstrong, Stuart, and Kaj Sotala. 2012. “How We’re Predicting AI—or Failing To.”
InBeyond AI: Artificial Dreams , edited by Jan Romportl, Pavel Ircing, Eva Zackova, Michal Polak, and
Radek Schuster, 52–75. Pilsen: University of West Bohemia.
This version contains minor changes.
Stuart Armstrong, Kaj Sotala
1.Introduction
Predictions about the future development of artificial intelligence are as confident as they
are diverse. Starting with Turing’s initial estimation of a 30% pass rate on Turing test
by the year 2000 (Turing 1950), computer scientists, philosophers and journalists have
never been shy to oﬀer their own definite prognostics, claiming AI to be impossible
(Jacquette 1987) or just around the corner (Darrach 1970) or anything in between.
What are we to make of these predictions? What are they for, and what can we gain
from them? Are they to be treated as light entertainment, the equivalent of fact-free
editorials about the moral decline of modern living? Or are there some useful truths to
be extracted? Can we feel confident that certain categories of experts can be identified,
and that their predictions stand out from the rest in terms of reliability?
In this paper, we start oﬀ by proposing classification schemes for AI predictions:
what types of predictions are being made, and what kinds of arguments or models are
being used to justify them. Diﬀerent models and predictions can result in very diﬀerent
performances, and it will be the ultimate aim of this project to classify and analyze their
varying reliability.
Armed with this scheme, we then analyze some of these approaches from the theo-
retical perspective, seeing whether there are good reasons to believe or disbelieve their
results. The aim is not simply to critique individual methods or individuals, but to con-
struct a toolbox of assessment tools that will both enable us to estimate the reliability of
a prediction, and allow predictors to come up with better results themselves.
This paper, the first in the project, looks specifically at AI timeline predictions: those
predictions that give a date by which we should expect to see an actual AI being devel-
oped (we use AI in the old fashioned sense of a machine capable of human-comparable
cognitive performance; a less ambiguous modern term would be “AGI,” Artificial Gen-
eralIntelligence). With the aid of the biases literature, we demonstrate that there are
strong reasons to expert that experts would notbe showing particular skill the field of
AI timeline predictions. The task is simply not suited for good expert performance.
Those theoretical results are supplemented with the real meat of the paper: a database
of 257 AI predictions,made in a period spanning from the 1950s to the present day. This
database was assembled by researchers from the Singularity Institute (Jonathan Wang
and Brian Potter) systematically searching though the literature,and is a treasure-trove of
interesting results. A total of 95 of these can be considered AI timeline predictions. We
assign to each of them a single “median AI” date, which then allows us to demonstrate
that AI expert predictions are greatly inconsistent with each other—and indistinguish-
able from non-expert performance, and past failed predictions.
1
How We’re Predicting AI—or Failing To
With the data,we further test two folk theorems: firstly that predictors always predict
the arrival of AI just before their own deaths, and secondly that AI is always 15 to 25
years into the future. We find evidence for the second thesis but not for the first.
This enabled us to show that there seems to be no such thing as an “AI expert” for
timeline predictions: no category of predictors stands out from the crowd.
2.Taxonomy of Predictions
2.1. PredictionTypes
There will never be a bigger plane built.
—Boeing engineer on the 247, a twin engine plane that held ten people.
The standard image of a prediction is some fortune teller staring deeply into the mists
of a crystal ball, and decreeing, with a hideous certainty, the course of the times to come.
Or in a more modern version, a scientist predicting the outcome of an experiment or
an economist pronouncing on next year’s GDP figures. But these “at date X, Y will
happen” are just one type of valid prediction. In general, a prediction is something that
constrains our expectation of the future. Before hearing the prediction, we thought the
future would have certain properties; but after hearing and believing it, we now expect
the future to be diﬀerent from our initial thoughts.
Under this definition, conditional predictions—“if A, then B will happen”—are also
perfectly valid. As are negative predictions: we might have believed initially that perpet-
ual motion machines were possible, and imagined what they could be used for. But once
we accept that one cannot violate conservation of energy, we have a diﬀerent picture of
the future: one without these wonderful machines and all their fabulous consequences.
For the present analysis, we will divide predictions about AI into four types:
Timelines and outcome predictions. These are the traditional types of predictions,telling
us when we will achieve specific AI milestones. Examples: An AI will pass theTur-
ing test by 2000 (Turing 1950); within a decade,AIs will be replacing scientists and
other thinking professions (Hall 2011).
Scenarios. These are a type of conditional predictions, claiming that if the conditions
of the scenario are met, then certain types of outcomes will follow. Example: If we
build a human-level AI that is easy to copy and cheap to run, this will cause mass
unemployment among ordinary humans (Hanson 1994).
Plans. These are a specific type of conditional prediction, claiming that if someone de-
cides to implement a specific plan,then they will be successful in achieving a partic-
2
Stuart Armstrong, Kaj Sotala
ular goal. Example: We can build an AI by scanning a human brain and simulating
the scan on a computer (Sandberg and Bostrom 2008).
Issues and metastatements. This category covers relevant problems with (some or all)
approaches to AI (including sheer impossibility results),and metastatements about
the whole field. Examples: an AI cannot be built without a fundamental new un-
derstanding of epistemology (Deutsch 2012); generic AIs will have certain (poten-
tially dangerous) behaviors (Omohundro 2008).
There will inevitably be some overlap between the categories, but this division is natural
enough for our purposes. In this paper we will be looking at timeline predictions. Thanks
to the eﬀorts of Jonathan Wang and Brian Potter at the Singularity Institute,the authors
were able to make use of extensive databases of this type of predictions, reaching back
from the present day back to the 1950s. Other types of predictions will be analyzed in
subsequent papers.
2.2. Prediction Methods
Just as there are many types of predictions, there are many ways of arriving at them—
consulting crystal balls, listening to the pronouncements of experts, constructing elabo-
rate models. Our review of published predictions has shown that the prediction meth-
ods are far more varied than the types of conclusions arrived at. For the purposes of this
analysis, we’ll divide the prediction methods into the following loose scheme:
1.Causal models
2.Non-causal models
3.The outside view
4.Philosophical arguments
5.Expert authority
6.Non-expert authority
Causal model are the staple of physics: given certain facts about the situation under
consideration (momentum, energy, charge, etc.) a conclusion is reached about what the
ultimate state will be. If the facts were diﬀerent, the end situation would be diﬀerent.
But causal models are often a luxury outside of the hard sciences, whenever we lack
precise understanding of the underlying causes. Some success can be achieved with non-
causal models: without understanding what influences what, one can extrapolate trends
into the future. Moore’s law is a highly successful non-causal model (Moore 1965).
The outside view is a method of predicting that works by gathering together specific
examples and claiming that they all follow the same underlying trend. For instance, one
3
How We’re Predicting AI—or Failing To
could notice the plethora of Moore’s laws across the spectrum of computing (in numbers
of transistors, size of hard drives, network capacity, pixels per dollar . . .), note that AI
is in the same category, and hence argue that AI development must follow a similarly
exponential curve (Kurzweil 1999).
Philosophical arguments are common in the field of AI;some are simple impossibility
statements: AI is decreed to be impossible for more or less plausible reasons. But the
more thoughtful philosophical arguments point out problems that need to be resolved
to achieve AI, highlight interesting approaches to doing so, and point potential issues if
this were to be achieved.
Many predictions rely strongly on the status of the predictor: their innate expertise
giving them potential insights that cannot be fully captured in their arguments, so we
have to trust their judgment. But there are problems in relying on expert opinion, as we
shall see.
Finally, some predictions rely on the judgment or opinion of non-experts. Journalists
and authors are examples of this, but often actual experts will make claims outside their
domain of expertise. CEO’s, historians, physicists, and mathematicians will generally be
no more accurate than anyone else when talking about AI, no matter how stellar they
are in their own field (Kahneman 2011).
Predictions can use a mixture of these approaches, and often do. For instance, Ray
Kurzweil’s “Law of Time and Chaos”uses the outside view to group together evolution-
ary development,technological development,and computing into the same category,and
constructs a causal model predicting time to the “Singularity”(Kurzweil 1999). Moore’s
law (non-causal model) is a key input to this Law, and Ray Kurzweil’s expertise is the
main evidence for the Law’s accuracy.
This is the schema we will be using in this paper, and in the prediction databases we
have assembled. But the purpose of any such schema is to bring clarity to the analysis,not
to force every prediction into a particular box. We hope that the methods and approaches
used in this paper will be of general use to everyone wishing to analyze the reliability and
usefulness of predictions, in AI and beyond. Hence this schema can be freely adapted
or discarded if a particular prediction does not seem to fit it, or if an alternative schema
seems to be more useful for the analysis of the question under consideration.
3.AToolbox of Assessment Methods
The purpose of this paper is not only to assess the accuracy and reliability of some of the
AI predictions that have already been made. The purpose is to start building a “toolbox”
of assessment methods that can be used more generally, applying them to current and
future predictions.
4
Stuart Armstrong, Kaj Sotala
3.1. Extracting Verifiable Predictions
The focus of this paper is squarely on the behavior of AI. This is not a philosophical
point; we are not making the logical positivist argument that only empirically verifiable
predictions have meaning (Carnap 1928). But it must be noted that many of the vital
questions about AI—can it built, when, will it be dangerous, will it replace humans,
and so on—all touch upon behavior. This narrow focus has the added advantage that
empirically verifiable predictions are (in theory) susceptible to falsification,which means
ultimately agreement between people of opposite opinions. Predictions like these have
a very diﬀerent dynamic to those that cannot be shown to be wrong, even in principle.
To that end, we will seek to reduce the prediction to an empirically verifiable format.
For some predictions, this is automatic: they are already in the correct format. When
Kurzweil wrote “One of my key (and consistent) predictions is that a computer will pass
the Turing test by 2029,” then there is no need to change anything. Conversely, some
philosophical arguments concerning AI, such as some of the variants of the Chinese
Room argument (Searle 1980), are argued to contain no verifiable predictions at all:
an AI that demonstrated perfect human behavior would not aﬀect the validity of the
argument.
And in between there are those predictions that are partially verifiable. Then the
verifiable piece must be clearly extracted and articulated. Sometimes it is ambiguity that
must be overcome: when an author predicts an AI“Omega point”in 2040 (Schmidhuber
2007), it is necessary to read the paper with care to figure out what counts as an Omega
point and (even more importantly) what doesn’t.
Even purely philosophical predictions can have (or can be interpreted to have) veri-
fiable predictions. One of the most famous papers on the existence of conscious states
is Thomas Nagel’s “What is it like to be a bat” (Nagel 1974). In this paper, Nagel ar-
gues that bats must have mental states, but that we humans can never understand what
it is like to have these mental states. This feels purely philosophical, but does lead to
empirical predictions: that if the bat’s intelligence were increased and we could develop
a common language, then at some point in the conversation with it, our understanding
would reach an impasse. We would try to describe what our internal mental states felt
like, but would always fail to communicate the essence of our experience to the other
species.
Many other philosophical papers can likewise be read as having empirical predic-
tions; as making certain states of the world more likely or less—even if they seem to
be devoid of this. The Chinese Room argument, for instance, argues that formal algo-
rithms will lack the consciousness that humans possess (Searle 1980). This may seem
to be an entirely self-contained argument—but consider that a lot of human behavior
revolves around consciousness, be it discussing it, commenting on it, defining it or in-
5
How We’re Predicting AI—or Failing To
tuitively noticing it in others. Hence if we believed the Chinese Room argument, and
were confronted with two AI projects, one based on advanced algorithms and one based
on modified human brains, we would be likely to believe that the second project is more
likely to result in an intelligence that seemed conscious than the first. This is simply be-
cause we wouldn’t believe that the first AI could ever be conscious, and that it is easier
to seem conscious when one actually is. And that gives an empirical prediction.
Note that the authors of the predictions may disagree with our “extracted” conclu-
sions. This is not necessarily a game breaker. For instance, even if there is no formal link
between the Chinese Room model and the prediction above, it’s still the case that the
intuitive reasons for believing the model are also good reasons for believing the predic-
tion. Our aim should always be to try to create useful verifiable predictions in any way
we can. In this way, we can make use of much more of the AI literature. For instance,
Lucas argues that AI is impossible because it could not recognize the truth of its own
Gödel sentence (Lucas 1961).1This is a very strong conclusion, and we have to accept
a lot of Lucas’s judgments before we agree with it. Replacing the conclusion with the
weaker (and verifiable) “self reference will be an issue with advanced AI,and will have to
be dealt with somehow by the programmers” gives us a useful prediction which is more
likely to be true.
Care must be taken when applying this method: the point is to extract a useful ver-
ifiable prediction, not to weaken or strengthen a reviled or favored argument. The very
first stratagems in Shopenhauer’s “The Art of Always being Right”(Schopenhauer 1831)
are to extend and over-generalize the consequences of your opponent’s argument; con-
versely, one should reduce and narrow down one’s own arguments. There is no lack of
rhetorical tricks to uphold one’s own position, but if one is truly after the truth, one
must simply attempt to find the most reasonable empirical version of the argument; the
truth-testing will come later.
This method often increases uncertainty, in that it often narrows the consequences of
the prediction, and allows more possible futures to exist, consistently with that predic-
tion. For instance, Bruce Edmonds (Edmonds 2008), building on the “No Free Lunch”
results (Wolpert and Macready 1995), demonstrates that there is no such thing as a
universal intelligence: no intelligence that performs better than average in every cir-
cumstance. Initially this seems to rule out AI entirely; but when one analyzes what this
means empirically,one realizes there is far less to it. It does not forbid an algorithm from
performing better than any human being in any situation any human being would ever
1. A Gödel sentence is a sentence G that can be built in any formal system containing arithmetic. G is
implicitly self-referential, as it is equivalent with “there cannot exist a proof of G”. By construction, there
cannot be a consistent proof of G from within the system.
6
Stuart Armstrong, Kaj Sotala
encounter, for instance. So our initial intuition, which was to rule out all futures with
AIs in them, is now replaced by the realization that we have barely put any constraints
on the future at all.
3.2. Clarifying and Revealing Assumptions
The previous section was concerned with the predictions’ conclusions. Here we will
instead be looking at its assumptions,and the logical structure of the argument or model
behind it. The objective is to make the prediction as rigorous as possible
Philosophers love doing this: taking apart argument, adding caveats and straight-
ening out the hand-wavy logical leaps. In a certain sense, it can be argued that ana-
lytic philosophy is entirely about making arguments rigorous. One of the oldest meth-
ods in philosophy—the dialectic (Plato, 380BCE)—also plays this role, with concepts
getting clarified during the conversation between philosophers and various Athenians.
Though this is perhaps philosophy’s greatest contribution to knowledge, it is not ex-
clusively the hunting ground of philosophers. All rational fields of endeavor do—and
should!—benefit from this kind of analysis.
Of critical importance is revealing hidden assumptions that went into the predictions.
These hidden assumptions—sometimes called Enthymematic gaps in the literature (Fal-
lis 2003)—are very important because they clarify where the true disagreements lie, and
where we need to focus our investigation in order to find out the truth of prediction.
Too often, competing experts will make broad-based arguments that fly past each other.
This makes choosing the right argument a matter of taste, prior opinions and our ad-
miration of the experts involved. But if the argument can be correctly deconstructed,
then the source of the disagreement can be isolated, and the issue can be decided on
much narrower grounds—and its much clearer whether the various experts have rele-
vant expertise or not (see Section 3.4). The hidden assumptions are often implicit,so it is
perfectly permissible to construct assumptions that the predictors were not consciously
aware of using.
For example, let’s look again at the Gödel arguments mentioned in Section 3.1. The
argument shows that formal systems of a certain complexity must be either incomplete
(unable to see that their Gödel sentence is true) or inconsistent (proving false state-
ments). This is contrasted with humans, who—allegedly—use meta-reasoning to know
that their own Gödel statements are true. It should first be noted here that no one has
written down an actual “human Gödel statement,”so we cannot be sure humans would
actually figure out that it is true.2Also, humans are both inconsistent and able to deal
2. One could argue that, by definition, a human Gödel statement must be one that humans cannot
recognize as being a human Gödel statement!
7
How We’re Predicting AI—or Failing To
with inconsistencies without a complete collapse of logic. In this, they tend to diﬀer
from AI systems, though some logic systems such as relevance logic do mimic the same
behavior (Routley and Meyer 1976). In contrast, both humans and AIs are not logi-
cally omniscient—they are not capable of proving everything provable within their logic
system (the fact that there are an infinite number of things to prove being the problem
here). So this analysis demonstrates the hidden assumption in Lucas’s argument: that
the behavior of an actual computer program running on a real machine is more akin to
that of a logically omniscient formal agent, than it would be to a real human being. That
assumption may be flawed or correct, but is one of the real sources of disagreement over
whether Gödelian arguments rule out artificial intelligence.
Again,it needs to be emphasized that the purpose is to clarify and analyze arguments,
not to score points for one side or the other. It is easy to phrase assumptions in ways that
sound good or bad for either “side.” It is also easy to take the exercise too far: finding
more and more minor clarifications or specific hidden assumptions until the whole pre-
diction becomes a hundred page mess of over-detailed special cases. The purpose is to
clarify the argument until it reaches the point where all (or most) parties could agree that
these assumptions are the real sources of disagreement. And then we can consider what
empirical evidence, if available, or expert opinion has to say about these disagreements.
There is surprisingly little published on the proper way of clarifying assumptions,
making this approach more an art than a science. If the prediction comes from a model,
we have some standard tools available for clarifying, though see Morgan and Henrion
(1990). Most of these methods work by varying parameters in the model and checking
that this doesn’t cause a breakdown in the prediction.
3.2.1. ModelTesting and Counterfactual Resiliency
Though the above works from inside the model,there are very few methods that can test
the strength of a model from the outside. This is especially the case for non-causal mod-
els: what are the assumptions behind Moore’s famous law (Moore 1965),or Robin Han-
son’s model that we are due for another technological revolution, based on the timeline
of previous revolutions (Hanson 2009)? If we can’t extract assumptions, we’re reduced
to saying “that feel right/wrong to me,”and therefore we’re getting nowhere.
The authors have come up with a putative way of testing the assumptions of such
models (in the case of Moore’s law, the empirical evidence in favor is strong, but there is
still the question of what is powering the law and whether it will cross over to new chip
technologies again and again). It involves giving the model a counterfactual resiliency
check: imagining that world history had happened slightly diﬀerently, and checking
whether the model would have stood up in those circumstances. Counterfactual changes
are permitted to anything that the model ignores.
8
Stuart Armstrong, Kaj Sotala
The purpose of this exercise is not to rule out certain models depending on one’s own
preferred understanding of history (e.g. “Protestantism was essential to the industrial
revolution, and was a fluke due to Martin Luther; so it’s very likely that the industrial
revolution would not have happened in the way or timeframe that it did,hence Hanson’s
model—which posits the industrial revolutions’s dates as inevitable—is wrong”). Instead
it is to illustrate the tension between the given model and other models of history (e.g.
“The assumptions that Protestantism was both a fluke and essential to the industrial
revolution are in contradiction with Hanson’s model. Hence Hanson’s model implies
that either Protestantism was inevitable or that it was non-essential to the industrial
revolution, a extra hidden assumption”). The counterfactual resiliency exercise has been
carried out at length in an online post (Armstrong 2012). The general verdict seemed
to be that Hanson’s model contradicted a lot of seemingly plausible assumptions about
technological and social development. Moore’s law, on the other hand, seemed mainly
dependent on the continuing existence of a market economy and the absence of major
catastrophes.
This method is new, and will certainly be refined in future. Again, the purpose of the
method is not to rule out certain models, but to find the nodes of disagreement.
3.2.2. More Uncertainty
Clarifying assumptions often ends up increasing uncertainty, as does revealing hidden
assumptions. The previous section focused on extracting verifiable predictions, which
often increases the range of possible worlds compatible with a prediction. Here, by
clarifying and caveating assumptions, and revealing hidden assumption, we reduce the
number of worlds in which the prediction is valid. This means that the prediction puts
fewer constraints on our expectations. In counterpart, of course, the caveated prediction
is more likely to be true.
3.3. Empirical Evidence
The gold standard in separating true predictions from false ones must always be empirical
evidence. The scientific method has proved to be the best way of disproving false hy-
potheses, and should be used whenever possible. Other methods, such as expert opinion
or unjustified models, come nowhere close.
The problem with empirical evidence is that . . . it is generally non-existent in the
AI prediction field. Since AI predictions are all about the existence and properties of
a machine that hasn’t yet been built, that no-one knows how to build or whether it
actually can be built, there is little opportunity for the whole hypothesis-prediction-
testing cycle. This should indicate the great difficulties in the field. Social sciences, for
instance,are often seen as the weaker cousins of the hard sciences,with predictions much
9
How We’re Predicting AI—or Failing To
more contentious and less reliable. And yet the social sciences make use of the scientific
method, and have access to some types of repeatable experiments. Thus any prediction
in the field of AI should be treated as less likely than any social science prediction.
That generalization is somewhat over-harsh. Some AI prediction methods hew closer
to the scientific method, such as the whole brain emulations model (Sandberg and
Bostrom 2008)—it makes testable predictions along the way. Moore’s law is a wildly
successful prediction, and connected to some extent with AI. Many predictors (e.g.
Kurzweil) make partial predictions on the road towards AI; these can and should be
assessed—track records allow us to give some evidence to the proposition “this expert
knows what they’re talking about.” And some models also allow for a degree of testing.
So the field is not void of empirical evidence; it’s just that there is so little of it, and to a
large extent we must put our trust in expert opinion.
3.4. Expert Opinion
Reliance on experts is nearly unavoidable in AI prediction. Timeline predictions are of-
ten explicitly based on experts’feelings; even those that consider factors about the world
(such as computer speed) need an expert judgment about why that factor is considered
and not others. Plans need experts to come up with them and judge their credibility.
And unless every philosopher agrees on the correctness of a particular philosophical ar-
gument, we are dependent to some degree on the philosophical judgment of the author.
It is the purpose of all the methods described above that we can refine and caveat a
prediction, back it up with empirical evidence whenever possible, and thus clearly high-
light the points where we need to rely on expert opinion. And so can focus on the last
remaining points of disagreement: the premises themselves (that is of course the ideal
situation: some predictions are given directly with no other basis but expert authority,
meaning there is nothing to refine).
Should we expect experts to be good at this task? There have been several projects
over the last few decades to establish the domains and tasks where we would expect
experts to have good performance (Shanteau 1992; Kahneman and Klein 2009). Table
1 summarizes the results:
Not all of these are directly applicable to the current paper (are predictions about
human level AIs predictions about things, or about behavior?). One of the most impor-
tant factors is whether experts get feedback, preferably immediate feedback. We should
expect the best expert performance when their guesses are immediately confirmed or
disconfirmed. When feedback is unavailable or delayed, or the environment isn’t one
that give good feedback, then expert performance drops precipitously (Kahneman and
Klein 2009; Kahneman 2011).
10
Stuart Armstrong, Kaj Sotala
Table 1: Table of task properties conducive to good and poor expert performance.
Good performance: Poor performance:
Static stimuli Dynamic (changeable) stimuli
Decisions about things Decisions about behavior
Experts agree on stimuli Experts disagree on stimuli
More predictable problems Less predictable problems
Some errors expected Few errors expected
Repetitive tasks Unique tasks
Feedback available Feedback unavailable
Objective analysis available Subjective analysis only
Problem decomposable Problem not decomposable
Decision aids common Decision aids rare
Table 1 applies to both domain and task. Any domain of expertise strongly in the
right column will be one where we expect poor expert performance. But if the individual
expert tries to move their own predictions into the left column (maybe by decomposing
the problem as far as it will go, training themselves on related tasks where feedback is
available . . .) they will be expected to perform better. In general, we should encourage
this type of approach.
When experts fail,there are often simple algorithmic models that demonstrate better
performance (Grove et al. 2000). In these cases, the experts often just spell out their
criteria, design the model in consequence, and let the model give its predictions: this
results in better predictions than simply asking the expert in the first place. Hence we
should also be on the lookout for experts who present their findings in the form of a
model.
As everyone knows, experts sometimes disagree. This fact strikes at the very heart of
their supposed expertise. We listen to them because they have the skills and experience to
develop correct insights. If other experts have gone through the same process and come
to an opposite conclusion,then we have to conclude that their insights do not derive from
their skills and experience, and hence should be discounted. Now if one expert opinion
is a fringe position held by only a few experts,we may be justified in dismissing it simply
as an error. But if there are diﬀerent positions held by large numbers of disagreeing
experts, how are we to decide between them? We need some sort of objective criteria:
we are not experts in choosing between experts, so we have no special skills in deciding
the truths on these sorts of controversial positions.
11
How We’re Predicting AI—or Failing To
What kind of objective criteria could there be? A good track record can be an indica-
tor, as is a willingness to make verifiable, non-ambiguous predictions. A better connec-
tion with empirical knowledge and less theoretical rigidity are also positive indications
(Tetlock 2005), and any expert that approached their task with methods that were more
on the left of the table than on the right should be expected to be more correct. But these
are second order phenomena—we’re looking at our subjective interpretation of expert’s
subjective opinion—so in most cases, when there are strong disagreement between ex-
perts, we simply can’t tell which position is true.
3.4.1. Grind Versus Insight
Some AI prediction claim that AI will result from grind: i.e. lots of hard work and
money. Other claim that AI will need special insights: new unexpected ideas that will
blow the field wide open (Deutsch 2012).
In general, we are quite good at predicting grind. Project managers and various lead-
ers are often quite good at estimating the length of projects (as long as they’re not directly
involved in the project (Buehler, Griffin, and Ross 1994)). Even for relatively creative
work,people have sufficient feedback to hazard reasonable guesses. Publication dates for
video games, for instance, though often over-optimistic, are generally not ridiculously
erroneous—even though video games involve a lot of creative design, play-testing, art,
programing the game “AI,” etc. Moore’s law could be taken as an ultimate example of
grind: we expect the global eﬀorts of many engineers across many fields to average out
to a rather predictable exponential growth.
Predicting insight, on the other hand, seems a much more daunting task. Take the
Riemann hypothesis, a well-established mathematical hypothesis from 1885, (Riemann
1859). How would one go about estimating how long it would take to solve? How
about the P=NPhypothesis in computing? Mathematicians seldom try and predict
when major problems will be solved, because they recognize that insight is very hard to
predict. And even if predictions could be attempted (the age of the Riemann’s hypothesis
hints that it probably isn’t right on the cusp of being solved), they would need much
larger error bars than grind predictions. If AI requires insights, we are also handicapped
by the fact of not knowing what these insights are (unlike the Riemann hypothesis,
where the hypothesis is clearly stated, and only the proof is missing). This could be
mitigated somewhat if we assumed there were several diﬀerent insights, each of which
could separately lead to AI. But we would need good grounds to assume that.
Does this mean that in general predictions that are modeling grind should be ac-
cepted more than predictions that are modeling insight? Not at all. Predictions that are
modeling grind should only be accepted if they can make a good case that producing
an AI is a matter grind only. The predictions around whole brain emulations (Sandberg
12
Stuart Armstrong, Kaj Sotala
and Bostrom 2008), are one of the few that make this case convincingly; this will be
analyzed in a subsequent paper.
3.4.2. Non-expert Opinion
It should be borne in mind that all the caveats and problems with expert opinion apply
just as well to non-experts. With one crucial diﬀerence: we have no reason to trust the
non-expert’s opinion in the first place. That is not to say that non-experts cannot come
up with good models, convincing timelines, or interesting plans and scenarios. It just
means that our assessment of the quality of the prediction depends only on what we
are given; we cannot extend a non-expert any leeway to cover up a weak premise or a
faulty logical step. To ensure this,we should try and assess non-expert predictions blind,
without knowing who the author is. If we can’t blind them, we can try and get a similar
eﬀect by asking ourselves hypothetical questions such as: “Would I find this prediction
more or less convincing if the author was the Archbishop of Canterbury? What if it
was Warren Buﬀet? Or the Unabomber?” We should aim to reach the point where
hypothetical changes in authorship do not aﬀect our estimation of the prediction.
4.Timeline Predictions
The practical focus of this paper is on AI timeline predictions: predictions giving dates
for AIs with human-comparable cognitive abilities. Researchers from the Singularity
Institute have assembled a database of 257 AI predictions since 1950, of which 95 in-
clude AI timelines.
4.1. Subjective Assessment
A brief glance atTable 1 allows us to expect that AI timeline predictions will generally be
of very poor quality. The only factor that is unambiguously positive for AI predictions is
that prediction errors are expected and allowed: apart from that,the task seems singularly
difficult,especially on the key issue of feedback. An artificial intelligence is a hypothetical
machine, which has never existed on this planet before and about whose properties we
have but the haziest impression. Most AI experts will receive no feedback whatsoever
about their predictions, meaning they have to construct them entirely based on their
untested impressions.
There is nothing stopping experts from decomposing the problem, or constructing
models which they then calibrate with available data, or putting up interim predictions
to test their assessment. And some do use these better approaches (see for instance
(Kurzweil 1999; Hanson 1994; Waltz 1988)). But a surprisingly large number don’t!
13
How We’re Predicting AI—or Failing To
Some predictions are unabashedly based simply on the feelings of the predictor (Good
1962; Armstrong 2007).
Yet another category are of the “Moore’s law hence AI” type. They postulate that
AI will happen when computers reach some key level, often comparing with some
key property of the brain (number of operations per second (Bostrom 1998), or neu-
rones/synapses3). In the division established in Section 3.4.1, this is pure “grind” argu-
ment: AI will happen after a certain amount of work is performed. But, as we saw, these
kinds of arguments are only valid if the predictor has shown that reaching AI does not
require new insights! And that step is often absent from the argument.
4.2. Timeline Prediction Data
The above were subjective impressions, formed while looking over the whole database.
To enable more rigorous analysis,the various timeline predictions were reduced to a sin-
gle number for purposes of comparison: this would be the date upon which the predictor
expected “human level AI”to be developed.
Unfortunately not all the predictions were in the same format. Some gave ranges,
some gave median estimates,some talked about superintelligent AI,others about slightly
below-human AI. In order to make the numbers comparable, one of the authors (Stuart
Armstrong) went through the list and reduced the various estimates to a single number.
He followed the following procedure to extract a “Median human-level AI estimate”:
When a range was given, he took the mid-point of that range (rounded down). If a
year was given with a 50% likelihood estimate,he took that year. If it was the collection of
a variety of expert opinions,he took the prediction of the median expert. If the predictor
foresaw some sort of AI by a given date (partial AI or superintelligent AI), and gave no
other estimate, he took that date as their estimate rather than trying to correct it in
one direction or the other (there were roughly the same number of subhuman AIs as
suphuman AIs in the list, and not that many of either). He read extracts of the papers
to make judgement calls when interpreting problematic statements like “within thirty
years”or “during this century”(is that a range or an end-date?). Every date selected was
either an actual date given by the predictor, or the midpoint of a range.4
It was also useful to distinguish between popular estimates, performed by journalists,
writers or amateurs, from those predictions done by those with expertise in relevant
fields (AI research, computer software development, etc.). Thus each prediction was
3. See for instance Dani Eder’s 1994 Newgroup posting http://www.aleph.se/Trans/Global/
Singularity/singul.txt
4. The data can be found at http://www.neweuropeancentury.org/SIAI-FHI\_AI\_
predictions.xls ; readers are encouraged to come up with their own median estimates.
14
Stuart Armstrong, Kaj Sotala
noted as “expert”or “non-expert”; the expectation being that experts would demonstrate
improved performance over non-experts.
Figure 1 graphs the results of this exercise (the range has been reduced; there were
seven predictions setting dates beyond the year 2100, three of them expert.)
Figure 1: Median estimate for human-level AI, graphed against date of prediction.
As can be seen, expert predictions span the whole range of possibilities and seem
to have little correlation with each other. The range is so wide—fifty year gaps between
predictions are common—that it provides strong evidence that experts are not providing
good predictions. There does not seem to be any visible diﬀerence between expert and
non-expert performance either,suggesting that the same types of reasoning may be used
in both situations, thus negating the point of expertise.
Two explanations have been generally advanced to explain poor expert performance
in these matters. The first, the so-called Maes-Garreau law5posits that AI experts pre-
dict AI happening towards the end of their own lifetime. This would make AI into
a technology that would save them from their own deaths, akin to a “Rapture of the
Nerds.”
The second explanation is that AI is perpetually fifteen to twenty-five years into the
future. In this way (so the explanation goes), the predictor can gain credit for working
on something that will be of relevance, but without any possibility that their prediction
could be shown to be false within their current career.
5. Kevin Kelly, editor of Wired magazine, created the law in 2007 after being influenced by Pattie
Maes at MIT and Joel Garreau (author of Radical Evolution).
15
How We’re Predicting AI—or Failing To
We’ll now look at the evidence for these two explanations.
4.2.1. Nerds Don’t Get Raptured
Fifty-five predictions were retained, in which it was possible to estimate the predictor’s
expected lifespan. Then the diﬀerence between their median prediction and this lifespan
was computed (a positive diﬀerence meaning they would expect to die before AI, a neg-
ative diﬀerence meaning they didn’t). A zero diﬀerence would be a perfect example of
the Maes-Garreau law: the predictor expects AI to be developed at the exact end of their
life. This number was then plotted again the predictor’s age in Figure 2 (the plot was
restricted to those predictions within thirty years of the predictor’s expected lifetime).
Figure 2: Diﬀerence between the predicted time to AI and the predictor’s life expectancy,
graphed against the predictor’s age.
From this,it can be seen that the Maes-Garreau law is not borne out by the evidence:
only twelve predictions (22% of the total) were within five years in either direction of
the zero point.
4.2.2. Twenty Years to AI
The “time to AI”was computed for each expert prediction. This was graphed in Figure 3.
This demonstrates a definite increase in the 16–25 year predictions: 21 of the 62 expert
16
Stuart Armstrong, Kaj Sotala
predictions were in that range (34%). This can be considered weak evidence that experts
do indeed prefer to predict AI happening in that range from their own time.
Figure 3: Time between the arrival of AI and the date the prediction was made, for
expert predictors.
But the picture gets more damning when we do the same plot for the non-experts,
as in Figure 4. Here, 13 of the 33 predictions are in the 16-25 year range. But more dis-
turbingly, the time to AI graph is almost identical for experts and non-experts! Though
this does not preclude the possibility of experts being more accurate,it does hint strongly
that experts and non-experts may be using similar psychological procedures when cre-
ating their estimates.
The next step is to look at failed predictions. There are 15 of those, most dating to
before the “AI winter” in the eighties and nineties. These have been graphed in Figure
5—and there is an uncanny similarity with the other two graphs! So expert predictions
are not only indistinguishable from non-expert predictions, they are also indistinguish-
able from past failed predictions. Hence it is not unlikely that recent predictions are
suﬀering from the same biases and errors as their predecessors
5.Conclusion
This paper, the first in a series analyzing AI predictions, focused on the reliability of AI
timeline predictions (predicting the dates upon which “human-level” AI would be de-
17
How We’re Predicting AI—or Failing To
Figure 4: Time between the arrival of AI and the date the prediction was made, for
non-expert predictors.
Figure 5: Time between the arrival of AI and the date the prediction was made, for
failed predictions.
18
Stuart Armstrong, Kaj Sotala
veloped). These predictions are almost wholly grounded on expert judgment. The biases
literature classified the types of tasks on which experts would have good performance,
and AI timeline predictions have all the hallmarks of tasks on which they would perform
badly.
This was borne out by the analysis of 95 timeline predictions in the database assem-
bled by the Singularity Institute. There were strong indications therein that experts per-
formed badly. Not only were expert predictions spread across a wide range and in strong
disagreement with each other, but there was evidence that experts were systematically
preferring a “15 to 25 years into the future”prediction. In this, they were indistinguish-
able from non-experts, and from past predictions that are known to have failed. There
is thus no indication that experts brought any added value when it comes to estimating
AI timelines. On the other hand, another theory—that experts were systematically pre-
dicting AI arrival just before the end of their own lifetime—was seen to be false in the
data we have.
There is thus strong grounds for dramatically increasing the uncertainty in any AI
timeline prediction.
19
How We’re Predicting AI—or Failing To
Acknowledgments
The authors wish to acknowledge the help and support of the Singularity Institute, the
Future of Humanity Institute and the James Martin School, as well as the individual
advice of Nick Bostrom, Luke Muelhauser, Vincent Mueller, Anders Sandberg, Lisa
Makros, Sean O’Heigeartaigh, Daniel Dewey, Eric Drexler and the online community
of Less Wrong.
References
Armstrong, Stuart. 2007. “Chaining God: A Qualitative Approach to AI, Trust and Moral Systems.”
Unpublished manuscript, October 20. Accessed December 31, 2012. http://www.neweuropeanc
entury.org/GodAI.pdf .
. 2012. “Counterfactual Resiliency Test for Non-Causal Models.” Less Wrong (blog), August 30.
http://lesswrong.com/lw/ea8/counterfactual\_resiliency\_test\_for\_noncausal/ .
Bostrom, Nick. 1998.“How Long Before Superintelligence?” International Journal of Futures Studies 2.
Buehler, Roger, Dale Griffin, and Michael Ross. 1994. “Exploring the ‘Planning Fallacy’: Why People
Underestimate Their Task Completion Times.” Journal of Personality and Social Psychology 67 (3):
366–381.
Carnap, Rudolf. 1928. Der Logische Aufbau der Welt [The logical structure of the world]. Berlin-
Schlachtensee: Weltkreis.
Darrach, Brad. 1970.“Meet Shakey, the First Electronic Person.” Life,November 20, 58–68.
Deutsch, David. 2012. “The Very Laws of Physics Imply That Artificial Intelligence Must Be Possible.
What’s Holding Us Up?” Aeon, October 3. Accessed December 6, 2012. http://www.aeonmagaz
ine.com/being-human/david-deutsch-artificial-intelligence/ .
Edmonds, Bruce. 2008. “The Social Embedding of Intelligence: Towards Producing a Machine That
Could Pass the Turing Test.” In Parsing the Turing Test: Philosophical and Methodological Issues in
the Quest for the Thinking Computer, edited by Robert Epstein, Gary Roberts, and Grace Beber, 211–
235. New York: Springer.
Fallis, Don. 2003.“Intentional Gaps In Mathematical Proofs.” Synthese 134 (1–2): 45–69.
Good, Irving John.1962. The Scientist Speculates: An Anthology of Partly-Baked Ideas. Edited by Alan James
Mayne and John Maynard Smith. New York: Basic Books.
Grove, William M., David H. Zald, Boyd S. Lebow, Beth E. Snitz, and Chad Nelson. 2000. “Clinical
Versus Mechanical Prediction: A Meta-Analysis.” Psychological Assessment 12 (1): 19–30.
Hall, John Storrs. 2011. Further Reflections on the Timescale of AI. Melbourn, Australia, November 30.
Paper presented at the Solomonoﬀ 85th Memorial Conference. http://www.solomonoff85thm
emorial.monash.edu/accepted\_papers.html .
Hanson, Robin. 1994. “If Uploads Come First: The Crack of a Future Dawn.” Extropy 6 (2).http:
//hanson.gmu.edu/uploads.html .
20
Stuart Armstrong, Kaj Sotala
. 2009. “The Economics of Brain Emulations.” In Unnatural Selection: The Challenges of Engineer-
ing Tomorrow’s People, edited by Peter Healey and Steve Rayner. Science in Society. Sterling, VA:
Earthscan.
Jacquette, Dale. 1987.“Metamathematical Criteria for Minds and Machines.” Erkenntnis 27 (1): 1–16.
Kahneman, Daniel. 2011. Thinking, Fast and Slow. 1st ed. New York: Farrar, Straus / Giroux.
Kahneman, Daniel, and Gary Klein. 2009. “Conditions for Intuitive Expertise: A Failure to Disagree.”
American Psychologist 64 (6): 515–526.
Kurzweil, Ray. 1999. The Age of Spiritual Machines: When Computers Exceed Human Intelligence. New York:
Viking.
Lucas, J. R. 1961.“Minds, Machines and Gödel.” Philosophy 36 (137): 112–127.
Moore, Gordon.1965.“Cramming More Components onto Integrated Circuits.” Electronics 38 (8): 114–
117.http://download.intel.com/museum/Moores\_Law/Articles-Press\_Releases/
Gordon\_Moore\_1965\_Article.pdf .
Morgan, M. Granger,and Max Henrion.1990. Uncertainty: A Guide to Dealing with Uncertainty in Quan-
titative Risk and Policy Analysis. New York: Cambridge University Press.
Nagel, Thomas.1974.“What Is It Like to Be a Bat?” Philosophical Review 83 (4): 435–450. http://www.
jstor.org/stable/2183914 .
Omohundro, Stephen M. 2008. “The Basic AI Drives.” In Artificial General Intelligence 2008: Proceedings
of the First AGI Conference, edited by Pei Wang,Ben Goertzel,and Stan Franklin,483–492.Frontiers
in Artificial Intelligence and Applications 171. Amsterdam: IOS.
Plato.380BCE. The Republic. Translated by Benjamin Jowett.Republication of 1894 standard text.Project
Gutenberg. http://www.gutenberg.org/files/1497/1497-h/1497-h.htm .
Riemann, Bernhard. 1859. “Ueber die Anzahl der Primzahlen unter einer gegebenen Grösse” [On the
number of primes less than a given magnitude]. Monatsberichte der Berliner Akademie (November):
671–680.
Routley, Richard, and Robert K. Meyer. 1976.“Dialectical Logic, Classical Logic, and the Consistency of
the World.” Studies in Soviet Thought 16 (1–2): 1–25.
Sandberg, Anders,and Nick Bostrom.2008. Whole Brain Emulation: A Roadmap. Technical Report,2008-
3. Future of Humanity Institute, University of Oxford. http://www.fhi.ox.ac.uk/wp-
content/uploads/brain-emulation-roadmap-report1.pdf .
Schmidhuber, Jürgen.2007.“The New AI: General & Sound & Relevant for Physics.”In Artificial General
Intelligence, edited by Ben Goertzel and Cassio Pennachin,175–198.CognitiveTechnologies.Berlin:
Springer.
Schopenhauer, Arthur.1831. The Art of Being Right: 38 Ways to Win an Argument [Eristische dialektik: die
kunst, recht zu behalten].Translated by Thomas Bailey Saunders. Republication of 1896 translation.
Wikisource. http://en.wikisource.org/wiki/The\_Art\_of\_Being\_Right .
Searle, John R. 1980.“Minds, Brains, and Programs.” Behavioral and Brain Sciences 3 (03): 417–424.
Shanteau, James. 1992. “Competence in Experts: The Role of Task Characteristics.” Organizational Be-
havior and Human Decision Processes 53 (2): 252–266.
Tetlock, Philip E. 2005. Expert Political Judgment: How Good is it? How Can We Know? Princeton, NJ:
Princeton University Press.
21
How We’re Predicting AI—or Failing To
Turing, A. M. 1950.“Computing Machinery and Intelligence.” Mind 59 (236): 433–460.
Waltz, David L.1988.“The Prospects for BuildingTruly Intelligent Machines.”In“Artificial Intelligence,”
Daedalus 117 (1): 191–212. http://www.jstor.org/stable/20025144 .
Wolpert, David H., and William G. Macready. 1995. No Free Lunch Theorems for Search. Santa Fe, NM:
The Santa Fe Institite, February 6. http://www.santafe.edu/media/workingpapers/95-02-
010.pdf .
22


Quantilizers: A safer alternative to maximizers for limited optimization

Quantilizers: A Safer Alternative to Maximizers for Limited Optimization
Jessica Taylor
Machine Intelligence Research Institute
jessica@intelligence.org
Abstract
In the eld of AI, expected utility maximizers
are commonly used as a model for idealized
agents. However, expected utility maximization
can lead to unintended solutions when the util-
ity function does not quantify everything the
operators care about: imagine, for example, an
expected utility maximizer tasked with winning
money on the stock market, which has no re-
gard for whether it accidentally causes a market
crash. Once AI systems become suciently intel-
ligent and powerful, these unintended solutions
could become quite dangerous. In this paper,
we describe an alternative to expected utility
maximization for powerful AI systems, which we
callexpected utility quantilization . This could
allow the construction of AI systems that do
not necessarily fall into strange and unantici-
pated shortcuts and edge cases in pursuit of
their goals.
1 Expected Utility Maximization
Many frameworks for studying idealized agents assume
that agents attempt to maximize the expectation of
some utility function, as seen in the works of Heckerman
and Horvitz [1], Bacchus and Grove [2], and Chajewska,
Koller, and Parr [3]. von Neumann and Morgenstern [4]
provide compelling arguments that any rational agent
with consistent preferences must act as if it is maximizing
some utility function, and Russell and Norvig [5, chap.
16] use the expected utility maximization framework
extensively as the basic model of an idealized rational
agent. Formally, let Abe the nite1set of available ac-
tions,Othe set of possible outcomes, and W:A! O
map each action to the predicted outcome distribution
resulting from that action2. LetU:O! [0;1] be a
Research supported by the Machine Intelligence Research
Institute (intelligence.org).
1Expected utility maximizers and quantilizers can also
be dened for innite set of actions, but in this paper we
discuss only nite action sets for simplicity.
2Dening Wis not straightforward. See Rosenbaum and
Rubin [6] for a formalization of the \causal eects" of anutility function, with U(o) being the utility of outcome
o. Then an expected utility maximizer is an agent that
chooses an action a2Athat maximizes E[U(W(a))].
We make no argument against expected utility max-
imization on the grounds of rationality. However, max-
imizing the expectation of some utility function could
produce large unintended consequences whenever Udoes
not accurately capture all the relevant criteria. Some
unintended consequences of this form can already be ob-
served in modern AI systems. For example, consider the
genetic algorithm used by Nguyen, Yosinski, and Clune
[8] to generate an image which would be classied by a
deep neural network as a starsh, with extremely high
condence. The resulting image ended up completely
unrecognizable, looking nothing at all like a starsh.
Of course, Nguyen, Yosinski, and Clune [8] intended
to develop images that would be mis-classied, but
this demonstrates the point that an algorithm directed
to nd the image that most classied as a starsh,
generated an image that was very strange indeed.
Another example of unintended solutions resulting
from maximization is given by Thompson [9], who used
genetic algorithms to \articially evolve" a circuit that
could dierentiate between two dierent inputs via a 10 x
10 eld programmable gate array (FPGA). The resulting
circuit worked, using only 21 of the 100 available FPGA
cells. Curiously, ve of those cells were not connected
to either the input or the output|the evolved circuit
was making use of them via the physical features of
that specic chip and machine (Thompson suggest that
the circuit was taking advantage of electromagnetic
coupling or power-supply loading). If the goal was to
design a circuit that could also work on other chips,
then this \maximization" process (of nding a very
small algorithm that, on that one circuit board, was
good at distinguishing between two inputs) produced
an unintended result that was dependent upon physical
features of the chip that the designers thought were
irrelevant. The resulting circuit performed well by the
test metric, but the result would not likely have been
usable on any other physical chips.
action, whose maximization results in causal decision theory;
see Soares and Fallenstein [7] for discussion of alternative
decision theories.
1
Unintended solutions of this form could become quite
dangerous in highly autonomous articially intelligent
systems with capabilities that strongly exceed those of
humans. Bostrom [10] discusses a number of potential
dangers stemming from expected utility maximization in
powerful agents. He considers \superintelligent" agents,
dened as agents that are \smarter than the best hu-
man brains in practically every eld," and describes
the problem of perverse instantiation , his term for the
seemingly-perverse unintended consequences of direct-
ing a powerful agent to attempt to maximize a utility
function which is not in fact aligned with our goals.
For example, Bostrom [10] illustrates this by describ-
ing a machine that, when told to make humans smile,
accomplishes this goal by paralyzing human faces into
permanent smiles, rather than by causing us to smile via
the usual means. Another example is given by Soares
[11], who describes an agent directed to \cure cancer"
which attempts to kidnap human test subjects. Clearly,
before an autonomous system can be granted great au-
tonomy and power, we must have some assurances that
it will not, in pursuit of its goals, produce any such
\perverse" unintended consequences.
In both the short term and the long term, the prob-
lem with expected utility maximization is that the sys-
tem's utility function likely will not capture all the com-
plexities of human value, as humans care about many
features of the environment that are dicult to capture
in any simple utility function [11].
Of course, no practical system acting in the real
world can actually maximize expected utility, as nd-
ing the literally optimal policy is wildly intractable.
Nevertheless, it is common practice to design systems
that approximate expected utility maximization, and
insofar as expected utility maximization would be un-
satisfactory, such systems would become more and more
dangerous as algorithms and approximations improve. If
we are to design AI systems which safely pursue simple
goals, an alternative may be necessary.
2 Expected Utility Quantilization
Given that utility maximization can have many unin-
tended side eects, Armstrong, Sandberg, and Bostrom
[12] and others have suggested designing systems that
perform some sort of \limited optimization," that is,
systems which achieve their goals in some non-extreme
way, without signicantly disrupting anything outside
the system or otherwise disturbing the normal course of
events. For example, intuition says it should be possible
to direct a powerful AI system to \just make paperclips"
in such a way that it runs a successful paperclip factory,
without ever attempting to turn as much of the universe
as possible into paperclips.
Simon [13] propose instead designing expected utility
\satiscers," agents that choose any action that achieves
an expected utility above some xed threshold. In our
notation, an expected utility satiscer is one that choosesan action from the set fa2A:E[U(W(a))]tgfor
some threshold t. (Indeed, Fallenstein and Soares [14]
describe a toy model of satiscing agents in simple envi-
ronments.) However, the satiscing framework is under-
dened and not necessarily satisfactory. When it comes
to powerful intelligent agents, it may often be that the
easiest way to satisce is to maximize: if the paperclip
AI system can easily create a sub-agent that tries as hard
as it can to make paperclips, it may be that the easiest
satiscing action is simply to construct a sub-agent that
attempts to approximate something like expected utility
maximization instead.
Abbeel and Ng [15] have suggested apprenticeship
learning as an alternative to maximization when it is
dicult to explicitly specify the correct goal. In appren-
ticeship learning, an AI system learns to closely imitate
a human expert performing a task. An agent which only
executes actions that its operators would have executed
seems likely to be immune to Bostrom's problem of per-
verse instantiation. Yet while mimicking humans avoids
many unintended eects of maximization, a system that
merely mimics humans cannot easily surpass the per-
formance of those it is mimicking (except perhaps by
performing the same actions faster and more reliably).
Such a technique could prove quite useful, but wouldn't
help us design algorithms intended to outperform hu-
mans by nding plans, strategies, and techniques that
the operators would not have been able to identify.
In this paper, we propose quantilization , which in-
terpolates between mimicry and maximization in order
to gain some advantages of both.
Denition 1 (Quantilizer) .Aq-quantilizer is an agent
that, when faced with a decision problem, returns a ran-
dom action in the top qproportion of some \base dis-
tribution" over actions, sorted by the expected utility
achieved if that action is executed.
Intuitively, a 0 :1-quantilizer (which we could also call
a \ten percentilizer") selects a random action from the
top 10% of actions. More formally, let Abe the action set
and
be some distribution over actions. For example,
could be uniformly random, or it could be a distribution
estimated from human actions. Let f: [0;1]!A be
a function that represents
, but sorted by expected
utility:
8x;y2[0;1]
x>y =)E[U(W(f(x)))]E[U(W(f(y)))]
(fx2[0;1]jf(x) =ag) =
(a)
whereis the Lebesgue measure on the reals. For any
01:
8c:A!R0
EA
[c(A)]1 =)EAp[c(A)]t:(2)
This constraint states that, for any cost function assign-
ing an expected cost of no more than 1 to a random
action from
, the expected cost of an action drawn
frompis no more than t. Then, we have the following:
Lemma 2 (pts under
).Letpsatisfy 2. Then p(a)
t
(a)for alla2A.
Proof. Suppose this were not true. Then for some a
we havep(a)>t
(a). Now dene
c(a) =1=
(a) ifa=a
0 otherwise
We have
EA
[c(a)] =X
a2A
(a)c(a) =
(a)c(a) = 1:Also,
EAp[c(a)] =X
a2Ap(a)c(a)
=p(a)c(a)
>t
(a)c(a)
=t:
Jointly, these statements contradict constraint 2.
Theorem 1 (Quantilizer optimality) .Chooseq=1=t.
Then, aq-quantilizer maximizes expected U-utility sub-
ject to constraint 2.
Proof. By Lemma 2, we cannot assign more that
(a)=q
probability to any action a, so the best we can do to
maximize expected utility is to assign
(a)=qprobability
to the best actions and 0 to the rest|and this is exactly
what aq-quantilizer does.
2.2 Targeted Impact
Expected utility quantilization is intended for use in
systems that may eventually need to make important
decisions, autonomously, in environments where negative
actions could have a large cost. For example, imagine an
AI system which is programmed to autonomously trade
stocks in order to make signicant sums of money, where
that money will in tun be put to use improving the world
in some potentially dramatic fashion. Unfortunately,
the bounds of equation (1)and the cost constraints
of equation (2)apply just as well for cost functions as
benet functions: just as running a q-quantilizer will not,
in expectation, be any more than 1 =qtimes as costly as
sampling from
directly; nor will it be any more than
1=qtimes more benecial than sampling from
directly.
Thus, in a sense, a q-quantilizer has a bounded impact|
it will not make the world all that dierent than running
an AI system that samples
directly would have made
it. What we really want, though, is not low impact, but
targeted impact.
A small modication to the quantilizer specica-
tion allows us to achieve the same constraints on costs
without necessarily constraining the potential benets.
Before explaining, rst note that the stock trading AI
system has two dierent types of side eect: one is that
it produces money which can be used by its operators;
the other are side eects of trading (for example, it
might crash the market in order to make more money).
The rst side eect might cause a signicant change in
the world, e.g., if the operators use the money to fund
a charity which is highly eective at reducing global
poverty. Nevertheless, we would prefer the system to
only be guaranteeing reduced impact according to the
second sort of side eect.
That is, in this setting, we can dene the cost func-
tionc:AR!R0, such that c(a;m) determines
how much cost an action ahas, if counterfactually it
generatesmdollars (rather than the actual amount of
4
money that it generates). Given a cost function of this
form, we would like to design a quantilizer such that
8c:AR!R0;m:R
EA
[c(A;m)]1 =)EAp[c(A;m)]1=q:
That is, we require that for all counterfactual amounts of
moneym, the side eects of the stock trading algorithm
have low cost compared to the side eects of executing
actions at random from
. Thus, a system obeying this
cost constraint does not need to worry about the extra
side eects of generating signicant amounts of money
for its operators.
Mathematically, quantifying over candmat the
same time is the same as quantifying over functions of
the forma7!c(a;m) for anycandm, so the condition
is equivalent to
8c:A!R0E
[c(A)]1 =)Ep[c(A)]1=q
which is the same as the original cost constraint. There-
fore, it is possible to design a quantilizer that is guaran-
teed to have low cost (compared to
) through eects
other than money , by the same methods that guarantees
it to have low overall cost.
2.3 Algorithms for Quantilization
Just as it is impractical to design algorithms that literally
maximize expected utility in complex environments, it is
similarly impractical to design algorithms that perfectly
quantilize expected utility. Any practical system based
on expected utility quantilization would similarly have
to use some form of approximation. Fortunately, it is
not hard to see how expected utility quantilization could
be approximated.
Suppose we can compute E[U(W(a))]for each ac-
tiona, and that we wish to approximate a q-quantilizer
with respect to a distribution
. To do this, we can run
the following algorithm: rst take nsamples from
,
and then sample from QU;W;^
;q, where ^
is the empirical
distribution estimated from the nsamples. As nap-
proaches innity, ^
approaches
, soQU;W;^
;qapproaches
QU;W;
;q . Thus, given some base distribution
which
is relatively easy to sample from, and some method
for approximating E[U(W(a))]and^
, an approximate
expected utility quantilizer can be constructed in a
straightforward manner.
Indeed, this algorithm yields insight into how safe it
is to perform a random search of actions sampled from
.
Ifq= 1=n, then this algorithm simply takes nsamples
from
and returns the one with the highest expected
utility, emulating a simple random search procedure.
Since this procedure approximates q-quantilization, the
bounds that apply to quantilization can be shown to
hold for random searches over samples of
, with some
minimal modications.3 Drawbacks and Open Problems
3.1 Acquiring the Base Distribution
Clearly, a quantilizer is only as good as its base dis-
tribution
. The above constraints on the amount of
costs that running an expected utility quantilizer can
incur are in terms of multiples of the expected cost from
sampling directly from
: clearly, if it is dangerous to
sample actions from
, then it is dangerous to run a
quantilizer using
.
Dening a useful base distribution is not a straight-
forward task. One could dene the base distribution
to be some uniformly random action, but in a complex
multi-dimensional space of strategies, a uniformly ran-
dom action is highly unlikely to be useful, and so even a
105-quantilizer would be unlikely to do anything useful.
If instead
described random strategies that a human
might actually execute, then it is much more likely to
be useful. Thus, a generative model of human strategies
would make expected utility quantilization a much more
viable strategy. (See Ng and Russell [17] and Baker,
Tenenbaum, and Saxe [18] for some initial work in this
direction.)
Of course, there are a number of situations in which
a simple generative model of human behavior would not
be sucient. For example, if humans are likely to mis-
understand the problem and thereby cause catastrophe,
then a simple generative model would not suce. (And
if a human were quite likely to design an expected utility
maximizing AI system, then a quantilizer may well also
generate a maximizer.)4
The study of how to generate safe and satisfactory
base distributions
for various dierent problem do-
mains remains an open problem.
3.2 Butter
y Eects
It could be that a non-negligible percentage of ordinary
actions that humans take are either very good or very
bad, due to butter
y eects. In this case, the cost of
sampling from fairly ordinary
distributions is actually
quite high, with the expectation of
being reasonable
only because the high costs are canceled out by equally
high benets. However, in situations such as these, it is
quite possible for a quantilizer to preserve the costs of

without necessarily preserving many of the benets.
For example, suppose that the quantilizer is trying
to make money on the stock market. There is a 10%
chance of the stock market crashing in the next year.
While a random sequence of trades made by a human
will not signicantly change the chance of a crash in
expectation, it might be the case that a certain sequence
of 3 ordinary trades (a sequence that a human would be
0.1% likely to make) will greatly increase the chance of
a stock market crash through butter
y eects. In this
4See Christiano [19] for additional discussion of superin-
telligent predictions of human decisions, and the diculties
that arise in this context.
5
case, a 0.001-quantilizer that can predict these eects
might make these trades if it can prot from the crash.
It is not clear if this scenario is even theoretically
possible, since it requires extremely accurate prediction
of the future. Nevertheless, it would be prudent to study
this problem further and possibly design variations on
quantilization that avoid this problem.
3.3 Quantilization in Repeated Games
Suppose we have nindependent games. On game i,
the agent chooses an action ai2Aiand receives utility
Ui(Wi(a)), where each Wi:A! Oreturns the out-
come for game i, and eachUi:O! [0;1] measures the
utility achieved in that game. There are two dierent
ways to design a quantilizer that plays these ngames.
We could design the agent to q-quantilize each game
independently, choosing from the best qproportion of
actions in each game independently according to ndif-
ferent base distributions
i. This distribution for the
sequence of actions in each game could be written asQn
i=1QUi;Wi;
i;q.
Alternatively, we could design the agent to
q-quantilize a sequence of actions for the joint
game. Here we dene the joint game as A=Qn
i=1Ai,W(a1;:::;an) = (W1(a1);:::;Wn(an)) and
U(o1;:::;on) =Pn
i=1Ui(oi). This action distribution
for aq-quantilizer with respect to the base distributionQn
i=1
icould be written as QU;W;Qn
i=1
i;qThis is quite
dierent from the rst scenario!
To give a concrete example, imagine two consec-
utive games that work as follows. In each game,
the agent must pick an integer from the set f0;1;2g,
and then gain that much utility. Thus, we dene
Ai=f0;1;2g;Wi(a) =a;Ui(o) =o. Each base dis-
tribution
iis uniform. Then, a 1 =3-quantilizer quan-
tilizing both games independently will choose from the
top third of actions in each game separately, and will
always yield the action sequence (2 ;2). By contrast,
a quantilizer quantilizing for the joint game will uni-
formly sample from the top third of action sequences
f(2;2);(1;2);(2;1)g, as illustrated in gure 2.
What could explain this dierence? If we dene a
cost function
c(a1;a2) =1 if (a1;a2) = (2;2)
0 otherwise
which states that the (2 ;2) action has terrible unintended
consequences, then independent 1 =3-quantilization will
accrue a cost 9 times that of sampling from
. This
cost blowup increases exponentially with n. On the
other hand, 1 =3-quantilization in the joint game will
accrue a cost only 3 times that of sampling from
. If
we believe that the cost function cmay not sum costs
independently between games, then quantilizing each
game independently could be very costly.
Figure 2: A representation of the 9 available action pairs
in the repeated game. Independent 1 =3-quantilization
will always select the striped action pair, while 1 =3-
quantilization on the joint game will select uniformly
from one of the three shaded action pairs.
Cost Independence Assumption
Suppose we added the assumption that c(a1;:::;an) =Pn
i=1ci(ai) for some ci:Ai!R0. Then a repeated
q-quantilizer would get total expected cost
EAiQUi;Wi;
i;q[c(A1;:::;An)] =nX
i=1EAiQUi;Wi;
i;q[ci(Ai)]
1
qnX
i=1EAi
i[ci(Ai)]
=1
qEAi
i[c(A1;:::;An)]
so repeated quantilization would no longer have a cost
blowup exponential in n.
However, adding this assumption may be dangerous.
Imagine a scenario in which almost all of the cost of
a sequence of actions comes from correlation between
actions in dierent games. For example, imagine that
each game consists of appending a byte to a computer
program, and the program is run when it consists of a
million bytes. Then some sequence of actions that each
individually have probability 1 in 256 (namely, append-
ing the bytes of a program implementing a computer
program that would be very destructive if run) could be
dangerous. Each individual action did not have a much
higher cost than adding a random byte to the program,
but taken all together, the sequence of actions still had
a very high cost. In this case and any analogous case,
assuming that the cost function is additive could be
quite dangerous.
In fact, attempting to design a \superintelligent" (in
the sense of Bostrom [10]) expected utility quantilizer
could lead to similar issues. Imagine asking a superintel-
ligent quantilizer for repeated actions that maximize the
long-term number of paperclips produced by a certain
factory. In this case, similar to the above example, each
action might play some small part in a complex plan to
turn large swaths of the world into paperclips.
However, this seems much less likely to be a problem
if the dierent goals that we have the system quantilize
6
with respect to are independent. For example, imagine
that we instead ask the system to repeatedly quantilize
a slightly dierent utility function Uiwhich only cares
about events which happen in hour i, with the quantilizer
run at the beginning of hour i. Intuitively, it seems that
this setup would be safer. However, no safety guarantees
have been given; nding a reasonable assumption about
the cost functions under which this form of independent
quantilization would be safe (if such an assumption
exists) remains an open problem.
4 Example Applications
Quantilizers cannot be used to eectively achieve all pos-
sible goals. If the system has to achieve an outcome that
it is extremely dicult to achieve by simply sampling
,
then the quantilizer, too, will have a hard time achieving
that outcome. For example, if
is a generative model of
human behavior, and the system faces a problem that a
human would be very unlikely to solve even one time in
a billion, then the quantilizer would also be hard pressed
to solve the same problem.
However, there are a number of tasks where an ex-
pected utility quantilizer would plausibly be able to
perform quite well. In this section, we will list a number
of examples of tasks that are plausibly solvable with
limited expected utility quantilizers.
Note that, in practice, these example applications
involve repeated quantilization. As explained in sec-
tion 3.3, repeated quantilization is not currently known
to be safe, but we will provisionally assume that it is
possible to develop some set of independent (possibly
time-indexed) objectives such that repeatedly quantiliz-
ing those objectives is in fact safe.
4.1 Synthesizing Highly-Rated Images
Imagine that we have a database containing images and
human ratings of these images, and that we would like
to synthesize new images that are expected to receive
high ratings. To do this, we could separately create a
generative model over images and a regression model
predicting the expected rating for an image. If the
regression model is a deep neural network, then as in
Nguyen, Yosinski, and Clune [8], the image that gets
the highest rating according to the model is unlikely to
resemble highly-rated images. Instead, we could use the
generative model over images as the base distribution
and then quantilize to generate a plausible image with
a high predicted rating.
4.2 Factory Management
Suppose we want a program to control a factory to create
a number of objects satisfying a certain specication,
using a xed amount of time and resources. Policies will
control parts of the factory, eventually producing many
objects.Maximizing the extent to which the resulting objects
adhere to the specications may result in unintended
solutions. Perhaps the resulting object will be useless
despite satisfying the specications, due to Goodhart's
law: \When a measure becomes a target, it ceases to be a
good measure." [20]. As an example, imagine a factory
that produces electronic toys, which has automated
devices that samples toys and automatically score them
to see how well they meet specications. Then, we can
imagine a expected utility maximizer guring out exactly
which toys will be sampled, and makes only those ones
to spec (the other ones being the cheapest possible piece
of plastic that triggers the toy-counter). Or perhaps the
automated devices dock points for toys that come o
the assembly line hot (as these toys generally become
deformed), and the expected utility maximizer nds a
way to supercool the toys and thus gain an extremely
high score, despite the fact that the supercooled toys
later shatter and are useless.
By contrast, if we build the AI system to quantilize
the extent to which the devices pass the specications,
then the resulting policy will be much more likely to be
of the form we expected: the task will almost certainly
be accomplished in a manner that was not incredibly
unlikely according to
.
4.3 Safely Transferring Information to Hu-
mans
A number of proposals for designing extremely capable
AI systems that are nevertheless safe to use for one
purpose or another include a step where the AI system
is expected to explain its plans to humans instead of
executing them [12, 10]. However, it is quite a dicult
task to dene a goal function that adequately describes
\provide an explanation that causes the human to under-
stand the plan." Yet, absent such a goal function, the
system's \explanation" is likely to be inadequate. If the
system's internal representation of the plan is foreign
to the operators, then they may not be able to inter-
pret most simple data dumps describing the system's
plan. Furthermore, if the system is more capable and
knowledgeable than the humans, then the humans may
also need a large number of relevant facts, assumptions,
and sub-strategies explained to them before they are
capable of evaluating the plan. But if the system has
to perform a complex optimization task to explain the
plan to the humans, then we face the same problem of
safe optimization again.
If the system is designed to maximize the extent to
which the human operators understand its plan, then
perverse instantiations could occur, and the result could
be quite dangerous. An expected utility maximizer
might exploit edge cases in the denition of \understand"
to fool or otherwise manipulate the humans in a fashion
that fullls the technically stated requirements of \make
the humans understand the plan," without generating
anything that we would actually term \understanding."
It seems very dicult to dene what it means for an
7
explanation to cause the human to understand such that
we would be comfortable instructing a powerful system
to maximize along this metric.
Quantilization provides an alternative: we could
build an expected utility quantilizers that selects an
explanation that is good at causing the human to \un-
derstand" the system's plans in the same way that they
would understand a plan as explained by another hu-
man, without as much risk of perverse instantiation. For
example, if the base distribution
is a generative model
of explanations created by humans, then moderately
strong quantilization might result in an explanation of
a plan suitable for human consumption.
4.4 Stock Trading
As explained in section 2.2, a quantilizer can be designed
to make trades on the stock market that are likely to
make money, but which have few other side eects be-
side the side eect that now the operators have much
more money. Specically, the important thing is that
expected utility quantilization gives us the ability to
design systems that would almost certainly not crash
the stock market (any more than sampling randomly
from
is likely to crash the stock market, which, if
is a generative model of a smart human trader, means
that the system is not much more likely to crash the
market than any given smart human trader).
This holds true even if the system is powerful enough
to nd a strategy for gaining incredible amounts of
money by causing the stock market to crash. An ex-
pected utility maximizer maximizing the amount of
money it receives in the stock market would crash the
market the moment it found some way to generate huge
amounts of prot from doing so, but an expected utility
quantilizer would not be much more likely to crash the
market than sampling randomly from
.
5 Conclusion
Directing a powerful AI system to maximize some utility
functionUcould result in large unintended consequences,
ifUdoes not in fact capture all the intricacies of hu-
man values. Many of these dangers can be mitigated
if, instead of designing expected utility maximizers, we
consider other agent architectures. Expected utility
quantilization is one alternative to expected utility max-
imization that allows some safety guarantees, given that
we have some base distribution
where sampling actions
from
is known to be relatively safe.
Expected utility quantilization is not a silver bullet.
Constructing a safe expected utility quantilizer requires
some method for generating a safe base distribution

, and there are a number of contexts where expected
utility quantilization is not guaranteed to yield good
results. For example, if the costs of sampling from

could in fact be high, then quantilization may not
lead to good outcomes. Furthermore, if a quantilizeris going to be used to complete the same task many
times sequentially, then running the quantilizer may be
signicantly more dangerous.
Yet despite these shortcomings, the preliminary re-
sults from investigating expected utility quantilization
are promising. Gaining a better understanding of the
benets and drawbacks of expected utility quantiliza-
tion could yield many more insights into how to create
powerful systems that are \domestic" in the sense of
Bostrom [10, chap. 9], who denes a domestic agent
as an agent with limited \scope in its activities and
ambitions." There are many open issues that make it
dicult to assess how safe it would actually be to run an
extremely capable expected utility quantilizer in an un-
restricted domain, but nevertheless, this area of research
does prove fruitful, and we are hopeful that further study
of expected utility quantilization may yield insights into
how to design autonomous AI systems that avoid the
problem of perverse instantiation.
6 Acknowledgements
The author would like to thank Nate Soares for help
preparing this paper, Benya Fallenstein and Andreas
Stuhlm uller for discussion and comments, and Stuart
Armstrong for helping to develop initial versions of this
idea. A grant from the Future of Life Institute helped
fund this work.
References
[1] David E. Heckerman and Eric J. Horvitz. \Problem
formulation as the reduction of a decision model". In:
in Proceedings of the Sixth Conference on Uncertainty
in Articial Intelligence . Elsevier Science Publisher,
1990, pp. 82{89.
[2] Fahiem Bacchus and Adam Grove. \Graphical Mod-
els for Preference and Utility". In: Proceedings of the
Eleventh Conference on Uncertainty in Articial In-
telligence . UAI'95. Montr eal, QC, Canada: Morgan
Kaufmann Publishers Inc., 1995, pp. 3{10. isbn: 1-
55860-385-9. url:http://dl.acm.org/citation.
cfm?id=2074158.2074159 .
[3] Urszula Chajewska, Daphne Koller, and Ronald Parr.
\Making Rational Decisions using Adaptive Utility Elic-
itation". In: In Proceedings of the Seventeenth National
Conference on Articial Intelligence . 2000, pp. 363{
369.
[4] John von Neumann and Oskar Morgenstern. Theory
of Games and Economic Behavior . 1st ed. Princeton,
NJ: Princeton University Press, 1944.
[5] Stuart J. Russell and Peter Norvig. Articial Intel-
ligence. A Modern Approach . 3rd ed. Upper Saddle
River, NJ: Prentice-Hall, 2010.
[6] Paul R. Rosenbaum and Donald B. Rubin. \The Cen-
tral Role of the Propensity Score in Observational
Studies for Causal Eects". English. In: Biometrika
70.1 (1983), pp. 41{55. issn: 00063444. url:http:
//www.jstor.org/stable/2335942 .
8
[7] Nate Soares and Benja Fallenstein. \Toward Ideal-
ized Decision Theory". In: (2015). arXiv: 1507.01986
[cs.AI] .
[8] Anh Nguyen, Jason Yosinski, and Je Clune. \Deep
neural networks are easily fooled: High condence pre-
dictions for unrecognizable images". In: arXiv preprint
arXiv:1412.1897 (2014).
[9] Adrian Thompson. \Articial Evolution in the Physi-
cal World". In: Evolutionary Robotics: From Intelligent
Robots to Articial Life (ER'97) . Ed. by T Gomi. AAI
Books, 1997, pp. 101{125.
[10] Nick Bostrom. Superintelligence. Paths, Dangers,
Strategies . New York: Oxford University Press, 2014.
[11] Nate Soares. The Value Learning Problem . Tech. rep.
2015{4. Berkeley, CA: Machine Intelligence Research
Institute, 2015. url:https://intelligence.org/
files/ValueLearningProblem.pdf .
[12] Stuart Armstrong, Anders Sandberg, and Nick
Bostrom. \Thinking Inside the Box. Controlling and
Using an Oracle AI". In: Minds and Machines 22.4
(2012), pp. 299{324. doi:10.1007/s11023-012-9282-
2.
[13] Herbert A. Simon. \Rational Choice and the Structure
of the Environment". In: Psychological Review 63.2
(1956), pp. 129{138.
[14] Benja Fallenstein and Nate Soares. \Problems of Self-
Reference in Self-Improving Space-Time Embedded
Intelligence". In: Articial General Intelligence. 7th
International Conference, AGI 2014, Quebec City, QC,
Canada, August 1{4, 2014. Proceedings . Ed. by Ben
Goertzel, Laurent Orseau, and Javier Snaider. Lec-
ture Notes in Articial Intelligence 8598. New York:
Springer, 2014, pp. 21{32. doi:10.1007/978-3-319-
09274-4\_3 .
[15] Pieter Abbeel and Andrew Y. Ng. \Apprenticeship
Learning via Inverse Reinforcement Learning". In:
Proceedings of the Twenty-rst International Confer-
ence on Machine Learning . ICML '04. Ban, Alberta,
Canada: ACM, 2004, pp. 1{. isbn: 1-58113-838-5. doi:
10.1145/1015330.1015430 .url:http://doi.acm.
org/10.1145/1015330.1015430 .
[16] Adith Swaminathan and Thorsten Joachims. \Coun-
terfactual Risk Minimization: Learning from Logged
Bandit Feedback". In: CoRR abs/1502.02362 (2015).
url:http://arxiv.org/abs/1502.02362 .
[17] Andrew Y. Ng and Stuart J. Russell. \Algorithms
for Inverse Reinforcement Learning". In: Proceedings
of the Seventeenth International Conference on Ma-
chine Learning (ICML-'00) . Ed. by Pat Langley. San
Francisco: Morgan Kaufmann, 2000, pp. 663{670.
[18] Chris L Baker, Joshua B Tenenbaum, and Rebecca R
Saxe. \Goal inference as inverse planning". In: Pro-
ceedings of the 29th annual meeting of the cognitive
science society . 2007.
[19] Paul Christiano. Avoiding Simulation Warfare with
Bounded Complexity Measures . 2011. url:https://o
rdinaryideas.wordpress.com/2011/12/21/avoidin
g-simulation-warfare-with-bounded-complexity-
measures/ .[20] Charles Goodhart. Problems of monetary management:
the UK experience . 1975.
9