FHI Oxford Technical Report #2018-2
Predicting Human Deliberative Judgments
with Machine Learning
Owain Evans, Andreas Stuhlmüllery, Chris Cundy, Ryan Careyy, Zachary
Kenton, Thomas McGrath, Andrew Schreibery
July 13, 2018
Abstract
Machine Learning (ML) has been successful in automating a range of
cognitive tasks that humans solve eﬀortlessly andquickly. Yet many real-
world tasks are diﬃcult andslow: people solve them by an extended
process that involves analytical reasoning, gathering external information,
anddiscussingwithcollaborators. Examplesincludemedicaladvice, judg-
ing a criminal trial, and providing personalized recommendations for rich
content such as books or academic papers.
There is great demand for automating tasks that require deliberative
judgment. Current ML approaches can be unreliable: this is partly be-
cause such tasks are intrinsically diﬃcult (even AI-complete) and partly
because assembling datasets of deliberative judgments is expensive (each
label might take hours of human work). We consider addressing this data
problem by collecting fastjudgments and using them to help predict de-
liberative ( slow) judgments. Instead of having a human spend hours on a
task, we might instead collect their judgment after 30 seconds or 10 min-
utes. These fast judgments are combined with a smaller quantity of slow
judgments to provide training data. The resulting prediction problem is
related to semi-supervised learning and collaborative ﬁltering.
We designed two tasks for the purpose of testing ML algorithms on
predicting human deliberative judgments. One task involves Fermi esti-
mation (back-of-the-envelope estimation) and the other involves judging
the veracity of political statements. We collected a dataset of 25,000 judg-
ments from more than 800 people. We deﬁne an ML prediction task for
predicting deliberative judgments given a training set that also contains
fast judgments. We tested a variety of baseline algorithms on this task.
Unfortunately our dataset has serious limitations. Additional work
is required to create a good testbed for predicting human deliberative
judgments. This technical report explains the motivation for our project
(which might be built on in future work) and explains how further work
can avoid our mistakes. Our dataset and code is available at https:
//github.com/oughtinc/psj .
University of Oxford
yOught Inc.
1
1 Introduction
1.1 Fast and slow judgments
Machine Learning has been successful in automating mental tasks that are quick
and eﬀortless for humans. These include visual object recognition, speech recog-
nition and production, and basic natural language prediction and comprehen-
sion [1, 2, 3, 4]. Andrew Ng states the following heuristic [5]:
If a typical person can do a mental task with less than one second
of thought, we can probably automate it using AI either now or in
the near future.1
In this technical report we refer to judgments that are quick (roughly 30 seconds
or less) and easy for most humans as fastjudgments. Fast judgments contrast
withslowjudgments, which may involve lengthy processes of deliberate rea-
soning, research, experimentation, and discussion with other people.2Many
important real-world tasks depend on slow judgments:
Predict the verdict of jury members in a criminal trial.
Predict which engineers will be hired by a company with an extensive
interview process.
Predict whether experts judge a news story to be fake or intentionally
misleading.
Predict a doctor’s advice to an unwell patient after a thorough medical
exam.
Predict how a researcher will rate a new academic paper after reading it
carefully.
Predict how useful a particular video lecture will be for someone writing
a thesis on recent Chinese history.
There is great demand for Machine Learning (ML) and other AI techniques
for predicting human slow judgments like these, especially in hiring workers,
detection of fake or malicious content, medical diagnosis, and recommenda-
tion [8, 9, 10, 11]. However ML approaches to predicting these slow judgments
are often unreliable [12, 13, 14]: even if they do reasonably well on a majority of
instances, they may have large errors on more demanding cases (e.g. on inputs
that would be tricky for humans or on inputs chosen by humans to try to fool
the algorithm).
One source of the unreliability for ML algorithms is optimizing for a subtly
wrong objective. Suppose a student gets video lecture recommendations from
1Ng intends it as a heuristic rather than a rigorous scientiﬁc conclusion.
2ThedistinctionissimilartothatbetweenSystem1(fast)andSystem2(slow)cognition[6,
7]. However, in this work we distinguish judgments by how long they take and whether they
make use of external information and not by the underlying cognitive process. For instance,
fast judgments can depend on quick application of analytical reasoning.
2
YouTube.3These recommendations may be optimized based on video popular-
ity (“Do users click on the video?”) and engagement (“Do users Like or share the
video?”). These metrics are mostly based on fastjudgments. Yet the student
would prefer recommendations based on slowjudgments, such as the evaluation
of another student who has carefully checked the lecture for factual accuracy by
tracking down the sources. Fast and slow judgments will sometimes coincide, as
when a lecture is inaudible or oﬀ-topic. Yet a lecture may seem useful on ﬁrst
glance while turning out to be riddled with inaccuracies that careful research
would expose.4
Predicting slow judgments in the tasks above is challenging for current ML
algorithms. This is due in part to the intrinsic diﬃculty of the tasks; predict-
ing how a student evaluates a lecture is arguably AI-complete [16]. Another
diﬃculty is that collecting slow judgments is inherently expensive: if it takes
ﬁve hours of fact-checking to recognize the errors in a lecture then a dataset of
millions of such evaluations is impractically expensive. Big datasets won’t solve
AI-complete problems but will improve ML performance.
Predicting slow judgments is also related to long-term AI Safety, i.e. the
problem of creating AI systems that remain aligned with human preferences
even as their capabilities exceed those of humans [17, 18, 19, 20, 21]. Rather
than create AI that shares only human goals, a promising alternative is to create
AI that makes decisions in the way a human would at every timestep [22, 23].
This approach of imitating human decision-making is only promising if it imi-
tates human deliberate (slow) judgments [24, 25]. A system making thousands
of human-like slow judgments per second could have super-human capabilities
while remaining interpretable to humans [26, 27].
1.2 Using Fast Judgments to Predict Slow Judgments
How can we get around the challenges of predicting slow judgments? One ap-
proach is to tackle the AI-completeness head on by trying to emulate the process
of human deliberate reasoning.5A second approach (which complements the
ﬁrst)istotacklethedataproblemheadonandﬁndwaystocollecthugedatasets
of real or synthetic slow judgments [38]. This technical report explores an indi-
rect approach to the data problem. Instead of collecting a big dataset of slow
judgments, we collect a small dataset of slow judgments along with a larger
quantity of side information related to the slow judgments.
What kind of side information would help predict a person’s slow judgment?
If Alice makes a slow judgment about a question, then Alice’s fastjudgment
about the same question is relevant side information. As noted above, in pre-
dicting a student’s thorough evaluation of a video lecture, it is helpful to know
3This is just meant as an example of a recommender system and is not a comment on
the actual YouTube algorithm. A paper [15] on YouTube recommendations states that they
optimize for whether users follow a recommendation (weighted by user watch-time). This will
mostly depend on fast judgments.
4It could be argued that YouTube, Facebook and other sites are optimizing for being
entertaining and keeping users on the site and that these are well predicted by fast judgments.
Yet it’s clear that users sometimes seek content that they would rate highly after a slow
judgment (e.g. for educational purposes, for help making a business decision). So the question
remains how to build ML algorithms for this task.
5There is a large and varied literature aiming to create algorithms that perform sophisti-
cated reasoning. Here are some recent highlights: [28, 29, 30, 31, 32, 33, 34, 35, 36, 37]
3
their fast judgment (e.g. after watching the video for only 30 seconds). Likewise,
a doctor’s guess about a diagnosis after 30 seconds will sometimes predict their
careful evaluation. Another kind of side information for predicting Alice’s slow
judgment about a question are the judgments of other people about the same
question. This is the idea behind collaborative ﬁltering [39], where a person’s
rating of a song is predicted from the ratings of similar people.
While collaborative ﬁltering has been widely studied and deployed, there is
little prior work on using fast judgments as side information for slow judgments.
The motivation for using fast judgments is that they are often easily available
and their cost is much lower than slow judgments. Human cognition is like an
“anytime” iterative algorithm: whereas our slow judgments are more discerning
and reliable, our fast judgments are well-typed and provide increasingly good
approximations to slow judgments over time.6For most judgment tasks we can
collect fast judgments from humans and these fast judgments will come at a
cost orders of magnitude cheaper than slow judgments. Where fast judgments
sometimes coincide with slow judgments and include uncertainty information7,
it may be better to spend a ﬁxed budget on a mix of slow and fast judgments
than on slow judgments alone. Often fast judgments are not just cheaper to
collect but are essentially free. YouTube, Facebook, Twitter and Reddit have
vast quantities of data about which content people choose to look at, share with
others, “Like”, or make a quick verbal comment on.
Using fast judgments as side information to predict slow judgments requires
modifying standard ML algorithms. While the objective is predicting slow judg-
ments, most of the training examples (e.g. most video lectures) only come with
fast judgments. This is related to semi-supervised learning [40, 41], distant su-
pervision [42] (where the training labels are a low-quality proxy for the true
labels), learning from privileged information [43], as well as to collaborative
ﬁltering.
1.3 Contributions and caveats
This tech report describes a project applying machine learning (ML) to pre-
dicting slow judgments. We designed tasks where slow judgments (deliberate
thinking and research) are required for doing well but quick judgments often
provide informative guesses. We collected a dataset of fast and slow human
judgments for these tasks, and formulated a set of prediction problems (predict-
ingheld-outslowjudgmentsgivenaccesstovaryingquantitiesofslowjudgments
at training time). We applied ML baselines: standard collaborative ﬁltering al-
gorithms, neural collaborative ﬁltering specialized to our tasks, and a Bayesian
hierarchical model.8.
Unfortunately our dataset turned out to be problematic and is unlikely to be
a good testing ground for predicting slow judgments . This report describes parts
of our project that may be usefully carried over to future work and summarizes
what we learned from our eﬀorts to create an appropriate dataset. The prob-
lem of predicting slow judgments (and of creating a dataset for the purpose)
6In doing a long calculation we might have no idea of the answer until we solve the whole
problem. For many real-world problems our quick guesses are somewhat accurate and gradu-
ally improve with time.
7That is, the human provides both a guess and a measure of conﬁdence.
8Code is available at https://github.com/oughtinc/psj
4
Fermi comparisons
weight of a blue whale in kg <50,000
population of Moscow smaller angle
in degrees between hands of clock at
1.45<15,000,000
driving distance in miles between Lon-
don and Amsterdam <371
weight of $600 worth of quarters in
poundsarea of Canada in square
miles <328,000,000Politifact truth judgments
Rob Portman : “Since the stimulus
package was passed, Ohio has lost over
100,000 more jobs.”
Republican Party : “Charlie Crist is
embroiled in a fraud case for steering
taxpayer money to a de facto Ponzi
scheme.”
Mark Zaccaria : “James Langevin
voted to spend $3 billion for a jet en-
gine no one wants.”
Figure 1: Example questions. In Fermi, people guess whether the left-hand side is
smaller than right-hand side. In Politifact they guess whether the speaker’s statement
is true.
was harder than expected. We hope to stimulate future work that avoids the
problems we encountered. Our main contributions are the following:
1. We designed two slow judgment tasks for humans: Fermi Comparisons
involve mental reasoning and calculation, and Politifact Truth Judg-
mentsinvolve doing online research to assess the veracity of a political
statement.9We created a web app to collect both fast and slow judgments
for the task. These tasks and the app could be used in future work.
2. Werelatepredictingslowjudgmentswithsideinformationtocollaborative
ﬁltering and we implement simple baselines based on this relation.
3. We diagnose problems with our dataset. First, while slow judgments were
signiﬁcantly more accurate than fast judgments, the diﬀerence was smaller
than expected. Second, variability among subjects was hard to distinguish
from noise; so ML algorithms could not exploit similarities among users
as in collaborative ﬁltering. Third, while there is clear variation in how
users respond to diﬀerent questions, this variation is very hard for current
ML algorithms to exploit.
2 Tasks and Datasets for Slow Judgments
Our overall goal was (A) to deﬁne the ML problem of predicting slow judgments
givenatrainingsetoffastandslowjudgments, and(B)toﬁndorcreatedatasets
which can be used to test algorithms for this problem.
The domain that humans are making fast/slow judgments about should be
AI-complete or closely related to an AI-complete problem.10Solving such a
9Both tasks require humans to decide some objective matter of fact. Yet there is no
requirement that slow judgments be about objective facts: e.g. a student’s judgment about a
lecture is partly based on their own preferences and interests.
10The important real-world problems of predicting slow judgments in Section 1.1 are plau-
sibly AI-complete.
5
task will (for some problem instances) depend on patterns or structures that
current Machine Learning algorithms do not capture. So while we cannot hope
for highly accurate predictions of slow judgments, we can seek ML algorithms
that “know what they know” [44]. Such algorithms exploit patterns they are
able to learn (producing conﬁdent accurate predictions) and otherwise provide
well-calibrated uncertain predictions [45, 46].
This section describes the AI-complete tasks we designed for testing al-
gorithms for predicting slow judgments. Before that we ﬁrst review related
datasets in this area.
2.1 Existing Datasets
Many ML datasets record human slow judgments for AI-complete tasks. These
include datasets of movie reviews [47], reviewer scores for academic papers [48],
and verdicts for legal cases [49]. There are also datasets where the ground-truth
labels are the product of extensive cognitive work by humans (e.g. theorem
proving [38], political fact-checking [14]) and these could potentially be used
to study human slow judgments. However, these datasets do not contain fast
judgments. In particular, for each task instance in the dataset, there’s a slow
judgment by a particular individual but there’s no corresponding fast judgment.
Moreover the datasets do not explicitly record information about the time and
resources than went into the slow judgment11. For example, the reviewers of
academic papers do not divulge whether they spent ten minutes or two hours
reviewing a paper.
Due to the lack of existing datasets, we created a new dataset recording
human slow and fast judgments for AI-complete tasks. We designed two tasks
for this purpose, which may be useful for future work in this area (even if our
dataset is not).
2.2 Two Judgment Tasks: Fermi and Politifact
Wecreatedtwotasksforhumansthatrequiredeliberate,multi-stepthinkingand
rely on broad world knowledge. In the Fermi Comparisons task (abbreviated
to “Fermi”) users determine which of two quantities is larger. This often involves
simple arithmetic, factual knowledge, and doing back-of-the-envelope estimates
to determine the order of magnitude of the quantity. Example questions are
show in Figure 1. Note that human subjects (who we refer to as “users”) are
not allowed to look up the quantities online as this would trivialize the task.
In thePolitifact Truth task (abbreviated to “Politifact”) users evaluate
whether a statement by a US politician or journalist is true or false (see Figure
1). They have access to metadata about the statement (see Figure 2) and are
allowed to do research online. For both tasks, users assign a probability that
the statement is true (with “0” being deﬁnitely false and “1” being deﬁnitely
true) and they enter their probability via the interface in Figure 2. Their goal
is to minimize mean squared error between their probabilistic judgment and the
ground-truth answer.
11A paper [50] on adversarial examples for humans does record human judgments under
diﬀerent time constraints. But this domain is not AI-complete and the slower judgments are
still pretty fast.
6
Figure 2: Question-answeringUIforFermiComparisons(above)andPolitifactTruth
(below) tasks. Users are not allowed to do online research for Fermi but are for
Politifact.
Questions and ground-truth answers for Fermi were constructed and computed
bytheauthors. Politicalstatementsandground-truth(i.e.thejudgmentsofpro-
fessional fact-checkers) for Politifact were downloaded from the politifact.com
API (following [14]).
The Fermi and Politifact tasks satisfy the following desirable criteria:
A generalized version of each task is AI-complete. Fermi questions depend
on mathematical reasoning about sophisticated and diverse world knowl-
edge (e.g. estimate the mass of the atmosphere). If novel Fermi questions
are presented at test time (as in a job interview for high-ﬂying undergrad-
uates), the task is AI-complete. Politifact questions can be about any
political topic (e.g. economics, global warming, international relations).
7
Answering them requires nuanced language understanding and doing so-
phisticated research on the web.
Fast judgments are informative about slow judgments. Givenaquestion(as
in Figure 2), fast judgments of the probability (e.g. less than 30 seconds)
will be informative about a user’s slow judgment (e.g. 5 or 10 minutes).
However some questions are too diﬃcult to solve with a fast judgment.
It is possible to make progress through analytical thinking or gathering
evidence. For Fermi, breaking down the problem into pieces and coming
up with estimates for each of the pieces is a useful cognitive strategy. For
Politifact (but not Fermi), participants are allowed to use a search engine
and to read web pages to check political facts.
The ground-truth is available for both Fermi and Politifact questions. The
problem of predicting slow judgments we discussed in the Introduction
is not limited to human judgments about objective facts (as the list of
examples in Section 1.1 makes clear). However, if the ground-truth is
available then this makes certain experiments possible.12
2.3 Data collection
Human participants (who we refer to as “users”) were recruited online13and
answered questions using the UI in Figure 2. Users see a question, a form for
entering their probability judgment, a progress bar that counts down the time
remaining, and (for Politifact only) a search engine tool.
For each question, users provides fast, medium and slow probability judg-
ments. In Fermi the user is presented with a question and has 15 seconds to
make the fastjudgment. Once the 15 seconds have elapsed, they get an addi-
tional 60 seconds in which they are free to change their answer as they see ﬁt
(themedium judgment). Finally they get another 180 seconds to make the slow
judgment. The setup for Politifact is identical but the time periods are 30, 90
and 360 seconds. (Users are free to use less than the maximal amount of time.
So for particularly easy or diﬃcult questions, the “slow” judgment may actually
only take 30 seconds or less.)
12In Politifact, the ground-truth is just an especially slow judgment of an expert in political
fact-checking and could be modeled as such. We did not try this in our experiments.
13We used Amazon’s Mechanical Turk (MT) for a pilot study. For the main experiment
we used volunteers recruited via social media with an interest in improving their probabilistic
judgments. We found it hard to design an incentive scheme for MT such that the users would
make a good-faith eﬀort to do well (and not e.g. declaring 50% on each judgment) while not
cheating by looking up Fermi answers.
8
0.0 0.2 0.4 0.6 0.8 1.0
Fast Judgment0.00.20.40.60.81.0Slow JudgmentFermi, Fast vs. SlowFigure 3: Correlation between fast and slow judgments for a given question-user pair
for Fermi (above) and Politifact (below). Blue markers are proportional in size to
number of instances and orange line is a linear regression through the data. Fast user
judgments are often 50% but then become more conﬁdent in either direction given
more time to think.
0.0 0.2 0.4 0.6 0.8 1.0
Fast Judgment0.00.20.40.60.81.0Slow JudgmentPolitifact, Fast vs. Slow
The Fermi data we collected consisted of 18,577 judgments, with a third
of these being slow judgments. There were 619 distinct users and 2,379 Fermi
estimation questions (with variable numbers of judgment per question). Each
user answered 12.9 distinct questions on average (median 5). The Politifact
dataset was smaller, containing 7,974 judgments, from 594 users, covering 1,769
9
statements. Each user answered 6.4 distinct questions on average (median 5).
2.4 Descriptive Statistics
Our goal was to use the dataset to train ML algorithms to predict slow judg-
ments and such a dataset should have the following features:
1.Large, clean, varied dataset : as usual a large dataset with minimal noise
is desirable. In particular there should be many judgments per user and
many distinct questions. Possible sources of noise should be minimized
(e.g. users are noisier when learning the task and if they pay less attention
due to tedious or overly diﬃcult questions).
2.Fast-Slow Correlation : fast judgments are informative about slow judg-
ments but not too much. For some users and some questions, fast judg-
ments may be similar to slow judgments. In other cases, the fast judgment
will be uncertain or wrong.
3.User Variation : individual users vary in their overall performance and
their performance in diﬀerent kinds of question. We want to predict the
slow judgments of an individual (not the ground-truth). In some tasks
listed in Section 1.1 users vary because of diﬀerent preferences. In Fermi
and Politifact users might vary in their areas of knowledge (e.g. science
vs. sport questions in Fermi).
4.Question Variation : individual questions vary in terms of how users an-
swer them. For example, some questions are hard (producing random or
systematically incorrect answers) and some are easy. This allows algo-
rithms to predict user answers better based on features of the question.
Did our data have the features above? As noted in Section 2.3 the dataset
was relatively small (especially Politifact) and many users only answered a small
number of questions.
Correlations between fast and slow judgments are shown in Figure 3. They
show that judgments started out uncertain but became more conﬁdent (in ei-
ther direction) given time, and that conﬁdent fast judgments were less likely to
change. Figure 4 shows that slow judgments were more accurate than fast on
average. Nevertheless, the diﬀerences between slow, medium and fast judgments
are not as large as we had hoped.
Did users vary in their accuracy? We expected users to vary in overall
accuracy and accuracy for diﬀerent kinds of question (e.g. some users do better
on science than sports questions). However the level of variation among users
was not high enough to rule out possibility that the best two-thirds of users did
not vary (see Figure 5). This is a problem with the design of our experiment
and we discuss it further in Section 4.
10
0.00 0.25 0.50 0.75
Mean squared error020406080100Number of UsersMean squared error of user's answers, Fermi
Fast
SlowFigure 4: Variation in User Performance in Fermi for fast and slow judgments.
Histogram shows number of users who obtained a particular level of performance
(measured by MSE). Note that 0:25is the MSE achieved by a user who always says
50%. While users have widely varying performance, much of this is due to noise and we
can’t rule out the possibility that most users do not vary in actual skill (see Figure 5).
Figure 5: Mean performance on Fermi for diﬀerent user quantiles. Users were divided
into quantiles based on performance on a random half of the data and the ﬁgure shows
their performance on the other half. Error bars show standard error from 5 random
divisions into quantiles. The graph suggests we can’t rule out the null hypothesis that
the best two-thirds of users don’t vary in their accuracy. (We removed users with less
thank=6 judgments; the graphs were similar when kwas set higher.)
Questions in both Fermi and Politifact did vary signiﬁcantly in diﬃculty for
human users (see Figure 6). However, there are less than 2500 questions for
each task. This makes it hard for algorithms to generalize from the question
text or meta-data without substantial pre-training.
11
Figure 6: Variation in question diﬃculty for human users. Histograms show number
of questions with a given MSE across users. Many questions had an MSE near zero
(most users got them conﬁdently right), while others were diﬃcult (MSE near 0.25)
or counter-intuitive (MSE greater than 0.25).
3 Predicting Slow Judgments with ML
This section presents a problem deﬁnition for predicting slow judgments using
ML and results for baseline algorithms on our dataset. The ML task and algo-
rithms are independent of the dataset and may be useful for future work in this
area.
3.1 Task: Predicting Slow Judgments with ML
3.1.1 Task Deﬁnition
For both Fermi and Politifact the data consists of user probability judgments
about questions. Each probability judgment h(q; u; t )2[0;1]depends on the
question text and meta-data features q, the user id u, and the time t2f0;1;2g
(where the indices correspond to fast, medium, and slow judgments respec-
tively).
We let ^h(q; u; t )be an algorithm’s prediction of the user judgment h(q; u; t ).
The task is to predict slow judgments h(q; u;2)from the test set. While only
slow judgments appear in the test set, the training set consists of fast, medium
and slow judgments.14The loss function is mean-squared error over all slow
judgments in the test set. Let T=f(q; u)gbe the set of question-user pairs
that appear in the test set. Then the objective is to minimize the mean squared
error over slow judgments in the test set:
1
jTjX
(q;u)2Th
h(q; u;2)^h(q; u;2)i2
(1)
Note that this problem setup is analogous to content-based collaborative
ﬁltering, where the task is to predict a user’s held-out rating for an item (e.g. a
movie), given the user id and a feature vector for the item [51].
14The test set is randomly sampled from the entire dataset and so the same users and
questions can appear both in train and test.
12
3.1.2 Train on Fast Judgments, Predict Slow
Slow judgments are much more expensive to collect than fast judgments. Yet for
questions that are either very easy or very diﬃcult for a user, the fast and slow
judgments will be very similar. So if a model can learn when fast judgments
are a good proxy for slow judgments, it can predict slow judgments reasonably
well from a much cheaper dataset (as discussed in Section 1.1).
In our dataset we have the user’s slow judgment for a question whenever we
have the fast judgment. Yet we can simulate the strategy of mostly collecting
fast judgments by masking many of the slow judgments from our training set.15
Our results are computed for three diﬀerent levels of masking. “Unmasked”
means that all slow judgments in the training set are included. “Heavy” means
90% of question-user pairs have the medium and slow removed and 7% have
just the slow removed. “Light” means we removed slow and medium judgments
for 60% of question-user pairs and removed just slow for 30%.16
3.2 Algorithms and Results
We ran the following baseline algorithms on our datasets:
1.Collaborative Filtering (KNN and SVD) : The prediction task is closely
related to collaborative ﬁltering (CF). The main diﬀerence is that instead
of predicting a user’s judgment about an item, we predict a user’s judg-
ment about a question at a given time (where we expect users to generally
converge to zero or one given more time). To reduce our task to CF, we
ﬂatten the data by treating each user-time pair (u; t)as a distinct user.
We applied the standard CF algorithms K-nearest-neighbors (KNN CF)
and singular value decomposition (SVD CF), using the implementation in
the Surprise library [52].
2.Neural Collaborative Filtering : We adapt the Neural CF algorithm [53]
to our task. A neural net is used to map the latent question and user
embeddings to a sequence of judgments for each time. Linear Neural CF
forces the judgments to change linearly over time.
3.Hierarchical Bayesian Linear Regression : To predict the user judgment
for a question we pool all user judgments for the same question (ignoring
user identity) and regress using a Bayesian linear regression model. This
model exploits the temporal structure of judgments but it discards the
question features and user identity.
4.Clairvoyant Model : Since user slow judgments are correlated with the
ground-truth, algorithms will do better on new questions to the extent
they can predict the ground-truth. Predicting ground-truth without side-
information is diﬃculty for Fermi and Politifact. However, we can investi-
gate how well a model would do if it had access to the ground-truth. The
15This is like the standard practice in semi-supervised learning, where researchers remove all
but 5% of the labels from the training set to simulate a situation where most data is unlabeled
16In order to achieve clear-cut separation of the training data and the held-out test set
we also implemented a masking procedure before doing the additional masking for Light and
Heavy. For each judgment in the test set, we removed the medium judgment made by the
same user about the same question from the training set. We also stochastically remove the
corresponding fast judgment with 80% probability.
13
“Clairvoyant” model simply predicts that each user will respond with the
ground-truth answer with full conﬁdence. The “Clairvoyant Mean” model
predicts the base-rate probability for a user given the ground-truth of a
question.
3.3 Results
Table 1 shows results for Fermi and Politifact with diﬀerent levels of masking of
slow judgments. Hierarchical Linear Regression and SVD perform best. Since
Hierarchical Linear regression ignores both question features and user identity,
this suggests it is diﬃcult for algorithms to improve predictions by modeling
questions and users. This is additional evidence that our dataset is problematic.
We had expected the Neural CF algorithms would do well, as they learn
latent representations for both questions and users and (unlike SVD and KNN)
they do not discard the temporal structure in the data. However, their poor
performance is partly explained by the diﬃculty (discussed in Section 2.4) of
distinguishing user variation from noise.17
The performance of Clairvoyant Mean suggests that non-clairvoyant algo-
rithms could be improved with a strong language and reasoning model that
accurately predicts the ground-truth.18
Table 1: Meansquaredtesterrorforvariousalgorithmsanddiﬀerentlevelsofmasking
slow judgments at training time. Note that Clairvoyant models had access to ground-
truth and other models did not.
Politifact Fermi
Unmasked Light Heavy Unmasked Light Heavy
KNN CF 0.127 0.130 0.133 0.115 0.125 0.134
SVD CF 0.124 0.126 0.127 0.102 0.113 0.112
Linear Neural CF 0.131 0.135 0.135 0.137 0.141 0.141
Neural CF 0.130 0.131 0.129 0.136 0.136 0.138
Hierarchical Lin. Reg. 0.123 0.126 0.127 0.098 0.107 0.114
Clairvoyant 0.242 0.242 0.242 0.216 0.216 0.216
Clairvoyant Mean 0.112 0.112 0.112 0.111 0.111 0.111
Always Guess 50% 0.137 0.137 0.137 0.138 0.138 0.138
4 Discussion: Mitigating Dataset Problems
Some problems with our dataset were described in Section 2.4. How could these
problems be mitigated in future work?
Problem: Fast and slow judgments were too similar
There weren’t big diﬀerences between fast and slow judgments. This can be
addressed by choosing a task that requires more thinking and research than
17Like Hierarchical Linear Regression, the Linear Neural CF assumes that judgments evolve
linearly over time. However, Linear Neural CF does not build in which data-points to regress
on and wasn’t able to learn this.
18We experimented with various language models (results not shown). We found it diﬃcult
to train or ﬁne-tune these models on our small dataset without overﬁtting.
14
the Fermi and Politifact tasks. The task should also be incremental , such that
a small amount of additional work yields a slightly better answer.19If addi-
tional work is unlikely to help, users will be averse to doing it (without big
compensation).
Problem: Variation between users was hard to distinguish from noise
Lack of discernible variation can be addressed by collecting substantially more
data per user and by trying to reduce noise (e.g. having practice questions,
having less ambiguous questions, having tasks for which users are consistently
fully engaged).
A related issue is that human responses were low in information content.
Users assigned probabilities (from 20 discrete options) to binary ground-truth
outcomes. Many questions were easy and were answered with 95-100% conﬁ-
dence, while others were very diﬃcult and answered with 50-55% conﬁdence.
Furthermore, users were rarely anti-correlated with the ground truth, so slow
judgments for true statements were generally answered somewhere in the range
60-100%, which is a small region of the response space in terms of mean squared
error. This problem of low information content could be addressed by having
a task where user responses are richer in information: e.g. scalar valued, cate-
gorical (with many categories), or a text response such as a movie review. The
cost of this change is that the ML modeling task becomes harder.
Another way to address this issue is to have a task in which the goal for users
is something other than guessing the ground-truth. There are many situations
in which people make slow judgments for questions that are not about objective
matters of fact. For example, when someone reviews a non-ﬁction book for
their blog, they consider both objective qualities of the book (“Does it get the
facts right?”) and also subjective qualities (“Did I personally learn things from
it? Will it help me make decisions?”). As noted above, there is nothing about
our task deﬁnition or modeling that assumes the questions have an objective
ground-truth.20
Problem: Question features were hard for algorithms to exploit
The questions in our tasks varied substantially in diﬃculty and content. Our
models couldn’t really exploit this variation, probably because (a) there were
lessthan2500questions, and(b)predictingwhetheraquestionischallengingfor
humans is an intrinsically diﬃcult task (drawing on NLP and world knowledge).
Having models that can make use of question and meta-data features is an
important goal for our research. If models can only predict slow judgments
based on other human judgments, they will not be able to generalize to new
questions that no humans have yet answered.
The most obvious ﬁx for this problem is to collect a much larger dataset.
If the dataset contained millions of questions then language models would be
better at learning to recognize easy vs. diﬃcult questions. While collecting
19This would also allow a series of intermediate times between fast and slow.
20We collected data for a third task (not discussed elsewhere in this report), where the aim
was to evaluate Machine Learning papers. We asked researchers to judge papers subjectively
(“How useful is this paper for your research?”) rather than objectively (“Should the paper
be accepted for a conference?”). Unfortunately we did not a suﬃcient amount of data from
volunteers. But we think some kind of variant on this task would be worth doing.
15
such a big dataset would be expensive, the cost could be mitigated by mainly
collecting fast judgments.21An alternative to collecting a large dataset is to
choose some object task (instead of Fermi and Politifact) for which pre-trained
language models are useful (e.g. something related to sentiment analysis rather
than judging the ground-truth of statements).
5 Conclusion
Machine Learning will only be able to automate a range of important real-world
tasks (see Section 1.1) if algorithms get better at predicting human deliberative
judgmentsinvariedcontexts. SuchtasksarechallengingduetoAI-completeness
and the diﬃculty and cost of data. We tried to create a dataset for evaluating
ML algorithms on predicting slow judgments. The previous section discussed
problems with our dataset and potential remedies. It’s also worth considering
alternative approaches to the challenge we outline in Section 1.1. First, some
tasks may be more fruitful to model than Fermi estimation and political fact
checking. Second, deliberation can be modeled explicitly by building on recent
deeplearningapproaches[28,29,30,31,32,33,34,35,36,37]. Tomoreprecisely
capture how humans deliberate, we could also record the individual steps people
take during deliberating (e.g. by requiring users to make their reasoning explicit
or by recording their use of web search). Finally, we acknowledge that while
predicting slow judgments is an important task for the future of AI, it may be
diﬃcult to make progress on today.
21It remains to be seen how well this can work if the goal is to predict slow judgments.
16
6 Acknowledgements
This work was supported by Future of Life Institute grant 2015-144846 (OE,
AS). Important early work was done by Neal Jean and Girish Sastry. We thank
Paul Christiano, David Krueger, and Tim Rocktäschel for valuable conversa-
tions. We are extremely grateful to everyone who volunteered and contributed
data to our project via the site: thinkagain.ought.org .
References
[1] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. arXiv
preprint arXiv:1709.01507 , 2017.
[2] Wayne Xiong, Jasha Droppo, Xuedong Huang, Frank Seide, Mike Seltzer,
Andreas Stolcke, Dong Yu, and Geoﬀrey Zweig. The microsoft 2016 conver-
sationalspeechrecognitionsystem. In Acoustics, Speech and Signal Process-
ing (ICASSP), 2017 IEEE International Conference on , pages 5255–5259.
IEEE, 2017.
[3] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep
Jaitly, ZonghengYang, ZhifengChen, YuZhang, YuxuanWang, RJSkerry-
Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectro-
gram predictions. arXiv preprint arXiv:1712.05884 , 2017.
[4] Jeremy Howard and Sebastian Ruder. Fine-tuned language models for text
classiﬁcation. arXiv preprint arXiv:1801.06146 , 2018.
[5] Andrew Ng. Nuts and bolts of building ai applications using deep learning.
NIPS, 2016.
[6] Daniel Kahneman and Patrick Egan. Thinking, fast and slow , volume 1.
Farrar, Straus and Giroux New York, 2011.
[7] Gerd Gigerenzer, Peter M Todd, the ABC Research Group, et al. Simple
heuristics that make us smart . Oxford University Press, 1999.
[8] Three ways machine learning is improving the hiring process.
https://www.forbes.com/sites/forbestechcouncil/2018/03/26/
three-ways-machine-learning-is-improving-the-hiring-process/
#4d9518c290e8 . Accessed: 2018-06-18.
[9] Fake news challenge. http://www.fakenewschallenge.org/ . Accessed:
2018-06-18.
[10] Arxiv sanity preserver. http://arxiv-sanity.com . Accessed: 2018-06-18.
[11] Wikimedia research: Detox. https://meta.wikimedia.org/wiki/
Research:Detox . Accessed: 2018-06-18.
[12] It’s easy to slip toxic language past alphabet’s toxic-comment
detector. https://www.technologyreview.com/s/603735/
its-easy-to-slip-toxic-language-past-alphabets-toxic-comment-detector/ .
Accessed: 2018-06-18.
17
[13] Jigsaw toxic comment classiﬁcation challenge. https://www.kaggle.com/
c/jigsaw-toxic-comment-classification-challenge . Accessed: 2018-
06-18.
[14] William Yang Wang. "Liar, Liar Pants on Fire": A new benchmark dataset
for fake news detection. arXiv preprint arXiv:1705.00648 , 2017.
[15] Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for
youtube recommendations. In Proceedings of the 10th ACM Conference on
Recommender Systems , pages 191–198. ACM, 2016.
[16] Wikipedia: AI-complete. https://en.wikipedia.org/wiki/
AI-complete . Accessed: 2018-06-18.
[17] Geoﬀrey Irving, Paul Christiano, and Dario Amodei. AI safety via debate.
arXiv preprint arXiv:1805.00899 , 2018.
[18] Nick Bostrom. Superintelligence: Paths, Dangers, Strategies . Oxford Uni-
versity Press, Inc., New York, NY, USA, 1st edition, 2014.
[19] Paul Christiano. AI Alignment Blog. https://ai-alignment.com . Ac-
cessed: 2018-06-18.
[20] William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans.
Trial without error: Towards safe reinforcement learning via human inter-
vention. arXiv preprint arXiv:1707.05173 , 2017.
[21] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg,
and Dario Amodei. Deep reinforcement learning from human preferences.
InAdvances in Neural Information Processing Systems , pages 4302–4310,
2017.
[22] Paul Christiano. Act-based agents. https://ai-alignment.com/
act-based-agents-8ec926c79e9c . Accessed: 2018-06-18.
[23] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learn-
ing. InAdvances in Neural Information Processing Systems , pages 4565–
4573, 2016.
[24] Owain Evans, Andreas Stuhlmüller, and Noah D Goodman. Learning the
preferences of bounded agents. NIPS Workshop on Bounded Optimality,
2015.
[25] Owain Evans, Andreas Stuhlmüller, and Noah D Goodman. Learning the
preferences of ignorant, inconsistent agents. In Proceedings of the Thirtieth
AAAI Conference on Artiﬁcial Intelligence , pages 323–329. AAAI Press,
2016.
[26] Paul Christiano. Approval maximizing rep-
resentations. https://ai-alignment.com/
approval-maximizing-representations-56ee6a6a1fe6 . Accessed:
2018-06-18.
18
[27] Paul Christiano and Ajeya Cotra. Iterated distilla-
tion and ampliﬁcation. https://ai-alignment.com/
iterated-distillation-and-amplification-157debfd1616 . Accessed:
2018-06-18.
[28] Drew A Hudson and Christopher D Manning. Compositional attention
networks for machine reasoning. arXiv preprint arXiv:1803.03067 , 2018.
[29] Richard Evans, David Saxton, David Amos, Pushmeet Kohli, and Edward
Grefenstette. Can neural networks understand logical entailment? arXiv
preprint arXiv:1802.08535 , 2018.
[30] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J
Gershman. Building machines that learn and think like people. Behavioral
and Brain Sciences , 40, 2017.
[31] Richard Evans and Edward Grefenstette. Learning explanatory rules from
noisy data. Journal of Artiﬁcial Intelligence Research , 61:1–64, 2018.
[32] Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš
Kočisk` y, and Phil Blunsom. Reasoning about entailment with neural at-
tention.arXiv preprint arXiv:1509.06664 , 2015.
[33] Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Dani-
helka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward
Grefenstette, TiagoRamalho, JohnAgapiou, etal. Hybridcomputingusing
a neural network with dynamic external memory. Nature, 538(7626):471,
2016.
[34] Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart
van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards AI-
completequestionanswering: Asetofprerequisitetoytasks. arXiv preprint
arXiv:1502.05698 , 2015.
[35] Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury,
Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. Ask
me anything: Dynamic memory networks for natural language processing.
InInternational Conference on Machine Learning , pages 1378–1387, 2016.
[36] Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and
Daan Wierstra. Draw: A recurrent neural network for image generation.
arXiv preprint arXiv:1502.04623 , 2015.
[37] OriolVinyals,CharlesBlundell,TimLillicrap,DaanWierstra,etal. Match-
ing networks for one shot learning. In Advances in Neural Information
Processing Systems , pages 3630–3638, 2016.
[38] Cezary Kaliszyk, François Chollet, and Christian Szegedy. Holstep: A
machine learning dataset for higher-order logic theorem proving. arXiv
preprint arXiv:1703.00426 , 2017.
[39] Yue Shi, Martha Larson, and Alan Hanjalic. Collaborative ﬁltering beyond
the user-item matrix: A survey of the state of the art and future challenges.
ACM Computing Surveys (CSUR) , 47(1):3, 2014.
19
[40] Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-
supervised learning (chapelle, o. et al., eds.; 2006). IEEE Transactions
on Neural Networks , 20(3):542–542, 2009.
[41] Avital Oliver, Augustus Odena, Colin Raﬀel, Ekin D Cubuk, and Ian J
Goodfellow. Realistic evaluation of deep semi-supervised learning algo-
rithms.arXiv preprint arXiv:1804.09170 , 2018.
[42] Jan Deriu, Maurice Gonzenbach, Fatih Uzdilli, Aurelien Lucchi, Valeria De
Luca, and Martin Jaggi. Swisscheese at semeval-2016 task 4: Sentiment
classiﬁcation using an ensemble of convolutional neural networks with dis-
tant supervision. In Proceedings of the 10th International Workshop on Se-
mantic Evaluation , number EPFL-CONF-229234, pages 1124–1128, 2016.
[43] David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vap-
nik. Unifying distillation and privileged information. arXiv preprint
arXiv:1511.03643 , 2015.
[44] Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch.
Alignment for advanced machine learning systems. Machine Intelligence
Research Institute , 2016.
[45] Volodymyr Kuleshov and Stefano Ermon. Estimating uncertainty online
against an adversary. In AAAI, pages 2110–2116, 2017.
[46] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Sim-
ple and scalable predictive uncertainty estimation using deep ensembles.
InAdvances in Neural Information Processing Systems , pages 6405–6416,
2017.
[47] Sentiment classiﬁcation on large movie review. https://www.kaggle.
com/c/sentiment-classification-on-large-movie-review . Accessed:
2018-06-18.
[48] Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen,
Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. A dataset of
peer reviews (peerread): Collection, insights and nlp applications. arXiv
preprint arXiv:1804.09635 , 2018.
[49] SaraKlingenstein,TimHitchcock,andSimonDeDeo. Thecivilizingprocess
in london’s old bailey. Proceedings of the National Academy of Sciences ,
111(26):9419–9424, 2014.
[50] Gamaleldin F Elsayed, Shreya Shankar, Brian Cheung, Nicolas Paper-
not, Alex Kurakin, Ian Goodfellow, and Jascha Sohl-Dickstein. Adver-
sarial examples that fool both human and computer vision. arXiv preprint
arXiv:1802.08195 , 2018.
[51] Oren Barkan, Noam Koenigstein, and Eylon Yogev. The deep journey from
content to collaborative ﬁltering. arXiv preprint arXiv:1611.00384 , 2016.
[52] Nicolas Hug. Surprise, a Python library for recommender systems. http:
//surpriselib.com , 2017.
20
[53] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-
Seng Chua. Neural collaborative ﬁltering. In Proceedings of the 26th In-
ternational Conference on World Wide Web , WWW ’17, pages 173–182,
Republic and Canton of Geneva, Switzerland, 2017. International World
Wide Web Conferences Steering Committee.
21

Owain Evans
Andreas Stuhlmüller
Chris Cundy
Ryan Carey
Zachary Kenton
Thomas McGrath
Andrew Schreiber


Racing to the precipice: a model of artificial intelligence development

 FHI TECHNICAL REPORT 

Racing to the Precipice : a Model of
Artificial Intelligence Development

Stuart Armstrong
Nick Bostrom
Carl Shulman

Technical Report #20 13-1

Cite as :
Armstrong, S. & Bostrom, N. & Shulman, C. (2013): “Racing to the precipice : a model
of artificial intelligence development ”, Technical Report #2013 -1, Future of Humanity
Institute, Oxford University: pp. 1 -8.

The views expressed herein are those of the author(s) and do not necessarily reflect the
views of the Futu re of Humanity Institute.

Racing to the precipice: a model of articial
intelligence development
Stuart Armstrong Nick Bostrom Carl Shulman
October 2013
Abstract
This paper presents a simple model of an AI arms race, where sev-
eral development teams race to build the rst AI. Under the assumption
that the rst AI will be very powerful and transformative, each team is
incentivised to nish rst { by skimping on safety precautions if need
be. This paper presents the Nash equilibrium of this process, where each
team takes the correct amount of safety precautions in the arms race.
Having extra development teams and extra enmity between teams can
increase the danger of an AI-disaster, especially if risk taking is more
important than skill in developing the AI. Surprisingly, information also
increases the risks: the more teams know about each others' capabilities
(and about their own), the more the danger increases.
1 Introduction
This paper presents a simplied model for analysing technology races. The
model was designed initially for races to construct articial intelligences (AIs).
But it can be applied to other similar races or competitions, especially techno-
logical races where there is a large advantage to reaching the goal rst.
There are arguments that the rst true AIs are likely to be extremely pow-
erful machines [Goo65, Cha10], but that they could end up being dangerous
[Omo08, Yud08] if not carefully controlled [ASB12]. The purpose of various
research projects such as `friendly AI' [MS12] is to design safety precautions
ensuring the creation of an AI with human-compatible values.
In this paper, we won't go that far. We will simply assume that there is
a denite probability of an AI-related disaster, and that the probability goes
up the more the AI development team skimps on precautions. This paper will
present a model of an `arms race' between AI development teams, and analyse
what factors increase and decrease the probability of such an AI-disaster.
Several factors contribute to increasing the danger: if building the AI de-
pends more on risk-taking than on skill, for instance. Reducing the enmity
between AI development teams helps, however, as does reducing the number of
teams.
But surprisingly, extra information can exacerbate the danger. The danger
is minimised when each team is ignorant of the AI-building capabilities of every
team { including their own capabilities. This is an interesting example of an
information hazard [Bos11].
1
2 The model
In this model, there are ndierent teams, competing to build the rst proper
AI. Each team has a certain level of AI-building capability c, and can choose
to take a certain level of safety precautions s, varying between s= 1 (total
safety) and s= 0 (complete lack of precautions). Then each team gets a score
by subtracting their safety from their capability ( cs), and the team with the
highest score `wins', by building their AI rst.
Once that AI is built, we need to check for the success of the project: That
will depend on the safety level of the winning team1. With probability s, they
have a successful and protable AI project. With probability 1 s, they have
caused an AI-disaster.
We'll assume that each team maximises expected utility, and we can renor-
malise their utilities to give utility 1 for a successful AI project, and 0 for an
AI-disaster.
We will further assume that each team has the same level of enmity etowards
each other team. This means that they will have a utility of 1 eif another
team builds a successful AI before they do. This varies between e= 0 (they are
indierent to who builds an AI) and e= 1 (another team building a successful
AI is just as bad as an AI-disaster). It would be possible to have enmities above
1 (this would correspond to teams that hate each other so much that an AI-
disaster is preferable to the triumph of the other team), but that is beyond the
scope of the current paper.
Further, we'll assume that each team's capability is drawn independently
and uniformly on the interval [0 ;], for a single given . A highcorresponds
to a situations where capability is the dominant factor in AI development: one
can achieve very little extra by skimping on precautions. Conversely, for low ,
incurring increased risk can make one much more likely to build the rst AI.
We'll assume that the teams choose their safety based on Nash equilibrium
considerations (this could happen in practice if, for instance, each team reduced
their safety a little bit, then a little bit more, then a little bit more, in response
to the other teams reducing their own safety). We'll then calculate the Nash
equilibria in three information scenarios: when teams don't know anyone's capa-
bilities, when they know their own capabilities, and when they know everyone's
capabilities2.
2.1 No information
In this situation, every team is ignorant of their own or any other team's ca-
pabilities3. Each team will thus choose the level of safety the think is best. In
1And only the winning team { if another team gets a disastrous AI rst by taking lower
precautions, they will `won' the race to build the rst AI.
2Of course, the model can be rened in various ways. One could make capacity information
uncertain and fuzzy, one could have dierent levels of enmity between dierent teams, one
could incorporate uncertainty about the safety levels and the ultimate outcomes, and so on.
Or one could have a dynamic process to determine the outcome, rather than rushing straight
to the Nash equilibrium. But the simple model is enough to gain useful insights.
3It may seem unusual for teams to not know their own capabilities in the real world.
However, this is close to the situation we nd ourselves with current AI research: people and
organisations have a pretty clear idea of what resources and knowledge they have, but don't
know how hard AI is or what routes are most likely to lead there. They are thus eectively
ignorant of their own AI-building capabilities.
2
a symmetric Nash equilibrium, this will be a single number s. Since all teams
draw their capabilities from independent identically distributed random vari-
ables, and have the same level of safety, they each have the same probability of
winning, namely 1 =n.
Upon changing stos, a team's chance of winning increases by
1
+O():
The utility gain from winning rather than letting another win is the safety level
stimes the enmity e. So the extra expected utility from winning is
se
+O():
But the decrease in safety brings in extra risks, were the team to win anyway.
This extra risk is simply time the probability of winning:

n+O():
All other eects are second order in . Setting the change to be zero (to rst
order) in, the general solution is thus
s=
en<en
1en:
The total probability of AI-disaster is 1 s, namely:
1
en<en
0en:
2.2 Private information
In this scenario, each team knows its own capabilities, but not those of the
other teams. Each team will choose a safety level s(x), dependent on their own
capabilities x. In a symmetric Nash equilibrium, each team chooses the same
functions. We may assume xs(x) is a function increasing in x4. In that case,
the team with the highest x, the highest capabilities, always wins.
Givenx, the probability of a team winning is equal to the probability of all
other teams having lower capability: this is
x
n1
. Then changing s(x) to
s(x), the team increases its probability of winning by:
(n1)xn2
(1s0(x))n1+O():
As in the previous case, the expected extra utility is setimes this:
se(n1)xn2
(1s0(x))n1+O():
4If makes no sense that a team with higher capability would have a lower chance of winning
(if so, they would voluntarily destroy part of their capability).
3
The loss in expected utility coming from winning at a lower safety level is:
x
n1
+O():
Solving these equations gives:
s(x) =x
ene+1x<en e+ 1
1xene+ 1:
The total probability of a AI-disaster is calculated by integrating, across all
values ofxin [0;], the risk level 1 s(x) times the probability that the winning
team will have capability x:
Z
0(1s(x))nxn1
ndx=(
1n
(n+1)(nee+1)<en e+ 1
(ene+1)n
(n+1)nene+ 1:
2.3 Public information
In this scenario, every team knows the capabilities of every other team. This
scenario is analysed somewhat dierently that the other. Let  be the dierence
between the capability of the top team and the second ranked team. The top
team always wins, but its safety level stopis determined by . When stop() =
=e, it is not in the interest of the second team to decrease its safety to compete
with the top team. The gain from winning does not compensate for the extra
risks run.
Thus the safety of the top team will be stop= =eif =e < 1 and 1
otherwise. The total probability AI-disaster is calculated by integrating, across
all values of  in [0 ;], the risk level 1 stop() = 1 =etimes the probability
that the dierence between the top two teams is :
Z
0(1stop())n()n1
nd =(
1
e(n+1)<e
(e)n+1
e(n+1)n
e(n+1)+ 1e:
3 Factors determining risk
3.1 Capability vs risk
Intuitively it is clear that increasing the importance of capability must decrease
overall risk. One is less inclined to skimp on safety precautions if can only get
a small advantage from doing so. This intuition is born out by the results: in
every situation, an increase of the importance of capability (an increase in )
reduces the risk. This is illustrated in gures 1a and 1b, as well as all subsequent
gures, demonstrating that the probability of AI-disaster are always decreasing
in. Indeed, around = 0 (i.e. when capability is nearly irrelevant to producing
the rst AI), the only Nash equilibrium is to take no safety precautions at all.
In terms of intervention, there is little we can do about the relative impor-
tance of capability, since this is largely determined by the technology. Our best
bet might be at present to direct research into approaches where there is little
return to risk-taking, prioritising some technological paths over others.
4
(a) Two teams
 (b) Five teams
Figure 1: Risk of dangerous AI arms race for two and ve teams, at enmity 1,
plotted against relative importance of capability. Three information-level sce-
narios: no capability information (full), private capability information (dashes),
and full capability information (dots).
3.2 Compatible goals
It is also intuitively clear that reducing enmity should reduce the risk of AI-
disaster. When competition gets less erce, teams would be willing to take less
risks. This intuition is also born out in our model, as decreases in ealways
reduce the risk. For illustration, contrast the graphs 2a and 2b, where the
enmity is 0:5, with the previous graphs 1a and 1b where the enmity was 1.
(a) Two teams
 (b) Five teams
Figure 2: Risk of dangerous AI arms race for two and ve teams, at enmity 0 :5,
plotted against relative importance of capability. Three information-level sce-
narios: no capability information (full), private capability information (dashes),
and full capability information (dots).
Enmity is something that we can work on by, for instance, building trust
between nations and groups, sharing technologies or discoveries, merging into
joint projects or agreeing to common aims. With this extra coordination, we
could also consider agreements to allow the teams to move away from the Nash
equilibrium, thus avoiding a race to the bottom when the situation is particularly
dangerous (such as low capability ). Friendly teams make friendly AIs.
5
3.3 The curse of too much information
Contrasting gure 1a with gure 2a (or gure 1a with gure 2b), one notices
a curious pattern. The no-information case is always safer than the other two
cases, but the relative safety of private information or common information
depends on the degree on enmity.
It is always better if none of the teams have any idea about anyone's capa-
bility. For maximal enmity ( e= 1), the private information scenario is similarly
safer than the common information one. But for lower e, this does not hold: for
lowand lowe, the public information scenario can be safer than the private
one. Asymptotically, though, the private information case is of order of 1 =n,
while the public information case is of order 1 =.
The reasons for this is that it is only worth taking risks in the private in-
formation if one's capability is low. So to get a high risk, one needs a winner
with low capability, which is equivalent to having all teams at low capability.
The probability of having a single team at low capability diminishes inversely
with. Hence the probability of having all teams at low capability diminishes
as 1=n. In the public information case, the winner will take risks if the second
ranked team is close to its own capability. Being close to a specic other team
is inversely proportional to , and hence the probability of this diminishes as
1=.
Our ability to manipulate the information levels of the teams or teams is
likely to be somewhat limited, though we can encourage them to share more
(in the low capability, low enmity situations) or alternatively to lock up their
private knowledge5(in situation of higher capability or enmity).
3.4 The number of competitors
Finally, we can consider what happens when more teams are involved. Intu-
itively, competition might spur a race to the bottom if there are too many
teams, and the model bears this out: in both the no-information and public
information cases, adding extra teams strictly increases the dangers.
The private information case is more complicated. At low capability, adding
more teams will certainly worsen the situation, but at higher capability, the
eect is reversed6. This can be seen in gure 3, which plots the private infor-
mation risk curves for two teams and ve teams.
In terms of leverage, the most helpful intervention we could undertake to
reduce the number of teams is to encourage groups to merge. Dividing teams
would only be useful in very specic situations, and would need some method
of ensuring that the divided teams did not recall each other's capabilities.
4 Conclusion
We've presented a model of AI arms race (which can be extended to other
types of arms races and competitions) in which teams of AI designers can get
ahead by skimping on safety. If the race takes some time there is a persistent
5Such secrecy can interfere with trust building, though, making it hard to reach agreements
between teams if such agreement is needed.
6This is because only the teams with low capability take risks in cases of private informa-
tion, and the more teams there are, the less likely it is that the winner will be low capability.
6
Figure 3: Risk of dangerous AI arms race for private information teams, at
enmity 0:5, plotted against relative importance of capability. The graph for two
teams is full, the graph for ve teams is dashed.
ratchet eect (or race to the bottom) in which each team reduces their safety
precautions slightly, until a Nash equilibrium is reached{ possibly one of very
high danger.
Several factors in
uence on the probability of AI disaster, three in an intu-
itive way, and one counter-intuitively.
Intuitively, if capability is much more important than the level of security
precautions taken, the overall outcome is safer. Similarly, reduced enmity be-
tween the teams produces better outcomes, as does reducing the number of
teams (in most situations).
Counter-intuitively, increasing the information available to all the teams
(about their own capability or progress towards AI, or that of the other teams)
increases the risk. This is a special case of an information hazard [Bos11]: we'd
be better o not knowing.
This model and variants of if may be of use in planning and regulating the
development of AIs and other disruptive technological innovations.
5 Acknowledgments
We are grateful to Anders Sandberg and Se an S. OhEigeartaigh for helpful
discussions.
References
[ASB12] Stuart Armstrong, Anders Sandberg, and Nick Bostrom. Thinking in-
side the box: Controlling and using an oracle ai. Minds and Machines ,
22:299{324, 2012.
[Bos11] Nick Bostrom. Information hazards: A typology of potential harms
from knowledge. Review of Contemporary Philosophy , pages 44{79,
2011.
7
[Cha10] David Chalmers. The singularity: A philosophical analysis. Journal
of Consciousness Studies , 17:7{65, 2010.
[Goo65] I.J. Good. Speculations concerning the rst ultraintelligent machine.
Advances in Computers , 6, 1965.
[MS12] Luke Muehlhauser and Anna Salamon. Intelligence explosion: Evi-
dence and import. In A. Eden, J. Sraker, J.H. Morr, and E. Stein-
hart, editors, The singularity hypothesis: A scientic and philosophical
assessment . Berlin: Springer, 2012.
[Omo08] Stephen M. Omohundro. The basic ai drives. Frontiers in Articial
Intelligence and applications , 171:483{492, 2008.
[Yud08] Eliezer Yudkowsky. Articial intelligence as a positive and negative
factor in global risk. In Nick Bostrom and Milan M. Cirkovi c, editors,
Global catastrophic risks , pages 308{345, New York, 2008. Oxford Uni-
versity Press.
8


How Useful Is Quantilization For Mitigating Specification-Gaming?

Published as a conference paper at ICLR 2019
HOW USEFUL IS QUANTILIZATION FOR MITIGATING
SPECIFICATION -GAMING ?
Ryan Carey
Oxford University
ryan.carey@philosophy.ox.ac.uk
ABSTRACT
For some tasks, there exists a goal that perfectly describes what the designer wants
the AI system to achieve. For many tasks, however, the best available proxy ob-
jective is only a rough approximation of the designer’s intentions. When given
such a goal, a system that optimizes the proxy objective tends to select degener-
ate solutions where the proxy reward is very different from the designer’s true
reward function. One way to counteract the tendency toward speciﬁcation-gaming
is quantilization, a method that interpolates between imitating demonstrations,
and optimizing the proxy objective. If the demonstrations are of adequate quality,
and the proxy reward overestimates performance, then quantilization has better
guaranteed performance than other strategies. However, if the proxy reward un-
derestimates performance, then either imitation or optimization will offer the best
guarantee. This work introduces three new gym environments: Mountain Car-RR,
Hopper-RR, and Video Pinball-RR, and shows that quantilization outperforms
baselines on these tasks.
1 I NTRODUCTION
Machine learning algorithms have often been most successful when there exists a known reward
function that coincides with the designer’s preferences. For many practical problems, it is hard for
the designer to write down a function that includes all aspects of what they want the machine learning
system to achieve, and instead much of their intent is left implicit. In such situations, maximizing the
best available proxy reward function can lead to unusual behaviour:
•A deep reinforcement learner, trained to play the game Coastrunners, maximized its score
by traveling in a small, ﬁxed circle, in order to hit a single boost, rather than completing the
race (Amodei & Clark, 2016).
•Agents evolved to walk, managed to maximize their velocity by growing large legs, and
then falling over (Ha, 2018; Sims, 1994; Lehman et al., 2018).
•Systems trained to swim or to locomote in simulated physical environments performed
these tasks by obtaining free energy, by exploiting errors in the mechanics of collisions,
and imprecision in the methods used for numerical integration (Sims, 1994; Lehman et al.,
2018).
Sometimes, a designer values this kind of creativity, and can use it to patch errors in the reward
function, or the physics model in which learning is occurring. In other cases, however, it is simply
unhelpful. Where an agent scores highly on the explicit reward but poorly with respect to the
designer’s desires, we say that the agent is gaming the designer’s speciﬁcations.
One way to avoid speciﬁcation-gaming is to transmit our implicit desires to the agent by providing
demonstrations of the task. One algorithm designed to mitigate speciﬁcation-gaming in this setting
is quantilization (Taylor, 2016). In quantilization, the agent orders the demonstrations according to
their attainment of the proxy reward, and then imitates just those demonstrations that fall in some top
q-quantile. Quantilization can surpass the average performance of the demonstrations, yet it only
diverges from the demonstrations to a limited degree, and so the extent to which it will game the
speciﬁcations is bounded.
1
Published as a conference paper at ICLR 2019
Figure 1: Three examples of speciﬁcation-gaming: A SARSA agent trained to increase rightward
displacement stays on the right hill rather than going to the ﬁnish-line; a PPO agent trained on average
ankle angle, leans forward and falls over rather than walking; a Rainbow agent traps the ball between
the upper-right bumpers, where it will bounce indeﬁnitely, rather than hitting the ball around the
table.
The aim of this piece is to investigate when quantilization avoids or does not avoid speciﬁcation-
gaming, and to understand the reasons underlying its success or failure. This includes a mixture of
theoretical and empirical investigation.
The theoretical analysis in Section 2 begins by introducing a new framework for describing scenarios
where the explicit reward differs from the performance function: the Robust Reward Problem. Then,
the performance guarantees for quantilizers are compared with those obtained by optimization and
imitation.
Section 3 presents three concrete Robust Reward Tasks, and describes how quantilization empirically
performs in these tasks relative to baselines. Section 4 reviews related work, then 5 concludes.
2 Q UANTILIZATION AND ITS THEORETICAL ANALYSIS
2.1 S ETTING AND GOAL
In order for an AI system to be robust to reward mis-speciﬁcation, it must perform well not just
for the explicit reward function that it has been given, but for some class of plausible performance
measures that might correspond to the designer’s intent (Hadﬁeld-Menell et al., 2017).
As a framework for analyzing these kinds of problems, it is useful to suppose that the agent’s
performance Ucan be decomposed into an explicit reward Uand an implicit (and unknown) reward
I. In order to constrain the possible values of I, the agent is also given some demonstrations. Formally,
we can deﬁne a robust reward problem as:
Deﬁnition 1 (Robust reward problem) .A robust reward problem is described by a tuple P=
hA;U;I;Di:
A, the action space
U:A!R, the explicit reward function
IRA, the space of implicit reward functions
D2A, the distribution over actions that comes from human demonstrations, where
denotes the probability simplex (i.e. A:=fa2RjAj:ai0;P
iai= 1g)
In a robust reward problem, the agent chooses a strategy S2A. Then, an action ais sampled from
S, and the performance is the sum of explicit reward and implicit reward: U(a) :=U(a) +I(a).
If a robust reward problem has an unlimited set of implicit rewards then no strategy can assure good
performance. So this paper focuses solely on implicit reward functions that meet the following
condition:
Deﬁnition 2 (k-Adequate demonstration) .A distribution D2Ademonstrates an implicit reward
functionI k-adequately for k2R+, iff:
EaD[I(a)]k
2
Published as a conference paper at ICLR 2019
The logic behind k-adequacy is that a good teacher may make leave some information out of their
utility function (or their explicit instructions). But any required behavior that is not encoded in the
utility function must instead properly demonstrated.
This paper analyzes which strategies achieve the best guaranteed performance across all k-adequately
demonstrated functions Iin a robust reward problem, i.e.:
max
S2Amin
I2I:EaD[I(a)]kEaS[U(a) +I(a)] (1)
2.2 Q UANTILIZATION AND PERFORMANCE GUARANTEES
Deﬁnition 3 (q-quantilizer) .LetAbe the random variable deﬁned by the distribution Dand let
(a1;:::;an)be a re-ordering1of the actionsAsuch thatU(a1)U(an).
Letq2(0;1]be the leniency. Let the threshold be Mq= minfM2R:P(U(A)> M )qgif
q<1andU(a1)ifq= 1, and deﬁneiqsuch thatU(aiq) =Mq. Then, aQqquantilizer is the agent
that uniformly samples actions from Qq:A! [0;1]where:
Qq(ai) =8
><
>:D(ai)
qifi>iq
1Pn
j=iq+1D(aj)
qifi=iq
0 otherwise
Intuitively, a 0:1-quantilizer selects a random action from the top 10% of the demonstrations, ranked
according to the explicit utility function U. Quantilization has optimization and imitation as its
extreme cases: a 1-quantilizer equals the demonstration distribution D, whereas in the limit of
decreasing leniency ( q!0), a quantilizer chooses the optimal action in the support of D. (Taylor,
2016).
A quantilizer only chooses actions at-most1
qtimes more frequently than the base distribution.
Lemma 4 (Quantilizer property) .Aq-quantilizer selects any action awith probability upper-bounded
by:Qq(a)D(a)
q.
It can now be demonstrated how well a quantilizer performs for robust reward problems, assuming
k-adequacy, for various different implicit reward spaces. Firstly, Taylor (2016) gives a guarantee
for robust reward problems with I(1;0]A. This is extended slightly to cover Robust Reward
Problems withI(1;c], as shown by a proof in Appendix A:
Theorem 5 (Quantilizer guarantee (Taylor, 2016)) .For anyk-adequately demonstrated reward
functionIin a robust reward problem PwithI(1;c]A,c2R+, eachq-quantilizer strategy
Qqhas performance lower-bounded by EaQq[U(a)]EaQq[U(a)]k
q+q1
qc.
Theorem 6 (Quantilizer optimality (Taylor, 2016)) .
aFor any strategy Sin a robust reward problem PwithI= (1;c]A,c2R+, ifSis
a distribution over the support of the demonstrations D, then there exists a k-adequately
demonstrated implicit reward I2Ithat upper-bounds performance as: EaS[U(a)]
EaQq[U(a)]k
q+(q1)
qcwhereq:= mina2supp(S)D(a)
S(a).
bFor any strategy Sfor which some action ahasS(a)>0,D(a) = 0 , some implicit reward
k-adequately demonstrated by Dgives performance unbounded below.
Taken together, these theorems imply that if I= (1;c]A, then the strategy that guarantees best
performance for all k-adequately demonstrated reward functions is the q-quantilizer.
Importantly, some algorithms that combine imitation and optimization may have performance un-
bounded below as in Theorem 6b. For example, as the number of actions approaches a continuum, all
1In cases where the quantile is not uniquely determined because different actions have the same utility, they
may be ordered arbitrarily, as per Taylor (2016)
3
Published as a conference paper at ICLR 2019
actions have D(a) = 0 , so any pure strategy in a continuum setting has reward unbounded below,
including all algorithms that choose one action to maximize a weighted sum of optimization and
imitation objectives.
The next results will extend Taylor (2016) by considering whether quantilization is optimal in two
further cases. First, the implicit reward may be bounded below by some c. Second, it may be
neither bounded above, nor below.
Theorem 7 (Optimizer/imitator guarantee) .For anyk-adequately demonstrated reward function I
in a robust reward problem PwithI[c;1)A,c2R+, theU-optimal action has performance
lower-bounded by U(a)maxa2AU(a)cand the demonstration distribution Dhas performance
lower-bounded by EaD[U(a)]EaD[U(a)]k.
Theorem 8 (Optimizer/imitator optimality) .For any strategy Sin a robust reward problem Pwith
I= (c;1]A,c2R+, there exists a k-adequately demonstrated implicit reward I2Ithat upper-
bounds performance as: EaS[U(a)]EaD[U(a)]kandEaS[U(a)]maxa2AU(a)c
These results imply that if I= (c;1]A, the strategy that gets best performance will be the U-
optimal action, or else will be the demonstration distribution D, and (and not q-quantilization with
q2(0;1)).
Theorem 9 (Imitator guarantee) .For anyk-adequately demonstrated reward function Iin a robust
reward problem PwithIRA, the demonstration distribution Dhas performance lower-bounded
byEaD[U(a)]k.
Theorem 10 (Imitator optimality) .For any strategy Sthat does not equal the demonstration distribu-
tionDin a robust reward problem PwithI=RA,c2R+, there exists a k-adequately demonstrated
implicit reward I2Ithat gives performance EaS[U(a)] =1
Proofs are given in the appendix. Altogether, although quantilization gives the best guarantee in the
upper-bounded implicit reward setting, this does not carry over to the unbounded implicit reward
setting, where it lacks any performance guarantee, nor to the lower-bounded implicit reward setting,
where its guarantee will be inferior to that of either imitation or optimization.
3 E XPERIMENTS
3.1 E NVIRONMENTS
Three robust-reward (RR) environments have been developed. In order of increasing complexity, they
are: Mountain Car-RR, Hopper-RR, and Video Pinball-RR. Each of these operates the same as a
corresponding environment from OpenAI Gym (Brockman et al., 2016), except that the reward is
changed. These environments (and the rest of the project) are available on Github2.
Mountain Car-RR is based on the usual Mountain Car gym environment, involving a car situated
between two hills. The performance is 1per timestep, which is the reward in the original gym
environment. What the agent is given, however, as its explicit (proxy) reward, is its rightward
displacement at each timestep.
In Hopper-RR, the performance is a 1 +horizontal movement per timestep, as in the original gym
environment. That is, the performance for a whole episode is a sum of episode duration and horizontal
displacement during that episode. The explicit reward is the average forward lean across the episode,
which is given at the end of each episode.
Video Pinball-RR is introduced because it is an Atari game where it is known that reward can be
achieved unconventionally, by trapping the ball between the upper-right bumpers, and nudging the
table repeatedly to keep the ball there indeﬁnitely. Although this is a clever solution, suppose that
the designer wants the AI system to achieve a high-score without carrying out this behaviour. Then,
let the explicit reward Ube the in-game score, and let the implicit reward, Ibeeach timestep if
the ball is between the upper right bumpers, and 0otherwise. Trajectories were taken from the Atari
Grand Challenge dataset (Kurin et al., 2017). Further details on these experiments are available in
Appendix B.
2www.github.com/mtrazzi/quantilizers
4
Published as a conference paper at ICLR 2019
Figure 2: The demonstration distributions for each of the three robust reward tasks. For Mountain
Car-RR and Hopper-RR, the implicit reward (difference between performance and explicit reward) is
unbounded. Implicit reward in Video Pinball-RR is upper-bounded at 0.
Task Trajectories Explicit reward ( U) True reward ( U)
Mountain Car-RR 901 Rightward displacement In-built score
Hopper-RR 3835 Forward lean In-built score
Video Pinball-RR 380 In-built score Score - proportion of time between bumpers
Table 1: Characteristics of the three datasets
To describe these multi-step problems as Robust Reward Problems, Amust be understood as the set of
agent policies. The implicit reward space may be set to I=RA, and the explicit and implicit reward
functionsUandIare the returns across a whole episode (these reward functions are summarized in
3.1).
3.2 M ETHODS
For each task, quantilizers with 0:125q0:5are compared to an imitator (i.e. a 1-quantilizer),
and to a reinforcement learning agent trained to maximize the proxy reward U.
For Mountain Car-RR and Hopper-RR, the imitator and quantilizers were trained with behavioral
cloning, using a two-layer 20-node MLP. In Hopper-RR, since the action consisted of controlling three
joints, three networks were used to prediction actions in f1;0;1g, and for rollouts, actions were
selected by taking the expected value of the predicted movement at each joint. The reinforcement
learning agent for Mountain Car-RR was a SARSA agent trained with a linear function approximator
with a radial basis function kernel. For Hopper, the reinforcement learning agent was proximal policy
optimization (PPO) (Schulman et al., 2017). For MountainCar-RR, 901 trajectories were used, which
were by the experimenter manually playing Mountain Car. For Hopper-RR, 3835 trajectories were
used, 1462 of which were produced by the author, and 2273 of them were produced by a student
volunteer, over a total time of around four hours.
For Video Pinball-RR, the imitator and quantilizers were trained with behavioral cloning using a
deep convolutional architecture, whereas the reinforcement learning agent was trained with Rainbow
(Hessel et al., 2018). In each of these, the convolutional architecture was the same, based on the
default settings from Hessel et al. (2018). For Video Pinball-RR, 380 trajectories were taken from the
second batch of the Atari Grand Challenge dataset (Kurin et al., 2017).
3.3 R ESULTS AND DISCUSSION
In experiments, there are (at least) two sources of additional error that may compromise the theoretical
guarantees. Firstly, the quantilizers (like the imitation algorithm) only have access to trajectories that
are sampled from the base distribution, but lack access to the base distribution itself. Second, the
quantilizer only approximates the distribution Qq, and so its performance may be worse than the
guarantee, which applies to Qqitself. So it is useful to see how these algorithms perform in practice,
and whether this coincides with the cases where they have theoretical guarantees.
5
Published as a conference paper at ICLR 2019
Figure 3: Performance of imitation, 0:5,0:25,0:1and0:01quantilizer (n=5 seeds), and
reinforcement learning agents (respectively SARSA, PPO and Rainbow), on each of the three robust
reward tasks. Higher score is better for true reward and explicit reward but worse for implicit loss.
For Mountain Car and Hopper, the explicit reward increases when qdecreases. For Video Pinball, no
clear trend appeared.
Although the quantilizers did not have any guarantee of outperforming the optimizers, they did.
In all three environments, the optimizer was able to obtain high explicit reward, but gamed the
speciﬁcations, such that it obtained low U.
The quantilizers were less successful when compared to the imitation baseline. In Hopper-RR, the
quantilizer with q= 0:01substantially outperformed the imitation agent. This was expected because
the quantilizers were imitating a higher-performing set of trajectories, as shown in Figure 3.1. The
quantilizers with larger qdid not substantially improve upon imitation. In Mountain Car-RR and
Video Pinball-RR, the quantilizers performed somewhat worse than imitation.
In theory, quantilization should be expected to perform best in Video Pinball-RR, where there actually
exists a guarantee, because implicit loss is upper-bounded at zero usage of the bumper trick (Figure
3.1). However, in practice, quantilization performed best in Hopper-RR, and only somewhat well in
Mountain Car-RR and Video Pinball-RR.
4 R ELATED WORK
Our work relates to the literature on speciﬁcation-gaming, and mixtures of optimization and imitation.
Speciﬁcation-gaming is a well-known phenomenon in evolutionary algorithms and in robotics
(Lehman et al., 2018), but the ﬁrst formal proposal for addressing it was quantilization (Taylor, 2016).
In Everitt et al. (2017), this was analyzed in the context of ergodic multi-step decision problems.
The present work goes further by considering a more general problem setting, by extending our
experiments to more complex problems, and by introducing human demonstrations. Interestingly,
quantilization was independently rediscovered as an approach to model-based planning, where
the proxy reward corresponds to the performance that a policy is estimated to obtain in a learned
world-model (Mishra et al., 2017), although they do not analyze in much detail why this approach
succeeds.
Quantilization is situated in a similar problem setting to broader efforts to combine optimization
(including reinforcement learning) with imitation. To the author’s knowledge, published algorithms
are all variations on using a weighted sum of reinforcement learning and imitation rewards, for
example Hester et al. (2018), which adds an initial pre-initialization step of pure imitation, and Nair
et al. (2018), which ignores imitation samples where performance is worse than that expected by the
AI system. These solutions, however, are not designed to deal with errors in the reward function, and,
as shown in Section 2, they can incur arbitrarily severe losses if they are used to do so.
6
Published as a conference paper at ICLR 2019
5 C ONCLUSION
To sum up, quantilization has a wide potential range of applicability for reducing speciﬁcation-gaming.
In theory, it only provides useful guarantees when the possible implicit reward is upper-bounded. In
practice, when the true reward is closer to the proxy reward, or local minima exist in the explicit
reward, it may be useful for a much wider range of cases.
ACKNOWLEDGMENTS
Thanks to Michaël Trazzi for feedback and codebase improvements. Thank you anonymous reviewers,
for suggested improvements to the text. Thanks to Marek Sklenka and Sam Clarke for help with data
collection.
REFERENCES
Dario Amodei and Jack Clark. Faulty reward functions in the wild, 2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and
Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.
Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. Reinforcement
learning with a corrupted reward channel. arXiv preprint arXiv:1705.08417 , 2017.
David Ha. Reinforcement learning for improving agent design. arXiv preprint arXiv:1810.03779 ,
2018.
Dylan Hadﬁeld-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse
reward design. In Advances in Neural Information Processing Systems , pp. 6765–6774, 2017.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan
Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in
deep reinforcement learning. In Thirty-Second AAAI Conference on Artiﬁcial Intelligence , 2018.
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan,
John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In
Thirty-Second AAAI Conference on Artiﬁcial Intelligence , 2018.
Łukasz Kidzi ´nski, Sharada Prasanna Mohanty, Carmichael Ong, Zhewei Huang, Shuchang Zhou,
Anton Pechenko, Adam Stelmaszczyk, Piotr Jarosik, Mikhail Pavlov, Sergey Kolesnikov, et al.
Learning to run challenge solutions: Adapting reinforcement learning methods for neuromuscu-
loskeletal environments. arXiv preprint arXiv:1804.00361 , 2018.
Vitaly Kurin, Sebastian Nowozin, Katja Hofmann, Lucas Beyer, and Bastian Leibe. The atari grand
challenge dataset. arXiv preprint arXiv:1705.10998 , 2017.
Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Julie Beaulieu, Peter J Bentley, Samuel
Bernard, Guillaume Belson, David M Bryson, Nick Cheney, et al. The surprising creativity of
digital evolution: A collection of anecdotes from the evolutionary computation and artiﬁcial life
research communities. arXiv preprint arXiv:1803.03453 , 2018.
Nikhil Mishra, Pieter Abbeel, and Igor Mordatch. Prediction and control with temporal segment
models. In International Conference on Machine Learning , pp. 2459–2468, 2017.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over-
coming exploration in reinforcement learning with demonstrations. In 2018 IEEE International
Conference on Robotics and Automation (ICRA) , pp. 6292–6299. IEEE, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy
optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.
Karl Sims. Evolving virtual creatures. In Proceedings of the 21st annual conference on Computer
graphics and interactive techniques , pp. 15–22. ACM, 1994.
Jessica Taylor. Quantilizers: A safer alternative to maximizers for limited optimization. In AAAI
Workshop: AI, Ethics, and Society , 2016.
7
Published as a conference paper at ICLR 2019
A P ROOFS
Proof of Lemma 4. For actionsa6=aiq, the upper bound follows immediately from Deﬁnition 3.
Foraiq, note that the probability of sampling actions from fa1;:::;aiqgis by construction higher than
q. Therefore:
qX
iiqD(ai)
=D(aiq) +X
i>iqD(ai)
)D(aiq)
q1P
i>iqD(ai)
q=Qq(aiq)
Theorem 5 (Quantilizer guarantee (Taylor, 2016)) .For anyk-adequately demonstrated reward
functionIin a robust reward problem PwithI(1;c]A,c2R+, eachq-quantilizer strategy
Qqwithq2(0;1]has performance lower-bounded by EaQq[U(a)]EaQq[U(a)]k
q+q1
qc.
Proof of Theorem 5. SinceIisk-adequately demonstrated, we have:
kEaD[I(a)]
=X
aD(a)(I(a)c) +c
X
aqQq(a)(I(a)c) +c (from Lemma 4 and I(a)c0)
=qEaQq[I(a)]cq+c
)EaQq[I(a)]k
q+(q1)
qc
Theorem 6 (Quantilizer optimality (Taylor, 2016)) .
aFor any strategy Sin a robust reward problem PwithI= (1;c]A,c2R+, ifSis
a distribution over the support of the demonstrations D, then there exists a k-adequately
demonstrated implicit reward I2Ithat upper-bounds performance as: EaS[U(a)]
EaQq[U(a)]k
q+(q1)
qcwhereq:= mina2supp(S)D(a)
S(a).
bFor any strategy Sfor which some action ahasS(a)>0,D(a) = 0 , some implicit reward
functionk-adequately demonstrated by Dgives performance unbounded below.
Proof of Theorem 6.
a) Letq= mina2supp(S)D(a)
S(a)2(0;1]andabe such thatD(a)
S(a)=q. Then, the function Ideﬁned
by:
I(a) =(
k(1D(a))c
D(a)a2a
c a 62a
hasEaS[I(a)]kso is ak-adequately demonstrated function. Furthermore, if we instead take
the expected value over S, we get the bound for the implicit reward of a q-quantilizer (per Theorem
5):
EaS[I(a)] =S(a)k(1D(a))c
D(a)+ (1S(a))c=k
D(a)=S(a)+D(a)=S(a)1
D(a)=S(a)c
8
Published as a conference paper at ICLR 2019
Sinceq= mina2supp(S)D(a)
S(a), the quantilizer chooses top actions at least as often as S:
P
i>iqD(ai)
qP
i>iqS(ai). The better actions beat the median, and the worse actions are
inferior:
U(aj)MqU(ai)forj iqD(ai)
q1
AMq+X
i>iqD(ai)
qU(ai)
0
@1X
i>iqD(ai)
q1
AMq+0
@X
i>iqD(ai)
qX
i>iqS(a)1
AMq+X
i>iqS(ai)U(ai)
0
@1X
i>iqS(a)1
AMq+X
i>iqS(ai)U(ai)
X
iS(ai)U(ai)
Adding these two ﬁgures yields EaS[U(a)]EaQq[U(a)]k
q+(q1)
qc, giving the result
(and showing that performance is at least as bad as a q-quantilizer).
b) If a strategy Splaces any probability on any action a0not supported by D, thenI(a) =m a2a
0 otherwise(wherem2R+) is ak-adequately-demonstrated implicit reward, which will give
implicit reward unbounded below for a particular value of q (for instance q=1
2) ifmis sufﬁciently
large, and hence the same for performance.
Theorem 7 (Optimizer/imitator guarantee) .For anyk-adequately demonstrated reward function I
in a robust reward problem PwithI[c;1)A,c2R+, theU-optimal action has performance
lower-bounded by U(a)maxa2AU(a)cand the demonstration distribution Dhas performance
lower-bounded by EaD[U(a)]EaD[U(a)]k.
Proof of Theorem 7. By assumption, I(a)2[c;1), so an optimizer that chooses a U-optimal
actionawill guarantee performance of U(a) =U(a) +I(a)U(a)cByk-adequacy,
imitation of Dwill guarantee performance of EaD[U(a)] =EaD[U(a)] +EaD[I(a)]
EaD[U(a)]k
Theorem 8 (Optimizer/imitator optimality) .For any strategy Sin a robust reward problem Pwith
I= (c;1]A,c2R+, there exists a k-adequately demonstrated implicit reward I2Ithat upper-
bounds performance as: EaS[U(a)]EaD[U(a)]kandEaS[U(a)]maxa2AU(a)c
Proof of Theorem 8. For any strategy S, letA0= arg min a2supp(D)S(a)
D(a). Let=S(A0)
D(A0)2[0;1].
Then we can decompose Sinto a weighted sum of the demonstration distribution D, and the
distributionS0, whereS=D+ (1)S0, where the support of S0does not intersect A0.
Then, there exists a k-adequately demonstrated implicit reward function
I=8
><
>:c a2supp(S0)
k+cpD(a2S0)
pD(a2A0)ifa2A0
0 otherwise
giving performance of:
EaS[U(a)] =EaD[U(a) +I(a)] + (1)EaS0[U(a) +I(a)]
=(EaD[U(a)]k) + (1)EaS0[U(a)c]
(EaD[U(a)k)(1)(max
a2AU(a)c)
9
Published as a conference paper at ICLR 2019
Since this is a convex combination of EaD[U(a)]kandmaxa2AU(a)c, it cannot be greater
than either.
Theorem 9 (Imitator guarantee) .For anyk-adequately demonstrated reward function Iin a robust
reward problem PwithIRA, the demonstration distribution Dhas performance lower-bounded
byEaD[U(a)]EaD[U(a)]k.
Proof of Theorem 9. Fromk-adequacy, imitation has the guarantee that EaD[U(a) +I(a)]
EaD[U(a)]k
Theorem 10 (Imitator optimality) .For any strategy Sthat does not equal the demonstration distribu-
tionDin a robust reward problem PwithI=RA,c2R+, there exists a k-adequately demonstrated
implicit reward I2Ithat gives performance EaS[U(a)] =1
Proof of Theorem 10. For any strategy Sthat does not equal the base distribution D, there can be
unbounded expected implicit loss. This is because such a strategy Smust contain at least one action
a+that is taken more frequently in Sthan inD, and at least one action athat is taken less frequently.
For any desired level of implicit reward l0, there exists a k-adequately demonstrated reward
I(a) =8
><
>:
D(a+)a=a+

D(a)a=a
0 otherwise
where=l
S(a)=D(a)S(a+)=D(a+)such that EaS[I(a)] =EaS[U(a)] +l, so the overall perfor-
mance EaS[U(a) +I(a)]is unbounded below.
B F URTHER EXPERIMENTAL DETAILS
Here, for each task, we provide details regarding the proxy reward, collection of trajectories, hyperpa-
rameter values, and other information to assist with replication.
In Mountain Car-RR, for the proxy reward, we used the rightward position of the cart, which starts at
(0:6;0:4)and is 0:5at the goal position (same as in OpenAI gym).
SARSA with a linear function approximator was used because if optimizer for U, it was able to
achieve good performance, unlike a deep q learner, which reaches a local minimum even when given
the true reward. SARSA was trained for 200 episodes, with learning rate 0:01with four radial basis
function kernels, with gamma values of 5,2,1, and 0:5, and 100components each. The quantilizer
and imitation systems were trained with a two layers, each 20nodes wide for 200 iterations, which
are the same hyperparameters used for quantilization and imitation in Hopper-RR.
In Hopper-RR, the proxy reward was the mean ankle angle (i.e. clockwise angle between the tibia
and the right part of the foot in radians) as a reward at the end of an episode. This incentivized gaits
that lean forward at the ankle, which is a desired property in reward shaping for gait Kidzi ´nski et al.
(2018), and in fact is correlated with performance in humans (cf. Figure 2). For gathering human
data, we discretized the original OpenAI gym action space (continuous in three dimensions) into
33actions (the user had six control keys, two for each axis, and could apply either a positive +1,
negative1or zero force to each one at each timestep). PPO was trained for 2048 steps with learning
rate3e4with gamma 0:99, and lambda 0:95.
In Video Pinball-RR, we deﬁne the true score UasU=U+IwhereUis the explicit reward (here
the in-game score) and Iis the implicit reward, times the proportion of time spent between the
bumpers, measured using a hard-coded function that tells if the ball is between the bumpers or not).
2[0;+1[is a weight denoting how strongly the designer disprefers the agent using the bumper
trick. We used standard Deepmind wrappers that downsample to 8484frames, skip multiple frames
at initialization and successive frames together in sets of four. In order to match the environment with
that of the Atari Grand Challenge data, we did not have the end of a life trigger the end of an episode,
nor did we clip rewards. The imitator and quantilizers were trained for 10;000steps, using a learning
rate of 1e3, a batch size of 512. The rainbow agent was trained for 50M frames. The imitator,
quantilizer and rainbow agent were all trained with the same deep convolutional architecture as in
10
Published as a conference paper at ICLR 2019
Hessel et al. (2018), using out-sizes of (32;64;64), kernel sizes of (8;4;3), stride-sizes of (4;2;1),
followed by two linear layers of size 512.
11