Learning to summarize with human feedback

Learning to summarize from human feedback
Nisan StiennonLong OuyangJeff WuDaniel M. ZieglerRyan Lowe
Chelsea VossAlec Radford Dario Amodei Paul Christiano
OpenAI
Abstract
As language models become more powerful, training and evaluation are increas-
ingly bottlenecked by the data and metrics used for a particular task. For example,
summarization models are often trained to predict human reference summaries and
evaluated using ROUGE, but both of these metrics are rough proxies for what we
really care about—summary quality. In this work, we show that it is possible to
signiﬁcantly improve summary quality by training a model to optimize for human
preferences. We collect a large, high-quality dataset of human comparisons be-
tween summaries, train a model to predict the human-preferred summary, and use
that model as a reward function to ﬁne-tune a summarization policy using reinforce-
ment learning. We apply our method to a version of the TL;DR dataset of Reddit
posts [ 63] and ﬁnd that our models signiﬁcantly outperform both human reference
summaries and much larger models ﬁne-tuned with supervised learning alone. Our
models also transfer to CNN/DM news articles [ 22], producing summaries nearly
as good as the human reference without any news-speciﬁc ﬁne-tuning.2We con-
duct extensive analyses to understand our human feedback dataset and ﬁne-tuned
models.3We establish that our reward model generalizes to new datasets, and that
optimizing our reward model results in better summaries than optimizing ROUGE
according to humans. We hope the evidence from our paper motivates machine
learning researchers to pay closer attention to how their training loss affects the
model behavior they actually want.
1 Introduction
Large-scale language model pretraining has become increasingly prevalent for achieving high per-
formance on a variety of natural language processing (NLP) tasks. When applying these models
to a speciﬁc task, they are usually ﬁne-tuned using supervised learning, often to maximize the log
probability of a set of human demonstrations.
While this strategy has led to markedly improved performance, there is still a misalignment between
this ﬁne-tuning objective—maximizing the likelihood of human-written text—and what we care
about—generating high-quality outputs as determined by humans. This misalignment has several
causes: the maximum likelihood objective has no distinction between important errors (e.g. making
up facts [ 41]) and unimportant errors (e.g. selecting the precise word from a set of synonyms); models
This was a joint project of the OpenAI Reﬂection team. Author order was randomized amongst {LO, JW,
DZ, NS}; CV and RL were full-time contributors for most of the duration. PC is the team lead.
2Samples from all of our models can be viewed on our website.
3We provide inference code for our 1.3B models and baselines, as well as a model card and our human
feedback dataset with over 64k summary comparisons, here.
34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.
Supervised learningHuman feedback
Pretrain onlyReference summariesFigure 1: Fraction of the time humans prefer our models’ summaries over the human-generated
reference summaries on the TL;DR dataset.4Since quality judgments involve an arbitrary decision
about how to trade off summary length vs. coverage within the 24-48 token limit, we also provide
length-controlled graphs in Appendix F; length differences explain about a third of the gap between
feedback and supervised learning at 6.7B.
are incentivized to place probability mass on all human demonstrations, including those that are
low-quality; and distributional shift during sampling can degrade performance [ 56,52]. Quality can
often be improved signiﬁcantly by non-uniform sampling strategies such as beam search [ 51], but
these can lead to repetition and other undesirable artifacts [ 69,23]. Optimizing for quality may be a
principled approach to overcoming these problems.
Our goal in this paper is to advance methods for training language models on objectives that more
closely capture the behavior we care about. To make short-term progress towards this goal, we
focus on abstractive English text summarization, as it has a long history in the NLP community
[16,8,54,59,50], and is a subjective task where we believe it is difﬁcult to quantify summary quality
without human judgments. Indeed, existing automatic metrics for evaluating summary quality, such
as ROUGE [39], have received criticism for poor correlation with human judgments [55, 45, 6, 33].
We follow the works of [ 3,73], who ﬁne-tune language models from human feedback using reward
learning [ 35]. We ﬁrst collect a dataset of human preferences between pairs of summaries, then train
a reward model (RM) via supervised learning to predict the human-preferred summary. Finally, we
train a policy via reinforcement learning (RL) to maximize the score given by the RM; the policy
generates a token of text at each ‘time step’, and is updated using the PPO algorithm [ 58] based on
the RM ‘reward’ given to the entire generated summary. We can then gather more human data using
samples from the resulting policy, and repeat the process. We follow the works of [ 48,4] and use
large pretrained GPT-3 models with as many as 6.7 billion parameters.
Our main contributions are four-fold.
(1) We show that training with human feedback signiﬁcantly outperforms very strong baselines
on English summarization. When applying our methods on a version of the Reddit TL;DR dataset
[63], we train policies via human feedback that produce better summaries than much larger policies
trained via supervised learning. Summaries from our human feedback models are preferred by our
labelers to the original human demonstrations in the dataset (see Figure 1).
(2) We show human feedback models generalize much better to new domains than supervised
models. Our Reddit-trained human feedback models also generate high-quality summaries of news
articles on the CNN/DailyMail (CNN/DM) dataset without any news-speciﬁc ﬁne-tuning, almost
matching the quality of the dataset’s reference summaries. We perform several checks to ensure
that these human preferences reﬂect a real quality difference: we consistently monitor agreement
rates amongst labelers and researchers, and ﬁnd researcher-labeler agreement rates are nearly as high
as researcher-researcher agreement rates (see Section C.2), and we verify models are not merely
optimizing simple metrics like length or amount of copying (see Appendices F and G.7).
4Throughout the paper, error bars represent 1 standard error.
2
(3) We conduct extensive empirical analyses of our policy and reward model. We examine the
impact of model and data size (Figure 6), study performance as we continue to optimize a given
reward model (Section 4.3), and analyze reward model performance using synthetic and human-
written perturbations of summaries (Section 4.3). We conﬁrm that our reward model outperforms
other metrics such as ROUGE at predicting human preferences, and that optimizing our reward model
directly results in better summaries than optimizing ROUGE according to humans (Section 4.4).
(4) We publicly release our human feedback dataset for further research. The dataset contains
64,832 summary comparisons on the TL;DR dataset, as well as our evaluation data on both TL;DR
(comparisons and Likert scores) and CNN/DM (Likert scores).
The methods we present in this paper are motivated in part by longer-term concerns about the
misalignment of AI systems with what humans want them to do. When misaligned summarization
models make up facts, their mistakes are fairly low-risk and easy to spot. However, as AI systems
become more powerful and are given increasingly important tasks, the mistakes they make will likely
become more subtle and safety-critical, making this an important area for further research.
2 Related work
Most directly related to our work is previous work using human feedback to train summarization
models with RL [ 3,73]. Bohm et al. [ 3] learn a reward function from a dataset of human ratings of
2.5k CNN/DM summaries, and train a policy whose summaries are preferred to a policy optimizing
ROUGE. Our work is most similar to [ 73], who also train Transformer models [ 62] to optimize human
feedback across a range of tasks, including summarization on the Reddit TL;DR and CNN/DM
datasets. Unlike us, they train in an online manner and ﬁnd the model highly extractive. They
note that their labelers prefer extractive summaries and have low agreement rates with researchers.
Compared to [ 73], we use signiﬁcantly larger models, move to the batch setting for collecting human
feedback, ensure high labeler-researcher agreement, and make some algorithmic modiﬁcations, such
as separating the policy and value networks.
Human feedback has also been used as a reward to train models in other domains such as dialogue
[25,68,21], translation [ 32,1], semantic parsing [ 34], story generation [ 72], review generation
[7], and evidence extraction [ 46]. Our reward modeling approach was developed in prior work
on learning to rank [ 40], which has been applied to ranking search results using either explicit
feedback [ 2,18] or implicit feedback in the form of click-through data [ 29,30]. In a related line of
research, human feedback has been used to train agents in simulated environments [ 10,24]. There
is also a rich literature on using RL to optimize automatic metrics for NLP tasks, such as ROUGE
for summarization [ 50,65,45,15,19], BLEU for translation [ 50,66,1,43], and other domains
[61,27,26]. Finally, there has been extensive research on modifying architectures [ 22,59] and
pre-training procedures [70, 36, 49, 60, 53, 14] for improving summarization performance.
3 Method and experiment details
3.1 High-level methodology
Our approach is similar to the one outlined in [ 73], adapted to the batch setting. We start with an
initial policy that is ﬁne-tuned via supervised learning on the desired dataset (in our case, the Reddit
TL;DR summarization dataset). The process (illustrated in Figure 2) then consists of three steps that
can be repeated iteratively.
Step 1: Collect samples from existing policies and send comparisons to humans. For each
Reddit post, we sample summaries from several sources including the current policy, initial policy,
original reference summaries and various baselines. We send a batch of pairs of summaries to our
human evaluators, who are tasked with selecting the best summary of a given Reddit post.
Step 2: Learn a reward model from human comparisons. Given a post and a candidate summary,
we train a reward model to predict the log odds that this summary is the better one, as judged by our
labelers.
Step 3: Optimize a policy against the reward model. We treat the logit output of the reward model
as a reward that we optimize using reinforcement learning, speciﬁcally with the PPO algorithm [ 58].
3
1
Collect human feedback
“j is better than k”
“j is better than k”
A Reddit post is
sampled f
r
om
the Reddit
TL;DR dataset.
V
arious policies
a
r
e used to
sample a set of
summaries.
T
wo summaries
a
r
e selected for
evaluation.
A human judges
which is a better
summary of the
post.
2
T
rain
r
ewa
r
d model
One post with
two summaries
judged by a
human a
r
e fed
to the
r
ewa
r
d
model.
The
r
ewa
r
d
model
calculates a
r
ewa
r
d
r
 for
each summar
y
.
The loss is
calculated based
on the
r
ewa
r
ds
and human label,
and is used to
update the
r
ewa
r
d model.
3
T
rain policy with PPO
A new post is
sampled f
r
om the
dataset.
The
r
ewa
r
d
model calculates
a
r
ewa
r
d for the
summar
y
.
The
r
ewa
r
d is
used to update
the policy via
PPO.
r
r
r
r
π
r
j
loss = log
(σ(r
j
- r
k
))
r
k
The policy
π

generates a
summary for the
post.
r
j
j
k
kFigure 2: Diagram of our human feedback, reward model training, and policy training procedure.
We provide a more thorough description of our procedure, including details of the reward model and
policy training and our quality control process, in the following sections. In practice, rather than
precisely iterating this sequence of three steps, we updated our data collection and training procedures
over the course of the project while accumulating labels (see Appendix C.6 for details).
3.2 Datasets and task
Datasets. We use the TL;DR summarization dataset [ 63], which contains ~3 million posts from
reddit.com across a variety of topics (subreddits), as well summaries of the posts written by the
original poster (TL;DRs). We additionally ﬁlter this dataset (see Appendix A) to ensure quality,
including using a whitelist of subreddits that are understandable to the general population. Crucially,
we also ﬁlter to include only posts where the human-written summaries contain between 24 and
48 tokens, to minimize the potential effect of summary length on quality (see Section 4.1 and
Appendix F). Our ﬁnal ﬁltered dataset contains 123,169 posts, and we hold out ~5% as a validation
set. For the remainder of this paper, we refer to this dataset simply as TL;DR.
We chose the TL;DR dataset over the more commonly used CNN/DM dataset primarily because
very strong performance can be attained on CNN/DM with simple extractive baselines. We ﬁnd in
Section 4.2 that our labelers prefer lead-3 over the CNN/DM reference summaries,5and that the
supervised T5 model [ 49] with low-temperature sampling already surpasses the reference summary
quality, while copying extensively from the article. On the other hand, simple extractive baselines
perform poorly on TL;DR in our human evaluations (see Appendix G.2). Instead of training on
CNN/DM, we study the transfer performance of our human feedback models to CNN/DM after being
trained to summarize Reddit posts.
Task. We deﬁne our ground-truth task as producing a model that generates summaries fewer than
48 tokens long that are as good as possible, according to our judgments. We judge summary quality
by how faithfully the summary conveys the original post to a reader who can only read the summary
and not the post (see Appendix C.5 for further discussion of criteria). Since we have limited capacity
to do comparisons, we hire labelers to do the comparisons for us. We rely on detailed procedures to
ensure high agreement between labelers and us on the task, which we describe in the next section.
5We manually check this result in Appendix E and ﬁnd we generally agree with labeler ratings.
4
[r/dating\_advice ]First date ever, going to the beach. Would like some tips
Hey Reddit! I (20M) would like some tips, because I have my ﬁrst ever date tomorrow (although I’ve had a gf for 3
years, but no actual dating happened), and we’re going to the beach.
I met this girl, we have mutual friends, at a festival a few days ago. We didn’t kiss, but we talked, held
hands, danced a bit. I asked her to go on a date with me, which was super hard as it is the ﬁrst time I’ve asked this
to anybody. What I mean to say is, it’s not like a standard \*ﬁrst\* date because we already spent some time together.
I’m really nervous and excited. I’m going to pick her up tomorrow, we’re cycling to the beach which will
take 30 minutes, and then what? I’m a bit scared. Should I bring something (the weather, although no rain and
sunny, is not super so no swimming), should we do something. I’d like all the tips I can get. Thanks!
Human written reference TL;DR 6.7B supervised model 6.7B human feedback model
First date after 3 years in a relation-
ship, going to the beach, terriﬁed.
What to bring with me, what to do?Going on a date with a girl I met
a few days ago, going to the beach.
What should I bring, what should
we do?Going on my ﬁrst ever date tomor-
row, cycling to the beach. Would
like some tips on what to do and
bring. I’m a bit nervous and excited.
Thanks!
Table 1: Example of post and samples on the TL;DR dataset, chosen to be particularly short. For
random samples (along with posts), see Appendix H and our website.
3.3 Collecting human feedback
Previous work on ﬁne-tuning language models from human feedback [ 73] reported “a mismatch
between the notion of quality we wanted our model to learn, and what the humans labelers actually
evaluated”, leading to model-generated summaries that were high-quality according to the labelers,
but fairly low-quality according to the researchers.
Compared to [ 73], we implement two changes to improve human data quality. First, we transition
entirely to the ofﬂine setting, where we alternate between sending large batches of comparison data6
to our human labelers and re-training our models on the cumulative collected data. Second, we
maintain a hands-on relationship with labelers:7we on-board them with detailed instructions, answer
their questions in a shared chat room, and provide regular feedback on their performance. We train all
labelers to ensure high agreement with our judgments, and continuously monitor labeler-researcher
agreement over the course of the project. See Appendix C.1 and C.5 for details.
As a result of our procedure, we obtained high labeler-researcher agreement: on a subset of compari-
son tasks, labelers agree with researchers 77% 2% of the time, while researchers agree with each
other 73%4% of the time. We provide more analysis of our human data quality in Appendix C.2.
3.4 Models
All of our models are Transformer decoders [ 62] in the style of GPT-3 [ 47,4]. We conduct our human
feedback experiments on models with 1.3 billion (1.3B) and 6.7 billion (6.7B) parameters.
Pretrained models. Similarly to [ 12,47], we start with models pretrained to autoregressively
predict the next token in a large text corpus. As in [ 48,4], we use these models as ‘zero-shot’
baselines by padding the context with examples of high-quality summaries from the dataset. We
provide details on pretraining in Appendix B, and on our zero-shot procedure in Appendix B.2.
Supervised baselines. We next ﬁne-tune these models via supervised learning to predict summaries
from our ﬁltered TL;DR dataset (see Appendix B for details). We use these supervised models to
sample initial summaries for collecting comparisons, to initialize our policy and reward models, and
as baselines for evaluation. In our ﬁnal human evaluations, we use T=0 to sample from all models, as
we found it performed better than higher temperatures or nucleus sampling (see Appendix B.1).
To validate that our supervised models are indeed strong baselines for comparison, we run our
supervised ﬁne-tuning procedure with our 6.7B model on the CNN/DM dataset, and ﬁnd that we
achieve slightly better ROUGE scores than SOTA models [71] from mid-2019 (see Appendix G.4).
6Our decision to collect comparisons rather than Likert scores is supported by recent work, e.g. [37].
7We recruited labelers from a freelancing platform, Upwork, and two labeling services, Scale and Lionbridge.
5
Reward models. To train our reward models, we start from a supervised baseline, as described
above, then add a randomly initialized linear head that outputs a scalar value. We train this model to
predict which summary y2fy0;y1gis better as judged by a human, given a post x. If the summary
preferred by the human is yi, we can write the RM loss as:
loss(r) =E(x;y0;y1;i)D[log((r(x;yi)r(x;y 1i)))]
wherer(x;y)is the scalar output of the reward model for post xand summary ywith parameters ,
andDis the dataset of human judgments. At the end of training, we normalize the reward model
outputs such that the reference summaries from our dataset achieve a mean score of 0.
Human feedback policies. We want to use the reward model trained above to train a policy that
generates higher-quality outputs as judged by humans. We primarily do this using reinforcement
learning, by treating the output of the reward model as a reward for the entire summary that we
maximize with the PPO algorithm [ 58], where each time step is a BPE token.8We initialize our
policy to be the model ﬁne-tuned on Reddit TL;DR. Importantly, we include a term in the reward that
penalizes the KL divergence between the learned RL policy RL
with parameters and this original
supervised model SFT, as previously done in [25]. The full reward Rcan be written as:
R(x;y) =r(x;y)log[RL
(yjx)=SFT(yjx)]
This KL term serves two purposes. First, it acts as an entropy bonus, encouraging the policy to
explore and deterring it from collapsing to a single mode. Second, it ensures the policy doesn’t learn
to produce outputs that are too different from those that the reward model has seen during training.
For the PPO value function, we use a Transformer with completely separate parameters from the
policy. This prevents updates to the value function from partially destroying the pretrained policy
early in training (see ablation in Appendix G.1). We initialize the value function to the parameters of
the reward model. In our experiments, the reward model, policy, and value function are the same size.
4 Results
4.1 Summarizing Reddit posts from human feedback
Policies trained with human feedback are preferred to much larger supervised policies. Our
main results evaluating our human feedback policies on TL;DR are shown in Figure 1. We measure
policy quality as the percentage of summaries generated by that policy that humans prefer over
the reference summaries in the dataset. Our policies trained with human feedback signiﬁcantly
outperform our supervised baselines on this metric, with our 1.3B human feedback model signiﬁcantly
outperforming a supervised model 10 its size (61% versus 43% raw preference score against
reference summaries). Our 6.7B model in turn signiﬁcantly outperforms our 1.3B model, suggesting
that training with human feedback also beneﬁts from scale. Additionally, both of our human feedback
models are judged by humans to be superior to the human demonstrations used in the dataset.
Controlling for summary length. When judging summary quality, summary length is a confound-
ing factor. The target length of a summary is implicitly part of the summarization task; depending on
the desired trade-off between conciseness and coverage, a shorter or longer summary might be better.
Since our models learned to generate longer summaries, length could account for much of our quality
improvements. We ﬁnd that after controlling for length (Appendix F), the preference of our human
feedback models vs. reference summaries drops by ~5%; even so, our 6.7B model summaries are still
preferred to the reference summaries ~65% of the time.
How do our policies improve over the baselines? To better understand the quality of our models’
summaries compared to the reference summaries and those of our supervised baselines, we conduct
an additional analysis where human labelers assess summary quality across four dimensions (or
“axes”) using a 7-point Likert scale [ 38]. Labelers rated summaries for coverage (how much important
information from the original post is covered), accuracy (to what degree the statements in the summary
are stated in the post), coherence (how easy the summary is to read on its own), and overall quality.
8Note that the reward model only gives rewards for entire summaries, and not at intermediate time steps. In
RL terminology, each episode terminates when the policy outputs the EOS token, and the discount factor
= 1.
6
Supervised
transferHuman feedbacktransfer
Pretrain
onlyReference summariesLead-3
SupervisedCNN/DMT5 CNN/DMfinetuning(a)
 (b)
Figure 4: Transfer results on CNN/DM. (a) Overall summary quality on CNN/DM as a function of
model size. Full results across axes shown in Appendix G.2. (b) Overall scores vs. length for the
6.7B TL;DR supervised baseline, the 6.7B TL;DR human feedback model, and T5 ﬁne-tuned on
CNN/DM summaries. At similar summary lengths, our 6.7B TL;DR human feedback model nearly
matches T5 despite never being trained to summarize news articles.
Figure 3: Evaluations of four axes of
summary quality on the TL;DR dataset.The results (Figure 3) indicate that our human feedback
models outperform the supervised baselines across every
dimension of quality, but particularly coverage. Although
our human labelers had a high bar for giving perfect overall
scores, summaries from our 6.7B PPO model achieve a 7/7
overall score 45% of the time (compared to 20% and 23%
for the 6.7B supervised baseline and reference summaries,
respectively).
4.2 Transfer to summarizing news articles
Our human feedback models can also generate excellent
summaries of CNN/DM news articles without any further
training (Figure 4). Our human feedback models signiﬁ-
cantly outperform models trained via supervised learning
on TL;DR and models trained only on pretraining corpora. In fact, our 6.7B human feedback model
performs almost as well as a 6.7B model that was ﬁne-tuned on the CNN/DM reference summaries,
despite generating much shorter summaries.
Since our human feedback models transferred to CNN/DM have little overlap in summary length
distribution with models trained on CNN/DM, with about half as many tokens on average, they are
difﬁcult to compare directly. Thus our evaluations in Figure 4 use a 7-point Likert scale on four
quality dimensions, as in Section 4.1 (see Appendix C.5 for labeler instructions). In Figure 4b we
show the average overall score at different summary lengths, which suggests our human feedback
models would perform even better if they generated longer summaries. Qualitatively, CNN/DM
summaries from our human feedback models are consistently ﬂuent and reasonable representations
of the article; we show examples on our website and in Appendix H.
4.3 Understanding the reward model
What happens as we optimize the reward model? Optimizing against our reward model is
supposed to make our policy align with human preferences. But the reward model isn’t a perfect
representation of our labeler preferences, as it has limited capacity and only sees a small amount of
comparison data from a relatively narrow distribution of summaries. While we can hope our reward
model generalizes to summaries unseen during training, it’s unclear how much one can optimize
against the reward model until it starts giving useless evaluations.
To answer this question, we created a range of policies optimized against an earlier version of our
reward model, with varying degrees of optimization strength, and asked labelers to compare samples
from them to the reference summaries. Figure 5 shows the results for PPO at a range of KL penalty
7
RM prediction
Actual preferenceFigure 5: Preference scores versus degree of
reward model optimization. Optimizing against
the reward model initially improves summaries,
but eventually overﬁts, giving worse summaries.
This ﬁgure uses an earlier version of our reward
model (see rm3 in Appendix C.6). See Appendix
H.2 for samples from the KL 250 model.
Ensemble of humans
Human baseline
64k
32k
16k
8kFigure 6: Reward model performance versus
data size and model size. Doubling amount of
training data leads to a ~1.1% increase in reward
model validation accuracy, whereas doubling
the model size leads to a ~1.8% increase. The
6.7B model trained on all data begins approach-
ing the accuracy of a single human.
coefﬁcients ( ). Under light optimization, the models improve (according to labelers). However, as
we optimize further, true preferences fall off compared to the prediction, and eventually the reward
model becomes anti-correlated with human preferences. Though this is clearly undesirable, we note
that this over-optimization also happens with ROUGE (see [ 45] and Appendix G.3). Similar behavior
has been observed in learned reward functions in the robotics domain [5].
How does reward modeling scale with increasing model and data size? We conduct an ablation
to determine how data quantity and model size affect reward modeling performance. We train 7
reward models ranging from 160M to 13B parameters, on 8k to 64k human comparisons from our
dataset. We ﬁnd that doubling the training data amount leads to a ~1.1% increase in the reward model
validation set accuracy, whereas doubling the model size leads to a ~1.8% increase (Figure 6).
What has the reward model learned? We probe our reward model by evaluating it on several
validation sets. We show the full results in Appendix G.6, and highlight them here. We ﬁnd that our
reward models generalize to evaluating CNN/DM summaries (Appendix G.7), agreeing with labeler
preferences 62.4% and 66.5% of the time (for our 1.3B and 6.7B models, respectively). Our 6.7B
reward model nearly matches the inter-labeler agreement value of 66.9%.
We also ﬁnd that our reward models are sensitive to small but semantically important details in
the summary. We construct an additional validation set by having labelers make minimal edits to
summaries to improve them. Our RMs prefer the edited summaries almost as often (79.4% for 1.3B
and 82.8% for 6.7B) as a separate set of human evaluators (84.1%). Further, when comparing the
reference summaries to perturbed summaries where the participants’ roles are reversed, our models
reliably select the original summary (92.9% of the time for 1.3B, 97.2% for 6.7B). However, our RMs
are biased towards longer summaries: our 6.7B RM prefers improving edits that make the summary
shorter only 62.6% of the time (vs. 76.4% for humans).
4.4 Analyzing automatic metrics for summarization
Evaluation. We study how well various automatic metrics act as predictors for human preferences,
and compare them to our RMs. Speciﬁcally, we examine ROUGE, summary length, amount of
copying from the post,9and log probability under our baseline supervised models. We present a full
matrix of agreement rates between these metrics in Appendix G.7.
We ﬁnd that our learned reward models consistently outperform other metrics, even on the CNN/DM
dataset on which it was never trained. We also ﬁnd that ROUGE fails to track sample quality as our
9We measure copying by computing the longest common subsequence of bigrams with the original Reddit
post or news article, and dividing by the number of bigrams in the summary.
8
Figure 7: Summary quality as a function of metric optimized and amount of optimization, using
best-of-N rejection sampling. We evaluate ROUGE, our main reward models, and an earlier iteration
of the 1.3B model trained on approximately 75% as much data (see Table 11 for details). ROUGE
appears to peak both sooner and at a substantially lower preference rate than all reward models.
Details in Appendix G.3.
models improve. While ROUGE has ~57% agreement with labelers when comparing samples from
our supervised baseline models, this drops to ~50% for samples from our human feedback model.
Similarly, log probability agreement with humans drops to 50% on comparisons between samples
from our human feedback models, while our RMs still perform above chance (62%). Scaling up the
size of the supervised model does not reliably improve log probability’s agreement with labelers.
Optimization. In Figure 7, we show that optimizing ROUGE using a simple optimization scheme
doesn’t consistently increase quality, as has been noted in [ 45]. Optimization against ROUGE peaks
both sooner and at a substantially lower quality rate than optimization against our reward models.
5 Discussion
Limitations. One limitation of our work is the time and cost required to produce our ﬁnal models.
Notably, ﬁne-tuning our 6.7B model with RL required approximately 320 GPU-days. Our data
collection procedure is also expensive compared to prior work — the training set took thousands of
labeler hours and required signiﬁcant researcher time to ensure quality. For this reason, we were
unable to collect baselines such as an equivalent amount of high-quality human demonstrations for
supervised baselines. See D for more discussion. We leave this ablation to future work. Nevertheless,
we believe reward modeling is more likely to scale to tasks where it is extremely skill-intensive or
time-consuming to provide good demonstrations.
Future directions. The methods in this paper could be applied to any task where humans can
compare samples, including dialogue, machine translation, question answering, speech synthesis, and
music generation. We expect this method to be particularly important for generating long samples,
where the distributional shift and degeneracy of maximum likelihood samples can be problematic. It
may be possible to improve sample efﬁciency by training to predict feedback across many tasks [ 42].
We are particularly interested in scaling human feedback to tasks where humans can’t easily evaluate
the quality of model outputs. In this setting, it is particularly challenging to identify whether an ML
system is aligned with the human designer’s intentions. One approach is to train ML systems to help
humans perform the evaluation task quickly and accurately [9].
There is also a rich landscape of human feedback methods beyond binary comparisons that could be
explored for training models [ 28,17,44,64]. For example, we could solicit high-quality demonstra-
tions from labelers, have labelers edit model outputs to make them better, or have labelers provide
explanations for why they preferred one model output over another. All of this feedback could be
leveraged as a signal to train more capable reward models and policies.
9
Broader impacts. The techniques we explore in this paper are generic techniques that could be
used in a wide variety of machine learning applications, for any task where it is feasible for humans
to evaluate the quality of model outputs. Thus, the potential implications are quite broad.
Our research is primarily motivated by the potential positive effects of aligning machine learning
algorithms with the designer’s preferences. Many machine learning applications optimize simple
metrics which are only rough proxies for what the designer intends. This can lead to problems, such
as Youtube recommendations promoting click-bait [ 11]. In the short term, improving techniques for
learning from and optimizing human preferences directly may enable these applications to be more
aligned with human well-being.
In the long term, as machine learning systems become more capable it will likely become increasingly
difﬁcult to ensure that they are behaving safely: the mistakes they make might be more difﬁcult to
spot, and the consequences will be more severe. For instance, writing an inaccurate summary of a
news article is both easy to notice (one simply has to read the original article) and has fairly low
consequences. On the other hand, imitating human driving may be substantially less safe than driving
to optimize human preferences. We believe that the techniques we explore in this paper are promising
steps towards mitigating the risks from such capable systems, and better aligning them with what
humans care about.
Unfortunately, our techniques also enable malicious actors to more easily train models that cause
societal harm. For instance, one could use human feedback to ﬁne-tune a language model to be more
persuasive and manipulate humans’ beliefs, or to induce dependence of humans on the technology, or
to generate large amounts of toxic or hurtful content intended to harm speciﬁc individuals. Avoiding
these outcomes is a signiﬁcant challenge for which there are few obvious solutions.
Large-scale models trained with human feedback could have signiﬁcant impacts on many groups.
Thus, it is important to be careful about how we deﬁne the ‘good’ model behavior that human labelers
will reinforce. Deciding what makes a good summary is fairly straightforward, but doing this for
tasks with more complex objectives, where different humans might disagree on the correct model
behavior, will require signiﬁcant care. In these cases, it is likely not appropriate to use researcher
labels as the ‘gold standard’; rather, individuals from groups impacted by the technology should be
included in the process to deﬁne ‘good’ behavior, and hired as labelers to reinforce this behavior in
the model.
We chose to train on the Reddit TL;DR dataset because the summarization task is signiﬁcantly more
challenging than on CNN/DM. However, since the dataset consists of user-submitted posts with
minimal moderation, they often contain content that is offensive or reﬂects harmful social biases.
This means our models can generate biased or offensive summaries, as they have been trained to
summarize such content. For this reason, we recommend that the potential harms of our models be
thoroughly studied before deploying them in user-facing applications.
Finally, by improving the ability of machine learning algorithms to perform tasks that were previously
only achievable by humans, we are increasing the likelihood of many jobs being automated, potentially
leading to signiﬁcant job loss. Without suitable policies targeted at mitigating the effects of large-scale
unemployment, this could also lead to signiﬁcant societal harm.
Acknowledgements
We’d like to thank Beth Barnes for help with labeler hiring and general encouragement; Geoffrey
Irving for guidance on earlier iterations of the project and inspiring conversations; Ben Mann, Tom
Brown, Nick Ryder, and Melanie Subbiah for training and evaluating our pretrained models; Chris
Hesse, Eric Sigler, Benjamin Chess, Christopher Berner, Clemens Winter, Mateusz Litwin, and many
others for supporting us through computing infrastructure improvements and maintenance; Scott
Gray for writing fast GPU kernels; Arvind Neelakantan and Wojciech Kryscinski for discussions on
how to present the work, experiment design, and what datasets to use; Shan Carter for help designing
the main diagram; Douwe Kiela, Zach Lipton, and Alex Irpan for providing feedback on the paper;
and Gretchen Krueger for co-writing the model card accompanying the paper.
Finally, we’d like to thank all of our contractors for providing the data that was essential for training
the models in this paper, including: Emill Jayson Caypuno, Rachelle Froyalde, Cyra Denura, Alex
Malek, Isik Agil, Reshmi Patel, William Yap, Natalie Silver, Erol Akbaba, Jennifer Brillo, Alexandra
10
Uifalean, Morris Stuttard, Russell Bernandez, Tasmai Dave, Rachel Wallace, Jenny Fletcher, Jian
Ouyang, Justin Dill, Maria Orzek, Megan Niffenegger, William Sells, Emily Mariner, Andrew Seely,
Lychelle Ignacio, Jelena Ostojic, Nhan Tran, Purev Batdelgar, Valentina Kezic, Michelle Wilkerson,
Kelly Guerrero, Heather Scott, Sarah Mulligan, Gabriel Ricafrente, Kara Bell, Gabriel Perez, and
Alfred Lee.
References
[1]D. Bahdanau, P. Brakel, K. Xu, A. Goyal, R. Lowe, J. Pineau, A. Courville, and Y . Bengio. An
actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086 , 2016.
[2]B. T. Bartell, G. W. Cottrell, and R. K. Belew. Automatic combination of multiple ranked
retrieval systems. In SIGIR’94 , pages 173–181. Springer, 1994.
[3]F. Böhm, Y . Gao, C. M. Meyer, O. Shapira, I. Dagan, and I. Gurevych. Better rewards yield
better summaries: Learning to summarise without references. arXiv preprint arXiv:1909.01214 ,
2019.
[4]T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan,
P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child,
A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray,
B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei.
Language models are few-shot learners. 2020.
[5]S. Cabi, S. Gómez Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna,
Y . Aytar, D. Budden, M. Vecerik, et al. Scaling data-driven robotics with reward sketching and
batch reinforcement learning. arXiv , pages arXiv–1909, 2019.
[6]A. T. Chaganty, S. Mussman, and P. Liang. The price of debiasing automatic metrics in natural
language evaluation. arXiv preprint arXiv:1807.02202 , 2018.
[7]W. S. Cho, P. Zhang, Y . Zhang, X. Li, M. Galley, C. Brockett, M. Wang, and J. Gao. Towards
coherent and cohesive long-form text generation. arXiv preprint arXiv:1811.00511 , 2018.
[8]S. Chopra, M. Auli, and A. M. Rush. Abstractive sentence summarization with attentive
recurrent neural networks. In Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies ,
pages 93–98, 2016.
[9]P. Christiano, B. Shlegeris, and D. Amodei. Supervising strong learners by amplifying weak
experts. arXiv preprint arXiv:1810.08575 , 2018.
[10] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement
learning from human preferences. In Advances in Neural Information Processing Systems ,
pages 4299–4307, 2017.
[11] P. Covington, J. Adams, and E. Sargin. Deep neural networks for youtube recommendations. In
Proceedings of the 10th ACM conference on recommender systems , pages 191–198, 2016.
[12] A. M. Dai and Q. V . Le. Semi-supervised sequence learning. In Advances in neural information
processing systems , pages 3079–3087, 2015.
[13] J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith. Fine-tuning
pretrained language models: Weight initializations, data orders, and early stopping. arXiv
preprint arXiv:2002.06305 , 2020.
[14] L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y . Wang, J. Gao, M. Zhou, and H.-W. Hon. Uniﬁed
language model pre-training for natural language understanding and generation. In Advances in
Neural Information Processing Systems , 2019.
[15] Y . Dong, Y . Shen, E. Crawford, H. van Hoof, and J. C. K. Cheung. Banditsum: Extractive
summarization as a contextual bandit. arXiv preprint arXiv:1809.09672 , 2018.
[16] B. Dorr, D. Zajic, and R. Schwartz. Hedge trimmer: A parse-and-trim approach to headline
generation. In Proceedings of the HLT-NAACL 03 on Text summarization workshop-Volume 5 ,
pages 1–8. Association for Computational Linguistics, 2003.
[17] S. Fidler et al. Teaching machines to describe images with natural language feedback. In
Advances in Neural Information Processing Systems , pages 5068–5078, 2017.
11
[18] N. Fuhr. Optimum polynomial retrieval functions based on the probability ranking principle.
ACM Transactions on Information Systems (TOIS) , 7(3):183–204, 1989.
[19] Y . Gao, C. M. Meyer, M. Mesgar, and I. Gurevych. Reward learning for efﬁcient reinforcement
learning in extractive document summarisation. arXiv preprint arXiv:1907.12894 , 2019.
[20] X. Glorot and Y . Bengio. Understanding the difﬁculty of training deep feedforward neural
networks. In Proceedings of the thirteenth international conference on artiﬁcial intelligence
and statistics , pages 249–256, 2010.
[21] B. Hancock, A. Bordes, P.-E. Mazare, and J. Weston. Learning from dialogue after deployment:
Feed yourself, chatbot! arXiv preprint arXiv:1901.05415 , 2019.
[22] K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom.
Teaching machines to read and comprehend. In Advances in neural information processing
systems , pages 1693–1701, 2015.
[23] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y . Choi. The curious case of neural text
degeneration. arXiv preprint arXiv:1904.09751 , 2019.
[24] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human
preferences and demonstrations in atari. In Advances in neural information processing systems ,
pages 8011–8023, 2018.
[25] N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and
R. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in
dialog. arXiv preprint arXiv:1907.00456 , 2019.
[26] N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and D. Eck. Sequence
tutor: Conservative ﬁne-tuning of sequence generation models with kl-control. In International
Conference on Machine Learning , pages 1645–1654. PMLR, 2017.
[27] N. Jaques, S. Gu, R. E. Turner, and D. Eck. Tuning recurrent neural networks with reinforcement
learning. 2017.
[28] H. J. Jeon, S. Milli, and A. D. Dragan. Reward-rational (implicit) choice: A unifying formalism
for reward learning. arXiv preprint arXiv:2002.04833 , 2020.
[29] T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of the eighth
ACM SIGKDD international conference on Knowledge discovery and data mining , pages
133–142, 2002.
[30] T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting click-
through data as implicit feedback. In ACM SIGIR Forum , volume 51, pages 4–11. Acm New
York, NY , USA, 2005.
[31] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980 , 2014.
[32] J. Kreutzer, S. Khadivi, E. Matusov, and S. Riezler. Can neural machine translation be improved
with user feedback? arXiv preprint arXiv:1804.05958 , 2018.
[33] W. Kryscinski, N. S. Keskar, B. McCann, C. Xiong, and R. Socher. Neural text summarization:
A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Nat-
ural Language Processing and the 9th International Joint Conference on Natural Language
Processing (EMNLP-IJCNLP) , pages 540–551, 2019.
[34] C. Lawrence and S. Riezler. Improving a neural semantic parser by counterfactual learning
from human bandit feedback. arXiv preprint arXiv:1805.01252 , 2018.
[35] J. Leike, D. Krueger, T. Everitt, M. Martic, V . Maini, and S. Legg. Scalable agent alignment via
reward modeling: a research direction. arXiv preprint arXiv:1811.07871 , 2018.
[36] M. Lewis, Y . Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V . Stoyanov, and
L. Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language
generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019.
[37] M. Li, J. Weston, and S. Roller. Acute-eval: Improved dialogue evaluation with optimized
questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087 , 2019.
[38] R. Likert. A technique for the measurement of attitudes. Archives of psychology , 1932.
12
[39] C.-Y . Lin and F. J. Och. Automatic evaluation of machine translation quality using longest
common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting on
Association for Computational Linguistics , page 605. Association for Computational Linguistics,
2004.
[40] T.-Y . Liu. Learning to rank for information retrieval . Springer Science & Business Media,
2011.
[41] J. Maynez, S. Narayan, B. Bohnet, and R. McDonald. On faithfulness and factuality in
abstractive summarization, 2020.
[42] B. McCann, N. S. Keskar, C. Xiong, and R. Socher. The natural language decathlon: Multitask
learning as question answering. arXiv preprint arXiv:1806.08730 , 2018.
[43] K. Nguyen, H. Daumé III, and J. Boyd-Graber. Reinforcement learning for bandit neural
machine translation with simulated human feedback. arXiv preprint arXiv:1707.07402 , 2017.
[44] T. Niu and M. Bansal. Polite dialogue generation without parallel data. Transactions of the
Association for Computational Linguistics , 6:373–389, 2018.
[45] R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization.
arXiv preprint arXiv:1705.04304 , 2017.
[46] E. Perez, S. Karamcheti, R. Fergus, J. Weston, D. Kiela, and K. Cho. Finding generalizable
evidence by learning to convince q&a models. arXiv preprint arXiv:1909.05863 , 2019.
[47] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language under-
standing by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-
assets/researchcovers/languageunsupervised/language understanding paper. pdf , 2018.
[48] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are
unsupervised multitask learners. OpenAI Blog , 1(8):9, 2019.
[49] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu.
Exploring the limits of transfer learning with a uniﬁed text-to-text transformer. arXiv preprint
arXiv:1910.10683 , 2019.
[50] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural
networks. arXiv preprint arXiv:1511.06732 , 2015.
[51] D. R. Reddy et al. Speech understanding systems: A summary of results of the ﬁve-year
research effort. department of computer science, 1977.
[52] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction
to no-regret online learning. In Proceedings of the fourteenth international conference on
artiﬁcial intelligence and statistics , pages 627–635, 2011.
[53] S. Rothe, S. Narayan, and A. Severyn. Leveraging pre-trained checkpoints for sequence
generation tasks. Transactions of the Association for Computational Linguistics , 2020.
[54] A. M. Rush, S. Chopra, and J. Weston. A neural attention model for abstractive sentence
summarization. arXiv preprint arXiv:1509.00685 , 2015.
[55] N. Schluter. The limits of automatic summarisation according to rouge. In Proceedings of the
15th Conference of the European Chapter of the Association for Computational Linguistics:
Volume 2, Short Papers , pages 41–45, 2017.
[56] F. Schmidt. Generalization in generation: A closer look at exposure bias. arXiv preprint
arXiv:1910.00292 , 2019.
[57] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous
control using generalized advantage estimation. In Proceedings of the International Conference
on Learning Representations (ICLR) , 2016.
[58] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization
algorithms. arXiv preprint arXiv:1707.06347 , 2017.
[59] A. See, P. J. Liu, and C. D. Manning. Get to the point: Summarization with pointer-generator
networks. arXiv preprint arXiv:1704.04368 , 2017.
[60] K. Song, X. Tan, T. Qin, J. Lu, and T.-Y . Liu. Mass: Masked sequence to sequence pre-training
for language generation. arXiv preprint arXiv:1905.02450 , 2019.
13
[61] P. Tambwekar, M. Dhuliawala, A. Mehta, L. J. Martin, B. Harrison, and M. O. Riedl. Con-
trollable neural story generation via reinforcement learning. arXiv preprint arXiv:1809.10736 ,
2018.
[62] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and
I. Polosukhin. Attention is all you need. In Advances in neural information processing systems ,
pages 5998–6008, 2017.
[63] M. Völske, M. Potthast, S. Syed, and B. Stein. Tl; dr: Mining reddit to learn automatic
summarization. In Proceedings of the Workshop on New Frontiers in Summarization , pages
59–63, 2017.
[64] S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with
unlikelihood training. arXiv preprint arXiv:1908.04319 , 2019.
[65] Y . Wu and B. Hu. Learning to extract coherent summary via deep reinforcement learning. In
Thirty-Second AAAI Conference on Artiﬁcial Intelligence , 2018.
[66] Y . Wu, M. Schuster, Z. Chen, Q. V . Le, M. Norouzi, W. Macherey, M. Krikun, Y . Cao, Q. Gao,
K. Macherey, et al. Google’s neural machine translation system: Bridging the gap between
human and machine translation. arXiv preprint arXiv:1609.08144 , 2016.
[67] Y . Yan, W. Qi, Y . Gong, D. Liu, N. Duan, J. Chen, R. Zhang, and M. Zhou. Prophetnet: Pre-
dicting future n-gram for sequence-to-sequence pre-training. arXiv preprint arXiv:2001.04063 ,
2020.
[68] S. Yi, R. Goel, C. Khatri, A. Cervone, T. Chung, B. Hedayatnia, A. Venkatesh, R. Gabriel,
and D. Hakkani-Tur. Towards coherent and engaging spoken dialog response generation using
automatic conversation evaluators. arXiv preprint arXiv:1904.13015 , 2019.
[69] H. Zhang, D. Duckworth, D. Ippolito, and A. Neelakantan. Trading off diversity and quality in
natural language generation. arXiv preprint arXiv:2004.10450 , 2020.
[70] J. Zhang, Y . Zhao, M. Saleh, and P. J. Liu. Pegasus: Pre-training with extracted gap-sentences
for abstractive summarization. arXiv preprint arXiv:1912.08777 , 2019.
[71] Y . Zhang, D. Li, Y . Wang, Y . Fang, and W. Xiao. Abstract text summarization with a convolu-
tional seq2seq model. Applied Sciences , 9(8):1665, 2019.
[72] W. Zhou and K. Xu. Learning to compare for better training and evaluation of open domain
natural language generation models. arXiv preprint arXiv:2002.05058 , 2020.
[73] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv-
ing. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 ,
2019.
14

Nisan Stiennon
Long Ouyang
Jeffrey Wu
Daniel Ziegler
Ryan Lowe
Chelsea Voss
Alec Radford
Dario Amodei
Paul F. Christiano


Reinforcement Learning in Newcomblike
Environments
James Bell
The Alan Turing Institute
London, UK
jbell@posteo.comLinda Linsefors
Independent Researcher
linda.linsefors@gmail.com
Caspar Oesterheld
Department of Computer Science
Duke University
Durham, NC, USA
caspar.oesterheld@duke.eduJoar Skalse
Department of Computer Science
University of Oxford
Oxford, UK
joar.skalse@cs.ox.ac.uk
Abstract
Newcomblike decision problems have been studied extensively in the decision
theory literature, but they have so far been largely absent in the reinforcement
learning literature. In this paper we study value-based reinforcement learning
algorithms in the Newcomblike setting, and answer some of the fundamental
theoretical questions about the behaviour of such algorithms in these environments.
We show that a value-based reinforcement learning agent cannot converge to a
policy that is not ratiﬁable , i.e., does not only choose actions that are optimal given
that policy. This gives us a powerful tool for reasoning about the limit behaviour
of agents – for example, it lets us show that there are Newcomblike environments
in which a reinforcement learning agent cannot converge to any optimal policy.
We show that a ratiﬁable policy always exists in our setting, but that there are
cases in which a reinforcement learning agent normally cannot converge to it (and
hence cannot converge at all). We also prove several results about the possible limit
behaviours of agents in cases where they do not converge to any policy.
1 Introduction
In this paper, we study decision scenarios in which outcomes depend not only on the choices made
and physically implemented, but also depend directly on the agent’s policy. As an example, consider
an autonomous vehicle (A V) whose goal it is to arrive at target destinations quickly while minimising
the probability of collisions. In practice, A Vs are careful drivers. It is easy to imagine an experiment
(or learning process) that might support careful driving: on each day, let the A V decide at random
between a careful and a more aggressive style of driving; other drivers on the road are unaware of
today’s chosen driving style and therefore behave the same around the A V on both types of days.
Presumably the decrease in accidents on careful days outweighs the increase in travel time.
However, imagine now that a type of A V was widely deployed. Then many of the drivers with whom
the A Vs interact on the road would know a lot about how these A Vs behave (e.g., from reading about
A Vs, or from having interacted with other A Vs of the same type in the past). In particular, if the other
drivers know that the A Vs rarely take risks, they might (whether rationally or irrationally) cut them
off more, not give them right of way, etc. relative to the above experiment. Indeed, this phenomenon
– human drivers bullying timid A Vs – has been reported in the real world (Condliffe, 2016; Liu et al.,
2020; cf. Cooper et al., 2019). As a result, the travel times of an A V are much longer if it always
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
follows (and is known to follow) a careful driving policy. Moreover, some of the safety beneﬁts of
careful choices disappear if the A V adopts a careful policy.
To comfortably model situations such as this one, we introduce Newcomblike decision processes
(NDPs). The name derives from Newcomb’s problem (Nozick, 1969), described in the next subsec-
tion, and similar problems that have been studied in the decision-theoretic literature. NDPs are a
generalisation of Markov decision processes wherein the transition probabilities and rewards depend
not only on the agent’s action, but also directly on the agent’s policy. Thus, for example, how
aggressively other cars move depends on the timidness of the A V’s policy. We describe NDPs in more
detail in Sect. 1.1. Importantly, the NDP model does not assume other agents in the environment
to respond rationally to the agent’s policy. Thus, some NDPs cannot comfortably be modelled as
games. (See Sect. 5.1 for a more detailed discussion of the relation between NDPs and game-theoretic
models.)
We believe that Newcomblike dynamics are commonplace when AI systems interact with other
(human or artiﬁcial) agents (Cavalcanti, 2010, Sect. 5; Oesterheld, 2019, Sect. 1; Conitzer, 2019).
The deployment of A Vs is rife with such dynamics. Besides aggressiveness, it might matter whether
a policy is simple and thus predictable to humans, for instance. Another real-world scenario is that of
recommendation systems: most readers of this paper have some idea of how these systems work and
make choices based on it. (“I would like to watch this cat video, but if I do, my recommendations
will soon be full of them.”) Thus, the success of a particular recommendation depends not only on
the recommendation itself, but also on the recommendation system’s policy.
We are interested in learning to play NDPs. More speciﬁcally, we study the behaviour of value-based,
model-free RL agents, who maintain a Q-function that assigns values to state–action pairs. We deﬁne
these in more detail in Sect. 1.2. As we will see, such agents do not in general learn optimal policies
in NDPs (as they do in MDPs). Nevertheless, we believe that studying them is an important ﬁrst step
in developing practical learning algorithms for NDPs due to the combination of the following points.
A)For illustrative purposes, the examples we discuss throughout this paper are simple and emphasise
the dependence of the environment on the policy. However, we think that most real-world
scenarios are only partially Newcomblike. For example, most of the A V’s environment changes
only in response to an A V’s actions and does not directly depend on the A V’s policy.
B)Value-based reinforcement learning algorithms are very well developed. In contrast, we would
have to develop specialised learning algorithms for general NDPs from scratch.
C)As our results will show, in some situations and when set up correctly (e.g., in terms of learning
rates) value-based learning converges to optimal policies, or at least to reasonable policies,
even under Newcomblike dynamics. For example, in a game of rock-paper-scissors against an
opponent who knows the agent’s policy, some value-based learning agents learn the optimal
policy of mixing uniformly.
In light of A–C, we think that the most realistic paths to developing learning algorithms for real-world
scenarios with Newcomblike dynamics will involve value-based RL. Speciﬁcally, one avenue toward
realistic algorithms is to develop extensions of value-based RL and can detect and correct failures
that might arise in Newcomblike dynamics. For that, we need to understand how value-based RL
behaves in NDPs.
Contributions In Sect. 2 we demonstrate that value-based RL algorithms can only converge to a
policy that is ratiﬁable – that is, to a policy ⇡for which all actions taken by ⇡have optimal expected
reward when following ⇡. In Sect. 3, we discuss the convergence properties of agents in Newcomblike
situations, and show that there are cases where value-based agents must fail to converge. The action
frequencies might converge, even when the policies do not. In Sect. 4, we establish some conditions
on any action frequency that an agent could converge to. We also show that there are decision
problems and agents where even the action frequencies do not converge.
1.1 Newcomblike Decision Processes
ANewcomblike decision process (NDP) is a tuple hS, A, T, R,  iwhere Sis a ﬁnite set of states ;
Ais a ﬁnite set of actions ;T:S⇥A⇥(S A) Sis a nondeterministic transition function ;
R:S⇥A⇥S⇥(S A) Ris a nondeterministic reward function , which we assume to be
bounded; and 2[0,1)is adiscount factor .
2
A policy ⇡:S Ais a function that nondeterministically maps states to actions. We use ⇡(a|s)to
denote the probability of taking action ain state swhile following the policy ⇡.TandRare functions
from states, actions, and policies. In other words, they allow the outcome of a decision to depend on
the distributions from which the agent draws its actions, rather than just the state and the action that
is in fact taken. Also note that T(s, a, ⇡ )andR(s, a, s0,⇡)are deﬁned even if ⇡(a|s)=0 . We say
that an NDP is a bandit NDP if it has only one state. We will sometimes use R(s, a, ⇡ )as a shorthand
forR(s, a, T (s, a, ⇡ ),⇡), and we will sometimes omit the state from T,R, and ⇡for bandit NDPs.
Moreover, we normally let =0for bandit NDPs.
Consider a distribution over initial states for an agent, and let ⇡be its policy, let xtbe the sequence
of states it visits and atthe sequence of actions it takes. We say ⇡is optimal for that distribution if it
maximises E[P1
i=0iR(xi,ai,ai+1,⇡)]. Note that unlike in the MDP case the optimal policy does
depend on the initial distribution, however this isn’t relevant in the bandit case.
As an example consider the eponymous Newcomb’s Problem.
Newcomb’s Problem (Nozick, 1969): There are two boxes in front of you; one opaque box, and one
transparent box. You can see that the transparent box contains $1,000. You can choose to either take
only the opaque box, or to take both boxes. The boxes have been placed in this room by an agent
who can predict your policy; if he believes that you will take only the opaque box then he has put
$1,000,000 in the opaque box, but if he believes that you will take both boxes then he has left the
opaque box empty. Do you take one box, or two?
A version of Newcomb’s Problem can be formalised as the following bandit NDP: S={s},
A={a1,a2},
R(a1,⇡)=⇢
0 w.p. ⇡(a2)
10 w.p. ⇡(a1)andR(a2,⇡)=⇢
5 w.p. ⇡(a2)
15 w.p. ⇡(a1),
where “w.p.” is short for “with probability”. The key feature of this NDP is that, for any ﬁxed policy,
a2(“two-boxing”) yields a higher reward than a1(“one-boxing”). But the expected reward of a policy
increases in ⇡(a1)s.t. the optimal policy is to always play a1.1We can view Newcomb’s problem as
a simple version of the A V dynamic described in the introduction, where a2is a driving action that
allows other drivers to cut the A V off at no risk.
We say that an NDP is continuous ifTandRare continuous in the policy. In this paper we work
mainly with continuous NDPs. This is in part because it is technically convenient, and in part because
we believe that continuity is satisﬁed in many realistic cases.2
1.2 Reinforcement Learning Agents
We consider value-based reinforcement learning agents. Such agents have two main components; a
Q-function S⇥A!Rthat predicts the expected future discounted reward conditional on taking
a particular action in a particular state, and a bandit algorithm that is used to select actions in each
state based on the Q-function. Given a policy ⇡, we use q⇡(a|s)to denote the (true) expected
future discounted reward conditional on taking action ain state swhile following the policy ⇡(and
conditional on all subsequent actions being chosen by ⇡). A model-free agent will update Qover time
to make it converge to q⇡when following ⇡. IfQis represented as a lookup table, the agent is said
to be tabular . If the state space is large, it is common to instead approximate q⇡(with e.g. a neural
network). For simplicity, we focus mostly on tabular agents. However, some of our results (Theorems
2 and 5) only assume that Qconverges to q⇡(for some q⇡) and therefore apply immediately to
non-tabular agents, as long as the function approximator for q⇡converges to the same q⇡.
1In most versions of Newcomb’s Problem, the predictor directly predicts the agent’s action with some ﬁxed
accuracy, and the agent is unable to randomise in a way that is unpredictable to the environment. This version of
the problem can be modelled as a regular MDP. However, we believe that our version is more realistic in the
context of AI. After all, AIs can at least act pseudo-randomly, while the distribution according to which they
choose is predictable if e.g. their source code is known.
2For example, even if the environment has direct access to the source code of the agent, it may in general not
be feasible to extract the exact action probabilities from the code. However, it is always possible to estimate the
action probabilities by sampling. If this is done then TandRwill depend continuously on the policy.
3
TheQ-values can be updated in different ways. One method is to use the update rule
Qt+1(at|st) (1↵t(st,at))Qt(at|st)+↵t(st,at)(rt+max
aQt(a|st+1)),
where atis the action taken at time t,stis the state visited at time t,rtis the reward obtained at time
t, and ↵t(s, a)is a learning rate. This update rule is known as Q-learning (Watkins, 1986). Other
widely used update rules include SARSA (Rummery and Niranjan, 1994) and Expected SARSA (van
Seijen et al., 2009). For the purposes of this paper it will not matter signiﬁcantly how the Q-values
are computed, as long as it is the case that if an agent converges to a policy ⇡in some NDP and
explores inﬁnitely often then Qconverges to q⇡. We will later see that this is the case for Q-learning,
SARSA, and Expected SARSA in continuous NDPs.
There are also several different bandit algorithms. Two types of agents that are widely used in practice
and that we will refer to throughout the paper are softmax agents and✏-Greedy agents . The policy of
a softmax agent with a sequence of temperatures t2R+is given by:
⇡t(a|s)=exp( Qt(a|s)/t)P
a02Aexp( Qt(a0|s)/t).
Unless otherwise stated we assume that t!0. The policy of an ✏-Greedy agent with a sequence
of exploration probabilities ✏t2[0,1]is⇡t(a|s)=1 ✏tifa= arg max a0Qt(a0|s)and
⇡t(a|s)=✏t/(|A|1)otherwise. Unless otherwise stated we assume that ✏t!0. We assume that
✏-Greedy breaks ties for argmax, so that there is always some a2Asuch that ⇡(a|s)=1 ✏t.W e
say that an agent is greedy in the limit if the probability that the agent takes an action that maximises
Qconverges to 1, and we say that it explores inﬁnitely often if it takes every action in every state
inﬁnitely many times.
1.3 Some Initial Observations
We here make three simple observations about NDPs that we will use to prove and understand the
results throughout this paper. First, a continuous NDP always has, for each possible distribution over
initial states, a policy ⇡that maximises the expected discounted reward E[R|⇡], since E[R|⇡]
exists and is continuous in ⇡, and since the set of possible policies is a compact set. Also note that an
NDP in which TorRis discontinuous may not have any such policy.
Second, whereas all MDPs have a deterministic optimal policy, in some NDPs all optimal policies
randomise. To see this we introduce another example we will look at in this paper.
Death in Damascus (Gibbard and Harper, 1976): Death will come for you tomorrow. You can
choose to stay in Damascus (where you are currently) or you can ﬂee to Aleppo. If you are in the
same city as Death tomorrow, you will die. Death has already decided which city he will go to –
however, he can predict your policy, and has decided to go to the city where he believes that you will
be tomorrow. Do you stay in Damascus, or ﬂee to Aleppo?
We formalise this as the bandit NDP S={s},A={aDamascus ,aAleppo }, and
R(aDamascus ,⇡)=⇢
0 w.p. ⇡(aDamascus )
10 w.p. ⇡(aAleppo )andR(aAleppo ,⇡)=⇢
10 w.p. ⇡(aDamascus )
0 w.p. ⇡(aAleppo ),
where “w.p.” is again short for “with probability”. In this NDP, randomising uniformly between
aDamascus andaAleppo is the unique optimal policy and in particular outperforms both deterministic
policies.
Note also that the Bellman optimality equation does not hold for NDPs. Even in Newcomb’s Problem,
as described above, Bellman’s optimality equation is not satisﬁed by the optimal policy.
2 Ratiﬁability
If an agent in the limit only takes the actions with the highest Q-values and it converges to some policy
⇡1, then it is clear that, for a given state, all actions in the support of ⇡1must have equal expected
utility given ⇡1. Otherwise, the Q-values would eventually reﬂect the differences in expected utility
and the agent would move away from ⇡1. Similarly, if the algorithm explores sufﬁciently often, the
4
actions that are taken with limit probability 0cannot be better given ⇡1than those taken by ⇡1.
After all, if they were better, the agent would have eventually ﬁgured this out and assigned them large
probability.
This condition on ⇡1resembles a well-known doctrine in philosophical decision theory: ratiﬁca-
tionism (see Weirich, 2016, Sect. 3.6, for an overview). One form of ratiﬁcationism is based on a
distinction between a decision – what the agent chooses – and the actthat is selected by that decision.
Very roughly, ratiﬁcationism then states that a decision is rational only if the acts it selects have the
highest expected utility given the decision. Concepts of causality are often invoked to formalise the
difference between the decision, the act, and their respective consequences. Our setup, however, has
such a differentiation built in: we will view the policy as the “decision” and the action sampled from
it as the “act”.
2.1 Strong Ratiﬁability
As hinted earlier, slightly different versions of the concept of ratiﬁability are relevant depending on
how much exploration a learning algorithm guarantees. We start with the stronger version, which
more closely resembles what decision theorists mean when they speak about ratiﬁability.
Deﬁnition 1. LetM✓Sbe a set of states. A policy ⇡is strongly ratiﬁable on Mifsupp( ⇡(·|
s))✓arg maxa2Aq⇡(a|s)for all s2M.
In Newcomb’s Problem the only strongly ratiﬁable policy is to play a2with probability 1. In Death in
Damascus, only the optimal policy (mixing uniformly) is strongly ratiﬁable. There can also be several
strongly ratiﬁable policies. For example, if you play the Coordination Game of Table 1 against an
opponent who samples his action from the same policy as you then there are three strongly ratiﬁable
policies; to select action awith probability 1, to select action bwith probability 1, and to select a
with probability 1/3andbwith probability 2/3.
Theorem 2. LetAbe a model-free reinforcement learning agent, and let ⇡tandQtbeA’s policy
andQ-function at time t. Let Asatisfy the following in a given NDP:
•Ais greedy in the limit, i.e. for all >0,P(Qt(⇡t(s))max aQt(a|s))!0ast!1.
•A’sQ-values are accurate in the limit, i.e. if ⇡t!⇡1ast!1, then Qt!q⇡1ast!1.
Then if A’s policy converges to ⇡1then ⇡1is strongly ratiﬁable on the states that are visited
inﬁnitely many times.
a b
a2,2 0,0
b0,0 1,1
Table 1: The Coordination GameIn Appendix A we show that the Q-values of a tabular
agent are accurate in the limit in any continuous NDP if
the agent updates its Q-values with SARSA, Expected
SARSA, or Q-learning, given that the agent explores in-
ﬁnitely often and uses appropriate learning rates. Since
we would expect most well-designed agents to have accu-
rateQ-values in the limit, Theorem 2 should apply very
broadly. Using Kakutani’s ﬁxed-point theorem, it can be shown that every continuous NDP has a
ratiﬁable policy.
Theorem 3. Every continuous NDP has a strongly ratiﬁable policy.
Of course, the fact that a ratiﬁable policy always exists does not necessarily mean that a reinforcement
learning agent must converge to it – we will consider the question of whether or not this is the case in
Sect. 3. It is also worth noting that a discontinuous NDP may not have any strongly ratiﬁable policy.
It is a topic of ongoing discussion among philosophical decision theorists whether (strong) ratiﬁability
should be considered a normative principle of rationality, see Weirich (2016, Sect. 3.6) for details. In
general, the policy ⇡that maximises E[R|⇡]may or may not be ratiﬁable, as shown by Death in
Damascus and Newcomb’s problem, respectively.
There is a correspondence between ratiﬁcationism and many game-theoretic concepts. For example,
if you are playing a zero-sum game against an opponent who can see your policy and plays some
distribution over best responses to it then ⇡can only be ratiﬁable if it is a maximin strategy. To give
another example, if you are playing a symmetric game against an opponent who follows the same
policy as you then ⇡is ratiﬁable if and only if (⇡,⇡)is a Nash equilibrium. Joyce and Gibbard (1998,
Sect. 5) discuss the relation in more detail.
5
2.2 Weak Ratiﬁability
We now show that even without inﬁnite exploration, ⇡1must still satisfy a weaker notion of
ratiﬁability.
Deﬁnition 4. LetM✓Sbe a set of states. A policy ⇡is weakly ratiﬁable on Mifq⇡(a|s)is
constant across a2supp( ⇡(s))for all s2M.
What makes this a weak version of ratiﬁability is that it does not put any requirements on the
expected utility of actions that ⇡does not take, it merely says that all actions that ⇡takes with
positive probability must have the same (actual) q-value. As a special case, this means that all
deterministic policies are weakly ratiﬁable. This includes one-boxing in Newcomb’s problem.
Nonetheless, there are bandit NDPs in which the optimal policy is not even weakly ratiﬁable. For
example, consider an NDP with actions a1,a2, where R(a1,⇡)= 100( ⇡(a1)1/2)2+1 and
R(a2,⇡)=100( ⇡(a1)1/2)2. The optimal policy mixes close to uniformly ( ⇡(a1)= 101/200),
but this is not weakly ratiﬁable, because R(a1,⇡)>R(a2,⇡).
Theorem 5. Same conditions as Theorem 2, but where A’sQ-values are only required to be accurate
in the limit for state-action pairs that Avisits inﬁnitely many times. Then ⇡1is weakly ratiﬁable on
the set of states that are visited inﬁnitely many times.
3 Non-Convergence of Policies
We have shown that most reinforcement learning algorithms can only converge to (strongly) ratiﬁable
policies. We now consider the question of whether they always converge to a policy at all. We ﬁnd
that this is not the case.
3.1 Theoretical Results
From Theorem 2 it follows that in e.g. Death in Damascus an ✏-Greedy agent who explores inﬁnitely
often cannot converge to any policy. After all, the only strongly ratiﬁable policy (and thus limit
policy) is to mix uniformly and an ✏-Greedy agent never mixes uniformly.
Perhaps more surprisingly, there are also NDPs in which a (slow-cooling) softmax agent cannot
converge to any policy. As an example, consider a bandit NDP with three actions a1,a2,a3, and
where the rewards R(ai,⇡)have expectations
⇡(ai+1)+4 ·133·⇡(ai)
[8j:⇡(aj)1/4]Y
j(⇡(aj)1/4). (1)
Fori=3, we here let ai+1=a1. We also require that the rewards are stochastic with a ﬁnite set of
outcomes such that the empirical Q-values are never exactly equal between different actions. We
call this the Repellor Problem . It has only one strongly ratiﬁable policy (mixing uniformly), but – as
illustrated by Figure 1 – when the current policy mixes close to uniformly, the softmax agent learns
(in expectation) to play less uniformly.
Theorem 6. LetAbe an agent that plays the Repellor Problem, explores inﬁnitely often, and updates
itsQ-values with a learning rate ↵tthat is constant across actions, and let ⇡tandQtbeA’s policy
andQ-function at time t. Assume also that for j6=i, if⇡t(ai),⇡t(aj)both converge to positive
values, then
⇡t(ai)⇡t(aj)
Qt(ai)Qt(aj)!
a.s.1 (2)
ast!1. Then ⇡talmost surely does not converge.
Line 2 is satisﬁed, for example, for softmax agents with tconverging to 0. Recall also that e.g.
Q-learning and SARSA are equivalent for bandit NDPs (if =0).
6
3.2 Empirical Results
Figure 1: The triangle shows the space of possible
policies in the Repellor Problem, parameterised
by the probability they assign to each of the three
actions. Plotted against this space is the expected
direction in which a softmax agent would change
its policy if playing a particular policy.Empirically, softmax agents converge (to
strongly ratiﬁable policies) in many NDPs, pro-
vided that the temperature decreases sufﬁciently
slowly. To illustrate this we will use Aymmet-
ric Death in Damascus , a version of Death in
Damascus wherein the rewards of aAleppo are
changed to be 5(instead of 0) with probabil-
ity⇡(aAleppo )and (as before) 10with the re-
maining probability. This NDP has only one
(strongly) ratiﬁable policy, namely to go to
Aleppo with probability 2/3and Damascus with
probability 1/3. This is also the optimal policy.
We use this asymmetric version to make it easier
to distinguish between convergence to the ratiﬁ-
able policy and the default of uniform mixing at
high temperatures. Figure 2 shows the probabil-
ity of converging to this policy with a softmax
agent and a plot of the policy on one run. We can
see that this agent reliably converges provided
that the cooling is sufﬁciently slow.
Figure 2: The left ﬁgure plots the probability of softmax converging in Asymmetric Death in
Damascus given n=n↵against ↵. More accurately it is a plot of the fraction of runs which
assigned a Q-value of at least 5.5 to the action of going to Aleppo after 5000 iterations. These are
empirical probabilities from 20,000 runs for every ↵that is a multiple of 0.025, and 510,000 runs for
each ↵that is a multiple of 0.005 between 0.5 and 0.55. Notice the “kink” at ↵=0.5. Based on our
experiments, this kink is not an artefact and shows up reliably in this kind of graph. The right-hand
ﬁgure shows how the action probabilities evolve over time for a single run (chosen to converge to the
mixed strategy) for ↵=0.3.
However, there are also fairly simple games in which it seems like softmax agents cannot converge.
Consider Loss-Averse Rock-Paper-Scissors (LARPS), the problem of playing Rock-Paper-Scissors
against an opponent that selects each action with the same probability as you, and where you assign
utility 1 to a win, 0 to a draw, and -10 to a loss. We conjecture that slow-cooling softmax agents do
not converge in LARPS. We have unfortunately not been able to prove this formally, but Figure 3
presents some empirical data which corroborates the hypothesis.
4 Convergence of Action Frequencies
We have seen that there are NDPs in which some reinforcement learning algorithms cannot converge
to any policy. But if they do not converge to any policy, what does their limit behaviour look like?
We now examine whether these algorithms converge to taking each action with some limit frequency ,
and what sorts of frequencies they can converge to.
7
4.1 Possible Frequencies in the Bandit Case
Figure 3: This ﬁgure shows ﬁve runs of a softmax
agent in LARPS, and plots ⇡(arock)against the total
number of episodes played. The agent’s Q-values
are the historical mean rewards for each action, and
t=1/logt.In this section we establish a number of con-
ditions that must be satisﬁed by any limit
action frequency of a value-based agent. We
consider agents that converge to determinis-
tic policies (such as ✏-Greedy agents), and
we limit our analysis to the bandit case (with
=0).
LetP⌃
t:A![0,1]be the frequency with
which each action in Ais taken in the ﬁrst
tsteps (for some agent and some bandit
NDP). Note that P⌃
tis a random variable.
By the law of large numbers, P⌃
t(a)
1/tPt
i=0⇡i(a)converges to 0almost surely
ast!1. Let ⇡abe the policy that takes
action awith probability 1, and let qa=q⇡a.
Theorem 7. Assume that there is some sequence of random variables (✏t0)ts.t.✏t!
t!1a.s.0and
for all t2Nit isX
a⇤2arg maxaQt(a)⇡t(a⇤)1✏t. (3)
LetP⌃
t!p⌃with positive probability as t!1. Then across all actions a2supp( p⌃),qa(a)is
constant.
That is, the actions played with positive limit frequency must all be equally good when played
deterministically. This condition is vaguely analogous to weak ratiﬁability, and is proven in roughly
the same way as Theorem 2.
Theorem 8. Same assumptions as Theorem 7. If |supp( p⌃)|>1then for all a2supp( p⌃)there
exists a02As.t.qa(a0)qa(a).
This condition is an instability condition. Say that multiple actions are taken with nonzero limit
frequency, and that action ahas the highest Q-value at time t. Then for other actions to be played
with positive limit frequency, other actions must at some point be believed to be optimal again (since
the probability of exploration goes to zero). Hence they cannot all be worse when explored while
mainly playing a, since acould otherwise be played forever.
Theorem 9. Same assumptions as Theorem 7. Let Ube the Q-value qa(a)which (by Theorem
7) is constant across a2supp( p⌃). For any a02Asupp( p⌃)that is played inﬁnitely often,
let frequency 1 of the exploratory plays of a0happen when playing a policy near elements of
{⇡a|a2supp( p⌃)}. Then either there exists a2supp( p⌃)such that qa(a0)U; orqa0(a0)<U.
Theorem 9 describes what circumstances are needed for an actions a0to be played with limit frequency
zero. One possibility is that exploration is done only ﬁnitely many times (in which case bad luck
could lead to low Q-values). A second possibility is that the exploration mechanism is “rigged” so
thata0is mostly played when playing policies outside the proximity of {⇡a|a2supp( p⌃)}. In
this case the utility of a0under some zero-limit-frequency policy might lead to low Q-values. If
exploration of a0is spread out more naturally then all but frequency zero of that exploration will
happen near elements of {⇡a|a2supp( p⌃)}. In this case, the only reason for a0to be played with
zero frequency is that exploring a0near some of the elements of {⇡a|a2supp( p⌃)}makes a0look
poor.
4.2 When is Frequency Convergence Possible?
We believe there are NDPs in which an ✏-Greedy agent cannot converge to any limit action frequency.
Speciﬁcally, we believe that LARPS is such an example. Figure 4a shows the directions in which the
frequencies of different actions evolve. The graph seems to have no attractor and hence we believe an
✏-Greedy agent cannot converge to any limit action frequency in this NDP. We have not been able to
rigorously prove this. However, experiments seem to conﬁrm this hypothesis. Figure 4b depicts ﬁve
8
(a) This ﬁgure plots the dynamics of LARPS for
an✏-Greedy agent. Each point represents a triplet
(fR,fS,fP), where fRdenotes the fraction of past time
steps at which aRwas estimated to be the best action,
and similarly for fS,fP. Plotted against this space is
the expected direction in which the frequencies will
change. For instance, if in the past aR, was mostly
played, then aPwill have the highest empirical Q-
values and will therefore be played more in the future.
(b) This ﬁgure shows ﬁve runs of an ✏-Greedy agent
in LARPS, and plots the proportion of past episodes
in which the agent played “rock” against the total
number of episodes played. The agent’s Q-values are
the historical mean rewards for each action, and its
✏-value is 0.01.
Figure 4
runs of ✏-Greedy in LARPS. We can see that the agents oscillate between different actions, and that
the periods increase in length.
5 Related Work
5.1 Learning in games
Some of the Newcomblike dynamics we have described in this paper could also be modelled as games,
especially as so-called Stackelberg games in which one player, the Stackelberg leader , chooses a
strategy ﬁrst and another player, the Stackelberg follower , then responds to that strategy. For example,
in the case of autonomous vehicles (A Vs), we might imagine that the A V company is the Stackelberg
leader and the human drivers are the Stackelberg followers.
That said, there are differences between NDPs and games. NDPs can react arbitrarily to the agent’s
policy, whereas in games, the other players play a best (i.e., expected-utility-maximising) response.
In many real-world situations, other agents in the environment cannot be comfortably modelled as
expected-utility-maximising agents. Interactions between A Vs and humans can serve as examples.
Most people probably do not reason rationally about small-probability, big-impact events, such as
car crashes. Also, humans will generally operate on simpliﬁed models of an A V’s policy (even
when more detailed models are available). Of course, a game-theoretic analysis can also be fruitful
and address issues that we ignore: By assuming all players to be rational, game theory can provide
recommendations and predictions for multiple agents simultaneously, while our NDP model considers
a single agent for a given environment. We believe that the NDP perspective provides additional
insight into learning in such situations.
Despite the differences between NDPs and games, there are some interesting parallels between
model-free learning in NDPs and in games, where similar learning methods are sometimes referred
to as ﬁctitious play (Brown, 1951). Fudenberg and Levine (1998, Chapter 2) show that ﬁctitious play
can only converge to a Nash equilibrium (for similar results about convergence to Nash equilibrium,
see e.g. Mazumdar et al., 2020, Oesterheld et al., 2021). As noted in Sect. 2.1, the concept of Nash
equilibrium resembles the concept of ratiﬁability. Shapley (1964) shows that ﬁctitious play can fail
to converge. However, there are many special cases in which convergence is guaranteed, including
two-player zero-sum games (Robinson, 1951) and generic 2⇥2games (Miyasawa, 1961).
5.2 Learning and Newcomblike problems
Other authors have discussed learning in Newcomblike problems. The most common setup is one in
which the learner assigns values directly to policies, or more generally to that which the agent chooses.
9
It is then usually shown that (among the policies considered) the agent will converge to taking the
one with the highest (so-called evidential) expected utility (Albert and Heiner, 2001; Oesterheld,
2018). This contrasts with our setup, in which the learner selects policies but assigns values to actions.
Oesterheld (2019) also studies agents who maximise reward in Newcomblike environments. However,
Oesterheld does not consider the learning process. Instead he assumes that the agent has already
formed beliefs and uses some form of expected utility maximisation. He also speciﬁcally considers
the implications of having some overseer assign rewards based on beliefs about the state of the world
(as opposed to having the reward come directly from the true world state).
6 Discussion and Further Work
We have seen that value-based reinforcement learning algorithms can fail to converge to any policy in
some NDPs, and that when they do converge, they can only converge to ratiﬁable policies. Decision
theorists have discussed whether ratiﬁability should be considered to be a sound normative principle.
Note that (as discussed in Sect. 2) the optimal policies ⇡are not in general ratiﬁable. We have also
examined the limit action frequencies that agents can converge to (even when the policies do not
converge). Still, there are NDPs in which many agents cannot converge even to any such frequency.
We gave some results on what actions can be taken with positive limit frequency. A loose connection
to ratiﬁability can still be drawn.
Overall, established decision-theoretical ideas can be used to understand and formally describe the
behaviour of “out-of-the-box” reinforcement learning agents in NDPs. However, their behaviour is
not always desirable. Our work elucidates possible failures. We hope that our work will thus enable
more accurate reasoning about the behaviour of RL agents in real-world situations, especially when
interacting with humans or other agents. We hold such improvements in understanding to be broadly
beneﬁcial to the safe design and deployment of AI systems.
Throughout the paper, we have noted speciﬁc open questions related to our results. For instance, can
the results in Sect. 4.1 be generalised beyond the bandit setting? There are also many topics and
questions about our setting that we have not touched on at all. For instance, our experimental results
indicate that convergence is often slow (considering how simple the given problems are). It might be
desirable to back up this impression with theoretical results. We have only studied simple value-based
model-free algorithms – the analysis could be extended to other reinforcement learning algorithms
(e.g., policy-gradient or model-based algorithms). Also, there are further ways in which we could
generalise our setting. One example is to introduce partial observability and imperfect memory into
the NDPs. The latter has been studied in game and decision theory (Piccione and Rubinstein, 1997;
Elga, 2000), but recently – under the name memoryless POMDP – also in reinforcement learning
(Azizzadenesheli et al., 2016; Steckelmacher et al., 2018; cf. Conitzer, 2019). What makes this
especially appealing in the NDP context is that problems related to imperfect memory relate closely
to Newcomblike problems (Briggs, 2010; Schwarz, 2015). One could also look for narrower classes
of NDPs in which RL agents are guaranteed to perform well in some sense.
Ultimately, the goal of this line of research is to develop learners that are able to deal effectively and
safely with Newcomblike dynamics. We hope that our results will be useful in developing extensions
of value-based RL that can detect and correct for the failures that arise when existing methods are
applied in Newcomblike settings. However, we should also consider alternative approaches that
do not hinge on insights from the present work. For example, a few recent papers (on learning in
non-Newcomblike settings) have considered learning to predict the expected utility as a function of
thepolicy (as opposed to traditional Qvalues, which are not paremeterised by the policy) (Harb et al.,
2020). In principle, learning such a policy evaluation function avoids the problems of the learners
considered in this paper. However, it remains to be seen how practical this approach is.
Acknowledgements
We thank our anonymous reviewers for their exceedingly helpful comments. We thank the Founda-
tional Research Institute (now the Center on Long-Term Risk) and the funders and organisers of the
AI Safety Camp 2018 on Gran Canaria for supporting this project. Caspar Oesterheld is thankful for
support by the National Science Foundation under Award IIS-1814056.
10
References
Albert, M. and Heiner, R. A. (2001). An indirect-evolution approach to newcomb’s problem. CSLE
Discussion Paper, No. 2001-01.
Azizzadenesheli, K., Lazaric, A., and Anandkumar, A. (2016). Open problem: Approximate planning
of pomdps in the class of memoryless policies. In JMLR: Workshop and Conference Proceedings ,
volume 49, pages 1–4.
Briggs, R. (2010). Putting a value on beauty. In Oxford Studies in Epistemology , volume 3, pages
3–24. Oxford University Press.
Brown, G. W. (1951). Iterative solutions of games by ﬁctitious play. In Koopmans, T. C., editor,
Activity Analysis of Production and Allocation , chapter XXIV , pages 371–376. John Wiley & Sons
and Chapman & Hall.
Cavalcanti, E. G. (2010). Causation, decision theory, and bell’s theorem: A quantum analogue of the
newcomb problem. The British Journal for the Philosophy of Science , 61(3):569–597.
Condliffe, J. (2016). Humans will bully mild-mannered autonomous cars. MIT Technology Review .
Conitzer, V. (2019). Designing preferences, beliefs, and identities for artiﬁcial intelligence. In
Proceedings of the Thirty-Third AAAI Conference on Artiﬁcial Intelligence (AAAI-19) Senior
Member / Blue Sky Track .
Cooper, M., Lee, J. K., Beck, J., Fishman, J. D., Gillett, M., Papakipos, Z., Zhang, A., Ramos, J.,
Shah, A., and Littman, M. L. (2019). Stackelberg punishment and bully-prooﬁng autonomous
vehicles. In Proceedings of the 11th International Conference on Social Robotics , volume 11876
ofLecture Notes in Computer Science , pages 368–377. Springer, Madrid, Spain.
Elga, A. (2000). Self-locating belief and the sleeping beauty problem. Analysis , 60(2):143–147.
Fudenberg, D. and Levine, D. K. (1998). The Theory of Learning in Games . MIT Press.
Gibbard, A. and Harper, W. (1976). Counterfactuals and two kinds of expected utility. Foundations
and Applications of Decision Theory , 1:125–162.
Hall, P. and Heyde, C. (1980). Martingale Limit Theory and its Applications . New York: Academic
Press.
Harb, J., Schaul, T., Precup, D., and Bacon, P.-L. (2020). Policy evaluation networks.
Joyce, J. M. and Gibbard, A. (1998). Causal decision theory. In Handbook of Utility Theory, Volume
1: Principles. , chapter 13, pages 627–666. Kluwer.
Kakutani, S. (1941). A generalization of brouwer’s ﬁxed point theorem. Duke Math. J. , 8(3):457–459.
Liu, P., Du, Y., Wang, L., and Ju, D. Y. (2020). Ready to bully automated vehicles on public roads?
Accident Analysis & Prevention , 137.
Mazumdar, E., Ratliff, L. J., and Sastry, S. S. (2020). On gradient-based learning in continuous
games. SIAM Journal on Mathematics of Data Science , 2(1):103–131.
Miyasawa, K. (1961). On the convergence of the learning process in a 2x2 non-zero-sum two-person
game.
Nozick, R. (1969). Newcomb’s problem and two principles of choice. In et al., N. R., editor, Essays
in Honor of Carl G. Hempel , pages 114–146. Springer.
Oesterheld, C. (2018). Doing what has worked well in the past leads to evidential decision theory.
Oesterheld, C. (2019). Approval-directed agency and the decision theory of newcomb-like problems.
Synthese .
Oesterheld, C., Demski, A., and Conitzer, V. (2021). A theory of bounded inductive rationality.
11
Piccione, M. and Rubinstein, A. (1997). On the interpretation of decision problems with imperfect
recall. Games and Economic Behavior , 20:3–24.
Robinson, J. (1951). An iterative method of solving a game. Annals of Mathematics , 54(2):296–301.
Rummery, G. and Niranjan, M. (1994). On-line q-learning using connectionist systems. Technical
report, Cambridge University Engineering Department.
Schwarz, W. (2015). Lost memories and useless coins: revisiting the absentminded driver. Synthese ,
192(9):3011–3036.
Shapley, L. S. (1964). Some topics in two-person games. In Dresher, M., Shapley, L. S., and Tucker,
A. W., editors, Advances in Game Theory , chapter 1. Princeton University Press.
Singh, S., Jaakkola, T., Littman, M. L., and Szepesvári, C. (2000). Convergence results for single-step
on-policy reinforcement-learning algorithms. Machine Learning , 38:287–308.
Steckelmacher, D., Roijers, D. M., Harutyunyan, A., Vrancx, P., Plisnier, H., and Nowè, A. (2018).
Reinforcement learning in pomdps with memoryless options and option-observation initiation sets.
InThe Thirty-Second AAAI Conference on Artiﬁcial Intelligence (AAAI-18) .
van Seijen, H., van Hasselt, H., Whiteson, S., and Wiering, M. (2009). A theoretical and empirical
analysis of expected sarsa. In 2009 IEEE Symposium on Adaptive Dynamic Programming and
Reinforcement Learning , pages 177–184.
Watkins, C. (1986). Learning from delayed rewards . PhD thesis.
Weirich, P. (2016). Causal decision theory. In The Stanford Encyclopedia of Philosophy . Spring 2016
edition.
12

James Bell
Linda Linsefors
Caspar Oesterheld
Joar Skalse