Learning Rewards from Linguistic Feedback.

Learning Rewards from Linguistic Feedback
Theodore R. Sumers,1Mark K. Ho,2Robert D. Hawkins,2
Karthik Narasimhan,1Thomas L. Grifﬁths1,2
1Department of Computer Science, Princeton University, Princeton, NJ
2Department of Psychology, Princeton University, Princeton, NJ
fsumers, mho, rdhawkins, karthikn, tomg g@princeton.edu
Abstract
We explore unconstrained natural language feedback as a
learning signal for artiﬁcial agents. Humans use rich and var-
ied language to teach, yet most prior work on interactive
learning from language assumes a particular form of input
(e.g., commands). We propose a general framework which
does not make this assumption, using aspect-based sentiment
analysis to decompose feedback into sentiment about the fea-
tures of a Markov decision process. We then perform an ana-
logue of inverse reinforcement learning, regressing the senti-
ment on the features to infer the teacher’s latent reward func-
tion. To evaluate our approach, we ﬁrst collect a corpus of
teaching behavior in a cooperative task where both teacher
and learner are human. We implement three artiﬁcial learn-
ers: sentiment-based “literal” and “pragmatic” models, and
an inference network trained end-to-end to predict latent re-
wards. We then repeat our initial experiment and pair them
with human teachers. All three successfully learn from in-
teractive human feedback. The sentiment models outperform
the inference network, with the “pragmatic” model approach-
ing human performance. Our work thus provides insight into
the information structure of naturalistic linguistic feedback as
well as methods to leverage it for reinforcement learning.
1 Introduction
For autonomous agents to be widely usable, they must be re-
sponsive to human users’ natural modes of communication.
For instance, imagine designing a household cleaning robot.
Some behaviors can be pre-programmed (e.g. how to use an
outlet to recharge itself), while others must be learned (e.g.
if a user wants it to charge in the living room or the kitchen).
But how should the robot infer what a person wants?
Here, we focus on unconstrained linguistic feedback as a
learning signal for autonomous agents. Humans use natural
language ﬂexibly and intuitively to express their desires via
commands, counterfactuals, encouragement, explicit prefer-
ences, or other forms of feedback. For example, if a human
encounters the robot charging in the living room as desired,
they may provide feedback such as “Great job.” If they ﬁnd it
charging in the kitchen, the human may respond with “You
should have gone to the living room” or “I don’t like see-
ing you in the kitchen.” Our approach of learning rewards
Copyright © 2021, Association for the Advancement of Artiﬁcial
Intelligence (www.aaai.org). All rights reserved.from such open-ended language differs from previous meth-
ods for interactive learning that use non-linguistic demon-
strations (Abbeel and Ng 2004; Argall et al. 2009; Ho et al.
2016), rewards/punishments (Knox and Stone 2009; Mac-
Glashan et al. 2017; Christiano et al. 2017), or language
commands (Tellex et al. 2011; Wang, Liang, and Manning
2016; Tellex et al. 2020). The agent’s learning challenge is
to interpret naturalistic feedback in the context of its behav-
ior and environment to infer the teacher’s preferences.
We formalize this inference as linear regression over fea-
tures of a Markov decision process (MDP). We ﬁrst de-
compose linguistic feedback into a scalar sentiment and
a target subset of the MDP’s features (a form of aspect-
based sentiment analysis (Hu and Liu 2004; Liu 2020)). We
then regress the sentiment against the features to infer the
teacher’s latent reward function. This enables learning re-
wards from arbitrary language. To extract target features, we
draw on educational research (Shute 2008) to ﬁrst map ut-
terances to elements of the MDP. For example, “Good job”
maps to prior behavior, whereas “You should have gone to
the living room” maps to an action. We then extract relevant
MDP features: intuitively, positive sentiment about an ac-
tion implies positive rewards on its features. We implement
two versions of this model: a “literal” learner using only
the explicit sentiment and a “pragmatic” learner with ad-
ditional inductive biases (Grice 1975). These models allow
us to study how humans express themselves; however, they
rely on domain-speciﬁc lexical groundings. We thus develop
a parallel approach: training an inference network end-to-
end to predict latent rewards from human interactions. In our
live evaluation, all three models learn from human feedback.
Our sentiment-based models achieve statistically-signiﬁcant
improvements over the inference network; the “pragmatic”
model approaches human performance. We outline related
work in Section 2, then introduce our sentiment model in
Section 3. Section 4 describes our task and experiment and
Section 5 details our model implementations. Finally, Sec-
tion 6 discusses results and Section 7 concludes.1
2 Background and Related Work
The work presented here complements existing methods that
enable artiﬁcial agents to learn from and interact with hu-
1Code and data: github.com/tsumers/rewards .arXiv:2009.14715v2  [cs.AI]  16 Dec 2020
action feedback update 1 2 3
learner teachersentiment
featuresA B
pink is good,
but go for
the squares
next time!
imperative descriptive evaluativeFigure 1: A: Episodes involve three stages. B: We use aspect-based sentiment analysis to factor utterances into sentiment and
features , then infer latent weights w(solid lines). This allows us to integrate multiple forms of feedback (dashed lines).
mans. For example, a large literature studies how agents
can learn latent preferences from non-linguistic human feed-
back. Algorithms such as TAMER (Knox and Stone 2009)
and COACH (MacGlashan et al. 2017) transform human-
generated rewards and punishments into quantities that
reinforcement learning (RL) algorithms can reason with.
Preference elicitation, which provides a user with binary
choices between trajectories, is a similarly intuitive train-
ing method (Christiano et al. 2017). Finally, demonstration-
based approaches use a set of expert trajectories to learn a
policy—as in imitation learning (Ross and Bagnell 2010)—
or infer an underlying reward function—as in inverse rein-
forcement learning (IRL) (Abbeel and Ng 2004). This idea
has been extended to settings in which agents are provided
with intentionally informative demonstrations (Ho et al.
2016) or themselves act informatively (Dragan, Lee, and
Srinivasa 2013), such as the more general “cooperative IRL”
setting (Hadﬁeld-Menell et al. 2016).
Another body of research has focused on linguis-
tic human-agent interaction. Dialogue systems (Artzi and
Zettlemoyer 2011; Li et al. 2016) learn to interpret user
queries in the context of the ongoing interaction, while
robots and assistants (Thomason et al. 2015; Wang et al.
2019; Thomason et al. 2020; Szlam et al. 2019) ground lan-
guage in their physical surroundings (for a review of lan-
guage and robotics, see Tellex et al. (2020)). A parallel line
of work in machine learning uses language to improve sam-
ple efﬁciency: to shape rewards (Maclin and Shavlik 1994;
Kuhlmann et al. 2004), often via subgoals (Kaplan, Sauer,
and Sosa 2017; Williams et al. 2018; Chevalier-Boisvert
et al. 2018; Goyal, Niekum, and Mooney 2019; Bahdanau
et al. 2019; Zhou and Small 2020). These approaches gen-
erally interpret and execute independent declarative state-
ments (e.g., queries, commands, or (sub)goals). Perhaps the
most similar works to ours are (MacGlashan et al. 2015; Fu
et al. 2019; Goyal, Niekum, and Mooney 2020), which per-
form IRL on linguistic input in the form of natural language
commands. Our work differs in two key ways: ﬁrst, we
use unconstrained and unﬁltered natural language; second,
we seek to learn general latent preferences rather than in-
fer command-contextual rewards. A somewhat smaller body
of work investigates such open-ended language: to correct
captioning models (Ling and Fidler 2017), capture environ-
mental characteristics (Narasimhan, Barzilay, and Jaakkola2018), or improve hindsight replay (Cideron et al. 2019). For
a review of language and RL, see Luketina et al. (2019).
We aim to recover the speaker’s preferences from natu-
ralistic interactions. Thus, unlike prior approaches, we do
notsolicit a speciﬁc form of language (i.e. commands, cor-
rections, or descriptions). We instead elicit naturalistic hu-
man teaching and develop inferential machinery to learn
from it. This follows studies of emergent language in other
domains including action coordination (Djalali et al. 2011;
Djalali, Lauer, and Potts 2012; Potts 2012; Ilinykh, Zarrieß,
and Schlangen 2019; Suhr et al. 2019), reference pragmat-
ics (He et al. 2017; Udagawa and Aizawa 2019), naviga-
tion (Thomason et al. 2019), and “Wizard of Oz” experi-
ments (Kim et al. 2009; Allison, Luger, and Hofmann 2018).
3 Learning Rewards from Language
In this section, we formalize our approach. We develop a
form of aspect-based sentiment analysis (Hu and Liu 2004;
Liu 2020) to decompose utterances into sentiment and MDP
features, then use linear regression to infer the teacher’s re-
wards over those features. This allows us to perform an anal-
ogy of IRL (Abbeel and Ng 2004; Jeon, Milli, and Dragan
2020) on arbitrary language. To extract MDP features, we
ground utterances into the learner’s context (Harnad 1990;
Mooney 2008) with mappings inspired by educational re-
search (Shute 2008; Lipnevich and Smith 2009).
3.1 Setup
We begin by deﬁning a learner agent whose interactions
with the environment are deﬁned by a Markov Decision Pro-
cess (MDPs) (Puterman 1994). Formally, a ﬁnite-horizon
MDPM=hS;A;H;T;Riis a set of states S, a set of ac-
tionsA, a horizonH2N, a probabilistic transition function
T:SA!(S), and a reward function R:SA!R.
Given an MDP, a policy is a mapping from states to ac-
tions,:S!A. An optimal policy ,, is one that
maximizes the future expected reward (value) from a state,
Vh(s) = maxaR(s;a) +P
s0T(s0js;a)Vh1(s0), where
V0(s) = maxaR(s;a). States and actions are characterized
by features, where:SA!f0;1gKis an indica-
tor function representing whether a feature is present for a
particular action ain states. We denote a state-action trajec-
tory by=hs0;a0;:::sT;aTi. Finally, we deﬁne the feature
counts over a set of state-action tuples as n:
n(fhs;aig) =X
s;a2fhs;aig(s;a) (1)
which we use to summarize arbitrary sets of state-action tu-
ples, including trajectories.
3.2 Interactive Learning from Language
We consider a setting where the reward function is hidden
from the learner agent but known to a teacher agent who
is allowed to send natural-language messages u(Fig. 1A).
We formulate the online learning task as Bayesian infer-
ence over possible rewards: conditioning on the teacher’s
language and recursively updating a belief state. Formally,
we assume that the teacher’s reward function is parameter-
ized by a latent variable w2RKrepresenting the rewards
associated with features :
R(s;a) =w>(s;a): (2)
We refer to these weights as the teacher’s preferences over
features. The learner is attempting to recover the teacher’s
preferences from their utterances, calculating P(wju).
Learning unfolds over a series of interactive episodes . At
the start of episode i, the learner has a belief distribution over
the teacher’s reward weights, P(wi), which it uses to iden-
tify its policy. First, the learner has an opportunity to actin
the world given this policy, sampling a trajectory i. Second,
they receive feedback in the form of a natural language ut-
teranceuifrom the teacher (and optionally a reward signal
from the environment). Third, the learner uses the feedback
toupdate its beliefs about the reward, P(wi+1jui;i), which
is then used for the next episode.
We now describe our general formal approach for in-
ferring latent rewards from feedback. We ﬁrst assume the
learner extracts the sentimentandtarget features ffrom
the teacher’s utterance, where f2RKis a vector describ-
ing which features the utterance relates to. Extracting a
sentiment and its target is known as aspect-based sentiment
analysis (Liu 2020). Ready solutions exist to distill senti-
ment from language (Hutto and Gilbert 2014; Kim 2014),
but extracting the target features is more challenging. We
detail our approach in Section 3.3.
We then formalize learning as Bayesian linear regression:
N(f>w;2
) (3)
We use a Gaussian prior: wi N (i;i). After each
episode, we perform Bayesian updates (Murphy 2007) to
obtain a posterior: P(wi+1jui;i) =N(i+1;i+1). Thus,
similar to IRL methods (Ramachandran and Amir 2007), the
regression sets the teacher’s latent preferences w2RK
to “explain” the sentiment. Intuitively, if the teacher says
“Good job,” a learner could infer the teacher has positive
weights on the features obtained by its prior trajectory. In
the next section, we formalize this mapping to features.
3.3 Extracting MDP Features from Language
The main challenge of aspect-based sentiment analysis is ex-
tracting target features from arbitrary language. To accom-
plish this, we draw on educational research (Lipnevich andSmith 2009; Shute 2008), which studies the characteristic
forms of feedback given by human teachers. We ﬁrst iden-
tify correspondences between these forms and prior work
in RL. We then show each form targets a distinct element
of the MDP (e.g., a prior trajectory). Mapping language to
these elements allows us to extract target features.
Evaluative Feedback . Perhaps the simplest feedback an
agent can receive is a scalar value in response to their ac-
tions (e.g. environmental rewards, praise, criticism). The
RL literature has previously elicited such feedback (+1/-1)
from human teachers (Thomaz and Breazeal 2008; Knox
and Stone 2009; MacGlashan et al. 2017). In our setting, we
consider how linguistic utterances can be interpreted as eval-
uative feedback. For example, “Good job” clearly targets the
learner’s behavior, i. We thus set the target features to the
feature counts obtained by that trajectory: f=n(i).2
Imperative Feedback . Another form of feedback tells
the learner what the correct action was. This is the gen-
eral form of supervised learning. In RL, it includes label-
ing sets of actions as good or bad (Judah et al. 2010; Chris-
tiano et al. 2017), learning from demonstrations (Ross and
Bagnell 2010; Abbeel and Ng 2004; Ho et al. 2016), and
corrections to dialogue agents (Li et al. 2016; Chen et al.
2017). In our setting, imperative feedback speciﬁes a coun-
terfactual behavior: something the learner should (or should
not) have done, e.g. “You should have gone to the living
room.” Imperative feedback is thus a retrospective version
of a command. Extracting features takes two steps: we ﬁrst
ground the language to a set of actions, then aggregate their
feature counts. Formally, we deﬁne a state-action grounding
functionG(u;S;A )which returns a set of state-action tuples
from the full set: G:u;S;A7!~S;~A, where ~SS;~AA.
We take the feature counts of these tuples as our target:
f=n(G(u;S;A )).
Descriptive Feedback . Finally, descriptive feedback pro-
vides explicit information about how the learner should
modify their behavior. Descriptive feedback is the most vari-
able form of human teaching, encompassing explanations
and problem-solving strategies. It is generally found to be
the most effective (Shute 2008; Lipnevich and Smith 2009;
van der Kleij, Feskens, and Eggen 2015; Hattie and Tim-
perley 2007). Supervised and RL approaches have used de-
scriptive language to improve sample efﬁciency (Srivastava,
Labutov, and Mitchell 2017; Hancock et al. 2018; Ling and
Fidler 2017) or communicate general task-relevant informa-
tion (Narasimhan, Barzilay, and Jaakkola 2018). In IRL, de-
scriptive feedback explains the underlying structure of the
teacher’s preferences and thus relates directly to features .3
If the human says ‘I don’t like seeing you in the kitchen,”
the robot should infer negative rewards for states and ac-
tions where it and the human are both in the kitchen. For-
mally, we deﬁne an indicator function over features desig-
2As an example, an alternative teaching theory could use an
inverse choice model (McFadden 1974; Train 2003). This would
posit a teacher giving feedback on the learner’s implied, latent pref-
erences, rather than their explicit, observed actions.
3In problem settings beyond IRL, such feedback may relate to
the transition function T:SA.
Figure 2: A: The learner collected objects with different rewards (top) masked with shapes and colors (bottom). The teacher
could see both views. B: Example reward function used to mask objects (here, objects worth 1-3 reward are rendered as pink
triangles; the teacher thus “prefers” pink squares, worth 8-10). C: Pairs played 10 episodes, each on a new level. D: Feedback
shifted from descriptive to evaluative as learners improved. Learners scored poorly on level 6, reversing this trend.
nating whether or not that feature is referenced in the utter-
ance:I:u;!f0;1gK. We then set f=I(u;).
Prior RL algorithms generally operate on one of these
forms. Interactions are constrained, as the algorithm solicits
feedback of a particular type. Our framework uniﬁes them,
allowing us to learn from a wide range of naturalistic human
feedback. Concretely, we deﬁne a grounding functionfG:
u7!form;form2fImperative, Evaluative, Descriptive g,
then extract faccordingly:
f2RK=8
><
>:n(i) iffG(u) =Evaluative
n(G(u;S;A ))iffG(u) =Imperative
I(u;) iffG(u) =Descriptive
(4)
This procedure is illustrated in Fig. 1. Table 1 shows ex-
amples of this decomposition for various forms. In Sec-
tion 4, we elicit and analyze naturalistic human-human
teaching, observing these three forms. In Section 5, we de-
scribe a pair of agents implementing this model. Finally, we
train an end-to-end neural network and probe its represen-
tations, showing that it learns to interpret different forms of
feedback in a similar manner.
4 Human-Human Instruction Dataset
To study human linguistic feedback and generate a dataset
to evaluate our method, we designed a two-player collabo-
rative game (Fig. 2). One player (the learner) used a robot
to collect a variety of colored shapes. Each yielded a re-
ward, which the learner could not see. The second player
(the teacher) watched the learner and could see the ob-
jects’ rewards. Teacher-learner pairs engaged in 10 interac-
tive episodes. We describe the experiment below.
4.1 Experiment and Gameplay
We recruited 208 participants from Amazon Mechanical
Turk using psiTurk (Gureckis et al. 2016). Participants were
paid $1.50 and received a bonus up to $1.00 based on the
learner’s score. The full experiment consisted of instructionsand a practice level, followed by 10 levels of gameplay. Each
level contained a different set of 20 objects. We generated
110 such levels, using 10 for the experiment and 100 for
model evaluation (Section 6). Collecting each object yields a
reward between -10 and 10. Objects were distributed to each
of the four corners (Fig. 2C). In each episode , the learner had
8 seconds to act(move around and collect objects), then the
teacher had unlimited time to provide feedback (send chat
messages). Both players were shown the score and running
bonus during the feedback phase. This leaked information
about the reward function to the learner, but we found it
was important to encourage active participation. The pri-
mary disadvantage is that the human baseline for human-
model comparisons beneﬁts from additional information not
seen by our models.
4.2 Task MDP and Rewards
Human control was continuous: the learner used the arrow
keys to steer the robot. However, the only rewarding actions
were collecting objects and there was no discounting. As a
result, the learner’s score was the sum of the rewards of the
collected objects. Due to object layout and short time hori-
zon, learners only had time to reach one corner. Each corner
had 5 objects, so there were 124 possible object combina-
tions per level.4We refer to these combinations as trajecto-
ries, and formalize the task as choosing one. Concretely,
the learner samples its beliefs wp(wi), then chooses
the optimal trajectory: := argmaxV=w>n(). To
induce teacher preferences, we assigned each teacher a re-
ward function which masked objects with shapes and colors.
Thus the distribution of actions and rewards on each level
were the same for all players, but the objects were displayed
differently depending on the assigned reward function. Our
reward functions combined two independent perceptual di-
mensions, with color (pink, blue, or yellow) encoding sign
4Choice of corner, then up to 5 objects: 4(5
1
+5
2
+5
3
+5
4
+5
5
) = 124 .
Utterance Feedback Form Grounding ( fG) Features ( f) Sentiment ( )
“Keep it up excellent” Evaluative n(i) Behavior-dependent +17
“Not a good move” Evaluative n(i) Behavior-dependent -10
“Top left would have been better” Imperative n(G(u;S;A )) Environment-dependent +17
“The light-blue squares are high valued” Descriptive I(u;) BlueSquare +13
“I think Yellow is bad” Descriptive I(u;) Yellow -16
Table 1: Example feedback from our experiment with feature / sentiment decomposition.
and shape (circles, squares, or triangles) encoding magni-
tude (Fig. 2B). We permuted the shapes and colors to gener-
ate 36 different functions.
4.3 Human-Human Results and Language
Our 104 human pairs played 10 games each, yielding 1040
total messages (see Table 1 for examples; note empty mes-
sages were allowed). We use our feedback classiﬁer (see
Section 5.1) to explore the prevalence of various forms of
feedback. We observe that humans use all three in a curricu-
lum structure known as “scaffolding” (Shute 2008): teachers
initially use descriptive feedback to correct speciﬁc behav-
ior, then switch to evaluative as the learners’ score improves
(Fig 2D). Teachers could send unlimited messages and thus
sometimes used multiple forms. Most episodes contained
evaluative (63%) or descriptive (34%) feedback; only 6%
used imperative . The infrequency of imperative feedback is
reasonable given our task: specifying the optimal trajectory
via language is more challenging than describing desirable
features. Not all pairs fared well: some learners did not lis-
ten, leading teachers to express frustration; some teachers
did not understand the task or sent irrelevant messages. We
do not ﬁlter these out, as they represent naturalistic human
language productions under this setting.
5 Agent Models
We now describe our three models. The ﬁrst (Section 5.1) di-
rectly implements our sentiment-based framework. The sec-
ond (Section 5.2) extends it with pragmatic biases based on
Gricean maxims (Grice 1975). Finally, we train a neural net
end-to-end from experiment episodes (Section 5.3).
5.1 “Literal” Model
Our literal model uses a supervised classiﬁer to implement
fGand a small lexicon to extract target features.
Utterance Segmentation and Sentiment. Teachers often
sent multiple messages per episode, each potentially con-
taining multiple forms of feedback. To process them, we ﬁrst
split each message on punctuation (!.,;), then treated each
split from each message as a separate utterance. To extract
sentiment, we used V ADER (Hutto and Gilbert 2014), which
is optimized for social media. V ADER provides an output
2[1;1], which we scaled by 30 (set via grid search).
Utterance Features. To implement fG, we labeled 685
utterances from pilot experiments and trained a logistic re-
gression on TF-IDF unigrams and bigrams, achieving a
weighted-F1 of .86. For evaluative feedback, as described inEq. 4, we simply used the feature counts from the learner’s
trajectory f=n(i). Imperative feedback requires a task-
speciﬁc action-grounding function G(u;S;A ). While ac-
tion grounding in complex domains is an open research
area (Tellex et al. 2020), in our experiment all imperative
language referenced a cluster of objects (e.g. “Top left would
have been better”). We thus used regular expressions to iden-
tify references to corners and aggregated features over ac-
tions in that corner. For descriptive feedback, we deﬁned a
similar indicator function I(u;)identifying features in the
utterance. We used relatively nameable shapes and colors,
so teachers used predictable language to refer to object fea-
tures (“pink”, “magenta”, “purple”, “violet”...). Again, we
used regular expressions to match these synonyms. Finally,
we normalizedkfk1= 1so all forms carry equal weight.
Belief Updates . Because players had seen object values
in practice levels ranging between -10 and 10, we initialized
our belief state as 0= 0,0=diag(25). This gives an
approximately 95% chance of feature weights falling into
that range. For each utterance, we perform Bayesian updates
to obtain posteriors P(wi+1jui;i) =N(i+1;i+1). We
use2
=1
2for all updates, which we set via grid search.
5.2 “Pragmatic” Model
We augment the “literal” model with two biases based on
pragmatic principles (Grice 1975). While pragmatics are of-
ten derived as a result of recursive reasoning (Goodman and
Frank 2016), we opt for a simpler heuristic approach.
Sentiment Pragmatics. The Gricean “maxim of quan-
tity” states that speakers bias towards parsimony. Empiri-
cally, teachers often referenced a feature or an action with-
out an explicit sentiment. Utterances such as “top left” or
“pink circles” implied positive sentiment (e.g. “pink circles
[are good]”). To account for this, we defaulted to a positive
bias (= 15 ) if the detected sentiment was neutral.
Reference Pragmatics. The Gricean “maxim of relation”
posits that speakers provide information that is relevant to
the task at hand. We assume utterances describe impor-
tant features, and thus unmentioned features are notuseful
for decision making. We implemented this bias by follow-
ing each Bayesian update with a second, negative update
(=30, set via grid search) to all features not referenced
by the original update, gradually decaying weights of un-
mentioned features.
5.3 End-to-end Inference Network
To complement our lexicon-based sentiment models, we
train a small inference network to predict the teacher’s la-
Figure 3: Left: learning within our experiment structure.
We plot averaged normalized score over the 10 learning
episodes; bars indicate 1 SE (68% CI). Right: learning with
speciﬁc feedback types. We plot averaged normalized score
on 100 test levels after each episode.
tent rewards. We use human data from our experiment to
learn an end-to-end mapping from the (u;)tuples to the
teacher’s reward parameters. Conceptually, this is akin to
“factory-training” a housecleaning robot, enabling it to sub-
sequently adapt to its owners’ particular preferences.
Model Architecture. We use a feed-forward architecture.
We tokenize the utterance and generate a small embedding
space (D=30), representing phrases as a mean bag-of-words
(MBOW) across tokens. We represent the trajectory with
its feature counts, n(). We concatenate the token embed-
dings with the feature counts, use a single fully-connected
128-width hidden layer with ReLU activations, then use a
linear layer to map down to a 9-dimension output.
Model Training. Our dataset is skewed towards positive-
scoring games, as players learned over the course of the ex-
periment. To avoid learning a default positive bias, we ﬁrst
downsample positive-score games to match negative-score
ones. This left a total of 388 episodes from 98 different
teachers with a mean score of 1.09 (mean of all games was
8.53). We augment the data by exchanging the reward func-
tion (Fig. 2B), simulating the same episode under a differ-
ent set of preferences. We take a new reward function and
switch both feature counts and token synonyms, preserving
the relationships between ui;i, andw. We repeat this for
all 36 possible reward functions, increasing our data vol-
ume and allowing us to separate rewards from teachers. We
used ten-fold CV with 8-1-1 train-validate-test splits, split-
ting both teachers and reward functions. Thus the network is
trained on one set of rewards (i.e. latent human preferences)
and teachers (i.e. linguistic expression of those preferences),
then tested against unseen preferences and language. We
used stochastic gradient descent with a learning rate of .005
and weight decay of 0.0001, stopping when validation set er-
ror increased. We train the network, including embeddings,
end-to-end with an L2 loss on the true reward.
Multiple Episodes. Given a (u;)tuple, our model pre-
dicts the reward ^wassociated with every feature. To evaluate
it over multiple trials in Section 6, we use a comparable up-
date procedure as our structured models. Concretely, we ini-
tialize univariate Gaussian priors over each feature 0= 0,
0= 25 , then run our inference network on each interactionModel Experiment Interaction Sampling
Ofﬂine Live n All Eval Desc Imp
Literal 30.6 34.7 46 40.5 38.7 40.6 16.7
Pragmatic 38.2 42.8 47 52.5 50.4 58.2 31.7
Inference 25.3 35.0 55 47.6 54.3 53.2 –
Human – 44.3 104 – – – –
Table 2: Normalized scores averaged over 10 episodes of
learning. “Experiment” plays the 10 experiment episodes
with a single human; “Interaction Sampling” draws ( u;)
tuples from the entire corpus and plays 100 test levels after
each update.
and perform a Bayesian update on each feature using our
model’s output as an observation with the same ﬁxed noise.
For each feature, P(wi+1jui;i) =N(i;i)N( ^w;1
2).
In all ofﬂine testing, we use the network from the appropri-
ate CV fold to ensure it is always evaluated on its test set
(teachers and rewards).
6 Results and Analysis
We seek to answer several questions about our models. First,
do they work : do they learn the humans’ reward function?
Second, does our sentiment approach provide an advan-
tage over the end-to-end learned model? And ﬁnally, do the
“pragmatic” augmentations improve the “literal” model? We
run a second interactive experiment pairing human teachers
with our models and ﬁnd the answer to all three is yes (Sec-
tion 6.1). We then analyze how our models learn by testing
forms of feedback separately (Section 6.2).
6.1 Learning from Live Interactions
To evaluate our models in an online setting, we recruited 148
additional participants from Proliﬁc, an online participant
recruitment tool, and paired each with one of three model
learners in our task environment. We measured the averaged
normalized score across all 10 levels (i.e. the mean percent-
age of the highest possible score achieved). To assess the ef-
fect of interactivity, we also evaluated the same three model
learners on replayed sequences of (u;)tuples drawn from
our earlier human-human experiment. The results are shown
in Fig. 3 and summarized in Table 2. We conducted a mixed-
effects regression (Kuznetsova, Brockhoff, and Christensen
2017) using performance as the dependent variable, includ-
ing ﬁxed effects of time (i.e. trial 1, trial 2, etc.), interactiv-
ity (i.e. live vs. ofﬂine), and learner model (i.e. neural vs.
literal vs. pragmatic), as well as an interaction between in-
teractivity and time. We also included random intercepts and
random effects for the learner model for each dyad to con-
trol for clustered variance. The categorical factor of the lan-
guage model was contrast-coded to ﬁrst compare the neural
against the two sentiment models and then compare the two
sentiment models directly against each other.
First, we found a signiﬁcant main effect of time,
t(4138) = 32 :77;p<: 001, indicating that performance im-
proves over successive levels. Second, although there was no
signiﬁcant main effect of interactivity, t(446) =:08;p=
Figure 4: Left: A trajectory from our experiment. Right: Inference network output given this trajectory. Top row: the model
learns to map feature-related tokens (descriptive feedback) directly to rewards, independent of the trajectory. Bottom left /
center: the model maps evaluative feedback (praise and criticism) through the feature-counts from the trajectory. Bottom right: a
failure mode. Due to the MBOW utterance representation, descriptive feedback with negative sentiment is not handled correctly.
:94, there was a signiﬁcant interaction between interactivity
and time,t(4138) = 2:32;p=:02, suggesting that the ben-
eﬁts of the live condition manifest over successive episodes
as the teacher provides feedback conditioned on the learner’s
behaviors. Finally, turning to the models themselves, we ﬁnd
that the “family” of sentiment models collectively outper-
form the neural network t(132) = 3:57;p < : 001and the
“pragmatic” sentiment model outperforms the “literal” one,
t(147) =2:37;p=:019. Post-hoc pairwise tests (Tukey
1953) ﬁnd an estimated difference of d= 7:8;95% CI:
[2:7;12:9]between the “pragmatic” and “literal” models;
d=3:77;95% CI:[11:8;4:3]between the neural and
“literal”; and d=11:5;95% CI:[19:4;3:7]between
neural and “pragmatic.” This suggests the end-to-end model
learns how to use most of the literal information in the data,
while the inductive biases we encoded into the “pragmatic”
further improve performance, comparable to human learn-
ers (with the caveat that our experiments used different plat-
forms to recruit participants).
6.2 Learning from Different Forms of Feedback
To characterize model learning from different “forms” of
feedback, we design a second evaluation independent of the
experiment structure. Our “episode” sequence is as follows:
we draw a (u;)tuple at random from the human-human ex-
periment, update each model, and have it acton our 100 pre-
generated test levels. We take its averaged normalized score
on these levels. We repeat this procedure 5 times for each
cross-validation fold, ensuring the learned model is always
tested on its hold-out teachers and rewards. This draws feed-
back from a variety of teachers and tests learners on a vari-
ety of level conﬁgurations, giving a picture of overall learn-
ing trends. Normalized scores over test levels are shown in
Fig. 3 and Table 2 (“Interaction Sampling”). We note that
all models improve when sampling the entire corpus (“All”)
versus being paired with an individual teacher (Section 6.1).
This suggests blending feedback across teachers helps miti-
gate individual idiosyncrasies. This particularly beneﬁts the
inference network, which outperforms the “literal” model.
We then use our feedback classiﬁer (Section 5.1) to selec-tively sample one form of feedback. This reveals that our
“pragmatic” augmentations help on all forms, but most dra-
matically on “Descriptive” feedback. Finally, we conﬁrm
our inference network learns to contextualize feedback ap-
propriately (Fig. 4). It maps evaluative feedback through
its prior behavior and descriptive (feature-based) tokens di-
rectly to the appropriate features. Inspection of utterances
reveals several failure modes on rarer speech patterns, most
notably descriptive feedback with negative sentiment.
7 Conclusion
We presented two methods to recover latent rewards from
naturalistic language: using aspect-based sentiment analy-
sis and learning an end-to-end mapping from utterances
to rewards. In evaluation with live humans, we ﬁnd our
sentiment-based models offer statistically signiﬁcant perfor-
mance gains over the end-to-end model. The “pragmatic”
model achieves near-human performance, suggesting that
humans do indeed draw inferences beyond the literal infor-
mation content. Finally, we note that the relative model per-
formance is qualitatively different across our three evalua-
tions: the learned model underperforms the “literal” ofﬂine,
performs equivalently in live experiments, and outperforms
it on randomly sampled interactions. This underscores the
importance of realistic (live) testing.
We see several future research directions. First, our sen-
timent models could be improved via theory-of-mind based
pragmatics, while our end-to-end model could beneﬁt from
stronger language models (recurrent networks or pre-trained
embeddings). Most promisingly, hybridizing sentiment and
learned approaches (Jiang et al. 2011; Xu et al. 2019) could
offer the best of both. We also see potential synergies with
instruction following: treating commands as “Imperative”
feedback could provide a preference-based prior for inter-
preting future instructions. Finally, we anticipate extend-
ing our approach to more complex MDPs in which humans
teach both rewards and transition dynamics (Narasimhan,
Barzilay, and Jaakkola 2018). In general, we hope the meth-
ods and insights presented here facilitate the adoption of
truly natural language as an input for learning.
Ethical Statement
Equipping artiﬁcial agents with the capacity to learn from
linguistic feedback is an important step towards improving
value alignment between humans and machines, one aspect
of developing systems that support safe and beneﬁcial in-
teractions. However, one future risk is expanding the set of
roles that such agents can play to those requiring signiﬁcant
interaction with humans – roles currently restricted to hu-
man agents. As a consequence, certain jobs may be more
readily replaced by automated systems. On the other hand,
being able to provide feedback to artiﬁcial agents verbally
could expand the group of people who will be able to inter-
act with those agents, potentially creating new opportunities
for people with disabilities or less formal training in com-
puter science.
References
Abbeel, P.; and Ng, A. Y . 2004. Apprenticeship Learning via In-
verse Reinforcement Learning. ICML, 1. New York, NY , USA.
Allison, F.; Luger, E.; and Hofmann, K. 2018. How Players Speak
to an Intelligent Game Character Using Natural Language Mes-
sages. Transactions of the Digital Games Research Association 4.
Argall, B. D.; Chernova, S.; Veloso, M.; and Browning, B. 2009.
A survey of robot learning from demonstration. Robotics and au-
tonomous systems 57(5): 469–483.
Artzi, Y .; and Zettlemoyer, L. 2011. Bootstrapping Semantic
Parsers from Conversations. In EMNLP 2011 , 421–432. ACL.
Bahdanau, D.; Hill, F.; Leike, J.; Hughes, E.; Hosseini, S. A.; Kohli,
P.; and Grefenstette, E. 2019. Learning to Understand Goal Speci-
ﬁcations by Modelling Reward. In ICLR 2019 .
Chen, L.; Yang, R.; Chang, C.; Ye, Z.; Zhou, X.; and Yu, K. 2017.
On-line Dialogue Policy Learning with Companion Teaching. In
Proceedings of the 15th Conference of the European Chapter of
the Association for Computational Linguistics , 198–204. Valencia,
Spain: Association for Computational Linguistics.
Chevalier-Boisvert, M.; Bahdanau, D.; Lahlou, S.; Willems, L.; Sa-
haria, C.; Nguyen, T. H.; and Bengio, Y . 2018. BabyAI: First Steps
Towards Grounded Language Learning With a Human In the Loop.
ArXiv abs/1810.08272.
Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; and
Amodei, D. 2017. Deep Reinforcement Learning from Human
Preferences. In NeurIPS , 4299–4307.
Cideron, G.; Seurin, M.; Strub, F.; and Pietquin, O. 2019. Self-
Educated Language Agent With Hindsight Experience Replay For
Instruction Following. ArXiv abs/1910.09451.
Djalali, A.; Clausen, D.; Lauer, S.; Schultz, K.; and Potts, C. 2011.
Modeling Expert Effects and Common Ground Using Questions
Under Discussion. In Proceedings of the AAAI Workshop on Build-
ing Representations of Common Ground with Intelligent Agents .
Washington, DC: AAAI Press.
Djalali, A.; Lauer, S.; and Potts, C. 2012. Corpus Evidence for
Preference-Driven Interpretation. In Aloni, M.; Kimmelman, V .;
Roelofsen, F.; Sassoon, G. W.; Schulz, K.; and Westera, M., eds.,
Proceedings of the 18th Amsterdam Colloquium: Revised Selected
Papers , 150–159. Berlin: Springer.
Dragan, A. D.; Lee, K. C.; and Srinivasa, S. S. 2013. Legibility
and Predictability of Robot Motion. In ACM/IEEE International
Conference on Human-Robot Interaction .Fu, J.; Korattikara, A.; Levine, S.; and Guadarrama, S. 2019. From
Language to Goals: Inverse Reinforcement Learning for Vision-
Based Instruction Following. ArXiv abs/1902.07742.
Goodman, N. D.; and Frank, M. C. 2016. Pragmatic Language
Interpretation as Probabilistic Inference. Trends in Cognitive Sci-
ences 20(11): 818 – 829.
Goyal, P.; Niekum, S.; and Mooney, R. J. 2019. Using natural
language for reward shaping in reinforcement learning. In IJCAI ,
2385–2391. AAAI Press.
Goyal, P.; Niekum, S.; and Mooney, R. J. 2020. PixL2R: Guid-
ing Reinforcement Learning Using Natural Language by Mapping
Pixels to Rewards.
Grice, H. P. 1975. Logic and Conversation. In Syntax and Seman-
tics: Vol. 3: Speech Acts , 41–58. New York: Academic Press.
Gureckis, T. M.; Martin, J.; McDonnell, J.; Rich, A. S.; Markant,
D.; Coenen, A.; Halpern, D.; Hamrick, J. B.; and Chan, P. 2016.
psiTurk: An open-source framework for conducting replicable be-
havioral experiments online. Behavior Research Methods 48(3):
829–842. ISSN 1554-3528.
Hadﬁeld-Menell, D.; Russell, S. J.; Abbeel, P.; and Dragan, A.
2016. Cooperative Inverse Reinforcement Learning. In Lee, D. D.;
Sugiyama, M.; Luxburg, U. V .; Guyon, I.; and Garnett, R., eds.,
NeurIPS , 3909–3917. Curran Associates, Inc.
Hancock, B.; Varma, P.; Wang, S.; Bringmann, M.; Liang, P.; and
R´e, C. 2018. Training Classiﬁers with Natural Language Explana-
tions. ACL .
Harnad, S. 1990. The symbol grounding problem. Physica D:
Nonlinear Phenomena 42(1-3): 335–346.
Hattie, J.; and Timperley, H. 2007. The Power of Feedback. Review
of Educational Research 77(1): 81–112.
He, H.; Balakrishnan, A.; Eric, M.; and Liang, P. 2017. Learning
Symmetric Collaborative Dialogue Agents with Dynamic Knowl-
edge Graph Embeddings. ACL .
Ho, M. K.; Littman, M.; MacGlashan, J.; Cushman, F.; and Auster-
weil, J. L. 2016. Showing versus Doing: Teaching by Demonstra-
tion. In NeurIPS , 3027–3035.
Hu, M.; and Liu, B. 2004. Mining and summarizing customer re-
views. In SIGKDD , 168–177.
Hutto, C. J.; and Gilbert, E. 2014. V ADER: A Parsimonious Rule-
Based Model for Sentiment Analysis of Social Media Text. In
ICWSM . The AAAI Press.
Ilinykh, N.; Zarrieß, S.; and Schlangen, D. 2019. MeetUp! A Cor-
pus of Joint Activity Dialogues in a Visual Environment. arXiv
abs/1907.05084.
Jeon, H. J.; Milli, S.; and Dragan, A. D. 2020. Reward-rational
(implicit) choice: A unifying formalism for reward learning. ArXiv
abs/2002.04833.
Jiang, L.; Yu, M.; Zhou, M.; Liu, X.; and Zhao, T. 2011. Target-
dependent twitter sentiment classiﬁcation. In AACL , 151–160.
Judah, K.; Roy, S.; Fern, A.; and Dietterich, T. G. 2010. Rein-
forcement Learning via Practice and Critique Advice. In AAAI ,
481–486.
Kaplan, R.; Sauer, C.; and Sosa, A. 2017. Beating Atari
with Natural Language Guided Reinforcement Learning. ArXiv
abs/1704.05539.
Kim, E. S.; Leyzberg, D.; Tsui, K. M.; and Scassellati, B. 2009.
How people talk when teaching a robot. In ACM/IEEE Interna-
tional Conference on Human-Robot Interaction , 23–30.
Kim, Y . 2014. Convolutional Neural Networks for Sentence Clas-
siﬁcation. EMNLP 2014 .
Knox, W. B.; and Stone, P. 2009. Interactively Shaping Agents via
Human Reinforcement: The TAMER Framework. In Proceedings
of the Fifth International Conference on Knowledge Capture , 9–16.
New York, NY , USA: Association for Computing Machinery.
Kuhlmann, G.; Stone, P.; Mooney, R.; and Shavlik, J. 2004. Guid-
ing a reinforcement learner with natural language advice: Initial
results in RoboCup soccer. AAAI Workshop - Technical Report .
Kuznetsova, A.; Brockhoff, P. B.; and Christensen, R. 2017.
lmerTest package: tests in linear mixed effects models. Journal
of statistical software 82(13): 1–26.
Li, J.; Miller, A. H.; Chopra, S.; Ranzato, M.; and Weston, J.
2016. Dialogue Learning With Human-In-The-Loop. ArXiv
abs/1611.09823.
Ling, H.; and Fidler, S. 2017. Teaching Machines to Describe Im-
ages with Natural Language Feedback. In NIPS .
Lipnevich, A.; and Smith, J. 2009. Effects of differential feedback
on students’ examination performance. Journal of Experimental
Psychology: Applied 15(4): 319–333.
Liu, B. 2020. Sentiment Analysis: Mining Opinions, Sentiments,
and Emotions . Cambridge University Press.
Luketina, J.; Nardelli, N.; Farquhar, G.; Foerster, J.; Andreas, J.;
Grefenstette, E.; Whiteson, S.; and Rockt ¨aschel, T. 2019. A Sur-
vey of Reinforcement Learning Informed by Natural Language. In
IJCAI 2019 , volume 57. AAAI Press.
MacGlashan, J.; Babes-Vroman, M.; desJardins, M.; Littman,
M. L.; Muresan, S.; Squire, S.; Tellex, S.; Arumugam, D.; and
Yang, L. 2015. Grounding English Commands to Reward Func-
tions. In Robotics: Science and Systems .
MacGlashan, J.; Ho, M. K.; Loftin, R.; Peng, B.; Wang, G.;
Roberts, D. L.; Taylor, M. E.; and Littman, M. L. 2017. Interactive
Learning from Policy-Dependent Human Feedback. JMLR.
Maclin, R.; and Shavlik, J. W. 1994. Incorporating advice into
agents that learn from reinforcements. AAAI.
McFadden, D. 1974. Conditional logit analysis of qualitative
choice behavior. Frontiers in Econometrics 105–142.
Mooney, R. J. 2008. Learning to Connect Language and Percep-
tion. In AAAI , 1598–1601.
Murphy, K. 2007. Conjugate Bayesian analysis of the Gaus-
sian distribution URL https://www.cs.ubc.ca/murphyk/Papers/
bayesGauss.pdf.
Narasimhan, K.; Barzilay, R.; and Jaakkola, T. 2018. Grounding
language for transfer in deep reinforcement learning. Journal of
Artiﬁcial Intelligence Research 63: 849–874.
Potts, C. 2012. Goal-Driven Answers in the Cards Dialogue Cor-
pus. In Arnett, N.; and Bennett, R., eds., Proceedings of the 30th
West Coast Conference on Formal Linguistics , 1–20. Somerville,
MA: Cascadilla Press.
Puterman, M. L. 1994. Markov Decision Processes: Discrete
Stochastic Dynamic Programming . John Wiley & Sons, Inc.
Ramachandran, D.; and Amir, E. 2007. Bayesian Inverse Rein-
forcement Learning. In IJCAI , volume 7, 2586–2591.
Ross, S.; and Bagnell, D. 2010. Efﬁcient reductions for imitation
learning. In AISTATS , 661–668.
Shute, V . J. 2008. Focus on Formative Feedback. Review of Edu-
cational Research 78(1): 153–189.Srivastava, S.; Labutov, I.; and Mitchell, T. 2017. Joint Concept
Learning and Semantic Parsing from Natural Language Explana-
tions. In EMNLP 2017 , 1527–1536. ACL.
Suhr, A.; Yan, C.; Schluger, J.; Yu, S.; Khader, H.; Mouallem, M.;
Zhang, I.; and Artzi, Y . 2019. Executing Instructions in Situated
Collaborative Interactions. EMNLP 2019 .
Szlam, A.; Gray, J.; Srinet, K.; Jernite, Y .; Joulin, A.; Synnaeve,
G.; Kiela, D.; Yu, H.; Chen, Z.; Goyal, S.; Guo, D.; Rothermel, D.;
Zitnick, C. L.; and Weston, J. 2019. Why Build an Assistant in
Minecraft? ArXiv abs/1907.09273.
Tellex, S.; Gopalan, N.; Kress-Gazit, H.; and Matuszek, C. 2020.
Robots That Use Language. Annual Review of Control, Robotics,
and Autonomous Systems 3(1): 25–55.
Tellex, S.; Kollar, T.; Dickerson, S.; Walter, M. R.; Banerjee, A. G.;
Teller, S.; and Roy, N. 2011. Understanding Natural Language
Commands for Robotic Navigation and Mobile Manipulation. In
AAAI .
Thomason, J.; Murray, M.; Cakmak, M.; and Zettlemoyer, L. 2019.
Vision-and-Dialog Navigation. ArXiv abs/1907.04957.
Thomason, J.; Padmakumar, A.; Sinapov, J.; Walker, N.; Jiang,
Y .; Yedidsion, H.; Hart, J.; Stone, P.; and Mooney, R. J. 2020.
Jointly Improving Parsing and Perception for Natural Language
Commands through Human-Robot Dialog. The Journal of Arti-
ﬁcial Intelligence Research (JAIR) 67: 327–374.
Thomason, J.; Zhang, S.; Mooney, R.; and Stone, P. 2015. Learning
to Interpret Natural Language Commands through Human-Robot
Dialog. In Proceedings of the 24th International Conference on
Artiﬁcial Intelligence , IJCAI’15, 1923–1929. AAAI Press.
Thomaz, A. L.; and Breazeal, C. 2008. Teachable robots: Under-
standing human teaching behavior to build more effective robot
learners. Artiﬁcial Intelligence 172(6-7): 716–737.
Train, K. 2003. Discrete Choice Methods with Simulation . Cam-
bridge University Press.
Tukey, J. W. 1953. Section of mathematics and engineering: Some
selected quick and easy methods of statistical analysis. Transac-
tions of the New York Academy of Sciences 16(2 Series II): 88–97.
Udagawa, T.; and Aizawa, A. 2019. A Natural Language Corpus
of Common Grounding under Continuous and Partially-Observable
Context. AAAI 33: 7120–7127. ISSN 2159-5399.
van der Kleij, F.; Feskens, R.; and Eggen, T. 2015. Effects of Feed-
back in a Computer-Based Learning Environment on Students’
Learning Outcomes: A Meta-Analysis. Review of Educational Re-
search 85.
Wang, S. I.; Liang, P.; and Manning, C. D. 2016. Learning Lan-
guage Games through Interaction. ACL .
Wang, X.; Huang, Q.; C ¸ elikyilmaz, A.; Gao, J.; Shen, D.; Wang,
Y .-F.; Wang, W. Y .; and Zhang, L. 2019. Reinforced Cross-Modal
Matching and Self-Supervised Imitation Learning for Vision-
Language Navigation. In CVPR , 6629–6638.
Williams, E. C.; Gopalan, N.; Rhee, M.; and Tellex, S. 2018. Learn-
ing to Parse Natural Language to Grounded Reward Functions with
Weak Supervision. In ICRA , 4430–4436.
Xu, H.; Liu, B.; Shu, L.; and Philip, S. Y . 2019. BERT Post-
Training for Review Reading Comprehension and Aspect-based
Sentiment Analysis. In NAACL .
Zhou, L.; and Small, K. 2020. Inverse Reinforcement Learning
with Natural Language Goals. ArXiv abs/2008.06924.

Theodore R
Sumers
Mark K
Ho
Robert D
Hawkins
Karthik Narasimhan
Thomas L
Griffiths


Combining reward information from multiple sources.

Combining reward information from multiple sources
Dmitrii Krasheninnikovy
University of AmsterdamRohin Shahy
UC BerkeleyHerke van Hoof
University of Amsterdam
Abstract
Given two sources of evidence about a latent variable, one can combine the informa-
tion from both by multiplying the likelihoods of each piece of evidence. However,
when one or both of the observation models are misspeciﬁed , the distributions
will conﬂict. We study this problem in the setting with two conﬂicting reward
functions learned from different sources. In such a setting, we would like to retreat
to a broader distribution over reward functions, in order to mitigate the effects of
misspeciﬁcation. We assume that an agent will maximize expected reward given
this distribution over reward functions, and identify four desiderata for this setting.
We propose a novel algorithm, Multitask Inverse Reward Design (MIRD), and
compare it to a range of simple baselines. While all methods must trade off between
conservatism and informativeness, through a combination of theory and empirical
results on a toy environment, we ﬁnd that MIRD and its variant MIRD-IF strike a
good balance between the two.
1 Introduction
While deep reinforcement learning (RL) has led to considerable success when given an accurate
reward function [ 28,21],specifying a reward function that captures human preferences in real-world
tasks is challenging [ 1,6]. Value learning seeks to sidestep this difﬁculty by learning rewards from
various types of data, such as natural language [ 22], demonstrations [ 33,11], comparisons [ 6],
ratings [ 7], human reinforcement [ 18], proxy rewards [ 13], the state of the world [ 27], etc. Many
approaches to value learning aim to create a distribution over reward functions [14, 23, 24, 13].
I want
 ?
Alice’s past
behavior says
she wants
Figure 1: Alice says she wants avocado toast,
but her past actions imply she prefers cake to
toast. What should the agent believe about the
true reward?A natural idea is to combine the information from
these diverse sources. However, the sources may
conﬂict. For example, in Figure 1, Alice might
say that she wants the healthy avocado toast, but
empirically she almost always chooses to eat the
tasty cake. A reward learning method based on
natural language may conﬁdently infer that Alice
would prefer the toast, whereas a method based
on revealed preferences [ 25] might conﬁdently
infer that she wants cake. Presumably one of the
algorithms made a poor assumption: perhaps the
natural language algorithm failed to model that
Alice only said that she wants avocado toast to impress her friends. Alternatively, maybe Alice wants
to start eating healthy, but chooses the cake in moments of weakness, and the revealed preferences
algorithm failed to model this bias. While it is unclear what the “true” reward is, we do know that at
least one of the methods is conﬁdently wrong. We would like to defer on the question of what the
“true” reward is, and instead retreat to a position of uncertainty. Formally, we want a distribution over
the true reward function given the inferred rewards e1;e2.
Work done at UC Berkeley
yCorrespondence to dmitrii.krasheninnikov@student.uva.nl, rohinmshah@berkeley.edu
Presented at the NeurIPS 2019 workshop on Learning with Rich Experience: Integration of Learning Paradigms
We could apply Bayes Rule: p(je1;e2)/p()p(e1j)p(e2j). However, when rewards
conﬂict due to misspeciﬁcation, one of the likelihoods will be very low, and the posterior will be low
everywhere, likely leading to garbage after normalization [ 10]. While it is possible to mitigate this by
making the assumptions more realistic, e.g. by modeling human biases [ 26,19,9], it is very hard to
get a perfect model [3, 29]. So, we aim for a method that is robust to misspeciﬁcation.
What makes for a good reward combination method? To answer this question, we must know how
the resulting distribution will be used. As in Hadﬁeld-Menell et al. [14], we assume that an agent
maximizes expected reward, given that the agent can gather more information about the reward . In
this framework, an agent must balance three concerns: actively learning about the reward [ 24,20],
preserving its ability to pursue the true reward in the future when it knows more [ 30,17], and acting
to gain immediate reward. Each concern suggests different desiderata for a reward combination
method, which we formalize for rewards functions that are linear in features of the state.
There are several reasonable methods for combining reward functions based on the intuition that
we should put weight on any parameter values “in between” the parameters of the given reward
functions. These reward-space methods operate in the space of parameters of the reward. However,
this ignores any details about the environment and its features. We introduce a method called Multi-
task Inverse Reward Design (MIRD) that leverages the environment to infer which reward functions
are compatible with the behaviors “in between” the behaviors incentivized by the two input rewards,
making it a behavior-space method. Through a combination of theory and empirical results on a
simple environment, we evaluate how well the proposed methods meet our desiderata, and conclude
that MIRD is a good option when the reward distribution must be used to act, while its variant
MIRD-IF is better for increased robustness and a higher chance of support on the true reward.
2 Background
Markov Decision Process (MDP). A MDPMis a tupleM=hS;A;T;r;Hi, whereSis the set
of states,Ais the set of actions, T:SAS7! [0;1]is the transition probability function,
r:S7!Ris the reward function, and H2Z+is the ﬁnite planning horizon. We consider MDPs
where the reward is linear in features, and does not depend on action: r(s;) =Tf(s), where
are the parameters deﬁning the reward function and fcomputes features of a given state. A policy
:S7!A speciﬁes how to act in the MDP. A trajectory : (SA )is a sequence of states and
actions, where the actions are sampled from a policy and the states are sampled from the transition
functionT. We abuse notation and write f()to denoteP
s2f(s). The feature expectations (FE) of
policyare the expected feature counts when acting according to :F=E[f()]. The value of
a policyis given byV=E[P
s2r(s;)] =TF. The optimal policy is = maxV.
Reward learning by simulating the past (RLSP). Shah et al. [27] note that environments where
humans have acted are already optimized for human preferences, and so contain reward information.
The RLSP algorithm considers the current state s0as the ﬁnal state in a human trajectory =
(sH;aH;:::;s 0;a0)generated by the human with preferences . The RLSP likelihood p(js0)is
obtained by marginalizing out (sH;aH;:::s1;a1;a0), since onlys0is actually observed.
Inverse Reward Design (IRD). Inverse reward design (IRD) notes that since reward designers often
produce rewards by checking what the reward does in the training environment using an iterative
trial-and-error process, the ﬁnal reward they produce is likely to produce good behavior in the
environments that they used during the reward design process . So,eneed not be identical to ,
but merely provides evidence about it. Formally, IRD assumes that the probability of specifying
a giveneis proportional to the exponent of the expected true return in the training environment:
p(ej)/exp(>Fe). IRD inverts this generative model of reward design to sample from the
posterior of the true reward: p(je)/p(ej)p().
3 Uses of the reward distribution
So far, we have argued that we would like to combine the information from two different sources
into a single distribution over reward functions. But what makes such a distribution good? We take
a pragmatic approach: we identify downstream tasks for such a distribution, ﬁnd desiderata for a
reward distribution for these tasks, and use these desiderata to guide our search for methods.
2
3.1 Optimizing the reward
We follow the framework of Hadﬁeld-Menell et al. [14] in which an agent maximizes expected return
under reward uncertainty. In this setting, maximizing the return causes the agent to use its actions
both to obtain (expected) reward, and to gather information about the true reward in order to better
obtain reward in the future [ 14,31]. We might expect that an agent could ﬁrst learn the reward and
then act; however, this is unrealistic: we would be unhappy with a household robot that had to learn
how we want walls to be painted before it would bake a cake. The agent must act given reward
uncertainty, but maintain its ability to optimize any of the potential rewards in the future when it will
know more about the reward. Put simply, it needs to preserve its option value while still obtaining as
much reward as it can currently get.
We would like to evaluate our reward distribution by plugging it into an algorithm that maximizes
expected reward given reward uncertainty, but unfortunately current methods [ 14,31] are computa-
tionally expensive and only work in environments where option value preservation is not a concern.
So, we analyze reward learning and option value preservation separately.
Active reward learning. An agent that learns rewards while acting will need to select queries with
care. Active reward learning aims to pick queries that maximize the information gained about the
reward, or minimize the expected regret [ 24,20]3. We analyse how the reward distributions resulting
from different reward combination methods ﬁt as priors for the active reward learning methods.
Option-value preservation. Low impact agents [ 30,17,2] take fewer irreversible actions, and so
preserve their ability to optimize multiple reward functions. However, work in this area does not
consider the possibility of learning more information about the reward function, and so these methods
penalize entire plans that would have a high impact. In contrast, we would like our agent to pursue
impactful plans, but pause when option value would be destroyed, and get more reward information.
For our purposes, we start with the attainable utility preservation (AUP) method introduced by Turner
et al. [30]. AUP formalizes the impact of an action on a reward function as the change in the Q-value
as a result of taking that action (relative to a no-op action ?). For our purposes, we only care about
cases where we are no longer able to optimize the reward function, as opposed to ones where we are
better able to optimize the reward function. So, we use the truncated difference summary function
proposed by Krakovna et al. [17] that only counts decreases in Q-values and not increases:
Impact (s;a) =Er[max(Qr(s;?)Qr(s;a);0)]: (1)
Turner et al. [30] penalize the agent whenever it would cause too much impact, using a hyperparameter
to trade off task reward with impact. They plan using the reward RtaskImpact (s;a). However,
by penalizing impact in the reward function, the agent is penalized for any plan that would cause
impact, whereas we want our agent to start on potentially impactful plans, but stop once a particular
action would destroy option value. So, we only penalize impact during action selection:
(s) = max
a(Q(s;a)Impact (s;a)): (2)
3.2 Acting in a new environment
Often, we wish to use the reward function in the same environment as the one in which it was speciﬁed
or inferred. For example, a human expert may have demonstrated how to bake a cake in a particular
kitchen. Since we know at training time that the agent will be deployed in the same kitchen, the only
guarantee we need is that the agent’s behavior is correct. If two rewards lead to the same behavior in
this kitchen, we don’t care which one we end up using.
However, we may instead want to apply a reward function learned in one environment to a different
environment. In this case, there may be new options in the test environment that we never saw during
training. As a result, even if two rewards led to identical behavior in the training environment, they
need not do so in the test environment. Hence when the goal is to transfer the reward distribution to a
new environment, it is crucial to ensure that the distribution contains the true reward, instead of just
containing a reward that gives rise to the same behavior as the true one in the training environment.
3We speculate that future active reward learning methods will also have to ensure that the queries are relevant
andtimely : in our household robot example, this would prevent the robot from asking about wall paint color
when it is meant to be baking a cake.
3
4 Desiderata for the reward posterior
We discuss properties of the posterior that inform the choice of a reward combination algorithm for
different applications of the resulting reward distribution. We focus on two applications: option-value
preserving planning and active reward learning.
4.1 Desired properties for any subsequent application
All else equal, we want the true reward to have high probability in the reward distribution. One model
for misspeciﬁcation is that the weight for each feature has some chance of being “corrupted”, and the
chance of the feature being corrupted in one reward is independent from that of the other reward.
If the fraction of misspeciﬁed values is small, then if two reward vectors’ values are different at a
particular index, it is likely that one of the values is correct. Given this, it is desirable for the reward
distribution to assign signiﬁcant probability to every reward function s.t. the value of the weight for
each feature is in one of the values of that feature’s weight of the input rewards. Formally, we denote
the set of such reward parameters
indep pts=f2Rnji2fe1
i;e2
ig8i2f1;:::;ngg: (3)
Desideratum 1 (Support on indep pts).All2indep ptsshould be in the support of p(je1;e2).
Note that when we do not need to transfer the reward function to a new environment, Desideratum 1
can be made weaker. We only require that the distribution have support on reward functions that give
rise to the same behavior as the rewards in indep pts, and not necessarily on indep ptsitself.
In order for the agent to act, the reward distribution must provide information about useful behavior:
Desideratum 2 (Informative about desirable behavior) .Ife1ande2give rise to the same behavior,
most of the reward functions in the distribution should give rise to this behavior.
The strong version of this desideratum states that if the FEs of fe1;e2gare equal on some feature,
the FEs of this feature of anyreward in the support of the posterior are equal to those of fe1;e2g:
8i2f1;ng:
Fe1
i=Fe2
i=) 82supp(p(je1;e2)) :F
i=Fe1
i=Fe2
i
:(4)
The medium version considers the FEs as a whole instead of per-feature:
Fe1=Fe2=) 82supp(p(je1;e2)) :F=Fe1=Fe2: (5)
The weak version assumes that the FEs of all 2indep ptsare the same, rather than just e1ande2:

82indep pts:F=Fe1=Fe2
=) 82supp(p(je1;e2)) :F=Fe1:(6)
Des. 2 allows option-value preserving planning to do useful things instead of performing the no-op
action all the time. Furthermore, by restricting possible behaviors, Des. 2 simpliﬁes the job of
approaches that actively learn the reward function. If the reward distribution used by our household
robot always incentivizes using soap while washing dishes (because both input rewards did so), the
robot’s active learning algorithm does not need to query the human about whether or not to use soap.
Des. 2 might be problematic if both input rewards agree on an undesirable behavior w.r.t. a particular
feature. This presents a natural tradeoff between informativeness and the abilities to preserve option
value and actively learn arbitrary reward functions: the more informative we assume the input rewards
are about the desired behavior, the less robust we are to their misspeciﬁcation.
4.2 Desired properties for option value preservation
Suppose we would like to preserve the option value of pursuing two different behaviors. Option-value
preservation is more robust when the distribution over FEs induced by the reward distribution places
signiﬁcant and approximately equal weight on the two behaviors. Intuitively, the “weight” AUP puts
on preserving expected return attainable by a particular behavior is proportional to the probability of
that behavior in the reward distribution. So we would like the FEs of the input rewards to be balanced
across the reward distribution. Concretely:
4
Desideratum 3 (Behavior-space balance) .The FEs of the rewards sampled from the distribution
should correspond to behaviors arising from the input reward functions with a similar frequency:
p(F=Fe1)p(F=Fe2)jp(je1;e2): (7)
4.3 Desired properties for active reward learning
Given a reward vector we deﬁne the tradeoff point between the i-th and j-th features of the state as the
ratio between the reward vector values corresponding to these features: T
i;j=i
j. The true reward’s
tradeoff point may be in between the tradeoff points of the input rewards, and so we would like our
distribution to support all such tradeoff points.
Desideratum 4 (Support on intermediate feature tradeoffs) .The reward posterior should have
support on rewards that prescribe all intermediate tradeoff points between the states’ features.
In practice tradeoff points only affect the behavior to the extent allowed by the environment. We only
need to maintain allintermediate tradeoff points if we want to transfer to a new environment.
5 Multitask Inverse Reward Design
Inspired by Inverse Reward Design [ 13], we introduce a reward combination method Multitask
Inverse Reward Design (MIRD). We assume that the two input rewards meet the IRD assumption for
two different rewards, namely that the input rewards generate nearly optimal behavior in the training
environment according to the corresponding task rewards. A straightforward way to construct the
distribution for the combined reward function is to require the FEs of the rewards in the distribution to
be convex combinations of the FEs of the input rewards: F2fbFe1+(1b)Fe2jb2[0;1]g:
To satisfy this requirement, we propose a distribution that depends on the behavior (set of trajectories)
D=f1;:::;dgresulting from the agents optimizing e1ande2:
p(je1;e2) =Z
Dp(jD)p(Dje1;e2)dD (8)
Each of the reward functions fiji=f1;:::;dggused to generate the trajectories iinDis sampled
from the Bernoulli mixture over the input reward functions. We introduce a random variable b2[0;1]
that determines the probability of sampling i=e1to generate the trajectory i:
p(Dje1;e2) =Z
bBeta(bj1;2)nY
i=1
bp(ije1) + (1b)p(ije2)
db (9)
We sample from p(iji)using policies arising from soft value iteration. For p(jD), we set
to the reward learned by Maximum Causal Entropy Inverse RL (MCEIRL) [ 33]. Denoting the Dirac
delta as(), we havep(jD) =(argmaxp(jD)):Having deﬁned all relevant conditional
distributions, we can sample from the joint p(;b;D;ije1;e2)and the marginal p(je1;e2).
Theorem 1. The FEs arising from rewards in the support of the MIRD posterior are convex
combinations of FEs of the input reward vectors e1ande2:
82supp(p(je1;e2)) :F=Fe1+ (1)Fe2; 2[0;1]: (10)
Proof. See appendix A for full proof. Key to the proof is the fact that MCEIRL ﬁnds such that its
FEs match those of the soft value iteration expert who generated the trajectories [ 33]. In our case, the
FEs ofmatchD, which contains trajectories arising from e1ande2in various proportions.
Corollary 1.1. The average FEs of the reward functions in the posterior are:
Ep(je1;e2)[F] =Eb[bFe1+ (1b)Fe2] =Eb[b]Fe1+ (1Eb[b])Fe2: (11)
Corollary 1.2. Eq. 11 implies that if the FEs of the e1ande2w.r.t. a given feature are equal, the
FEs of all rewards in the distribution for that feature equal the FEs of e1ande2for that feature. Thus
MIRD satisﬁes the strong version of Des. 2 (informative about desirable behavior).
5
Corollary 1.3. When1=2the probability of sampling i=e1equals the probability of sampling
i=e2, and hence the probabilities of observing behaviors Fe1andFe2when optimizing a
reward function from the posterior are equal. Hence Des. 3 (behavior balance) is satisﬁed.
Theorem 1 allows us to bound the expected return true>F; 2supp(p(je1;e2))of the true
reward function truein terms of expected true returns of Fe1andFe2:
Theorem 2. The minimum expected true return following any of the rewards in the MIRD posterior
equals the minimum true return following one of the input rewards:
min
2supp(p(je1;e2))true>F= min
2fe1;e2gtrue>F: (12)
Proof. See Appendix B.
Theorem 2 is the best possible regret bound for following rewards in a distribution resulting from
reward combination: the worst the agent can do when following a reward from the distribution is no
worse than when just randomly choosing to follow one of the input reward functions.
Generally FEs arising from 2indep ptsnfe1;e2gare not convex combinations of Fe1andFe2,
so MIRD does not satisfy any version of Des. 1 (support on indep pts). In addition, when e1ande2
conﬂict on more than one feature, Des. 4 (support on intermediate feature tradeoffs) is not satisﬁed.
Independent features formulation. Since both Des. 1 and Des. 4 aim to ensure robustness to reward
misspeciﬁcation, one of our primary goals, we would like a variant of MIRD that satisﬁes them.
An algorithm that supports rewards that give rise to all potential behaviors arising from reward
functions in indep ptsis a simple modiﬁcation of MIRD. The only change required in the original
formulation of MIRD is sampling the reward vectors figgiving rise to the trajectories in Dfrom
indep ptsinstead of justfe1;e2g. Nowbis a vector of dimensionality jindep ptsj, sampled from
Dirichlet(bj1;:::;jindep ptsj). Eachiis sampled from a Multinoulli distribution over indep pts,
parameterized by b. We refer to this version of MIRD as MIRD independent features, or MIRD-IF.
6 Analysis
We ﬁrst introduce several baselines each of which does not meet some of the outlined desiderata, and
then analyze all reward combination methods in the context of our modiﬁed version of AUP.
6.1 Reward-space baselines
Additive. The additive reward combination method is commonly used in previous work utilizing
reward combination [ 8,27,16]. The reward posterior is simply p(je1;e2) =((e1+e2)),
whereis the hyperparameter used to balance the two input reward functions.
Gaussian. Here, we use the standard Bayesian framework, and assume that e1ande2are generated
fromwith Gaussian noise:
p(je1;e2)/p()p(e1j)p(e2j) =p()pN(e1j;2
1I)pN(e2j;2
2I);
where Iis the identity matrix, pNis the multivariate Gaussian probability density function, and
f1;2gare hyperparameters controlling the standard deviations. One can express higher trust in a
given input reward by lowering its . For our experiments, we use an uninformative Gaussian prior
(that is,!1 ) and set1=2= 1. The posterior is then p(je1;e2) =pN(je1+e2
2;1
2I).
Convex combinations of input reward vectors (CC-in). Consider a scenario where the input
reward functions are correctly speciﬁed for two different tasks. A natural reward-space analogue to
MIRD would be to consider a distribution over all convex combinations of the input reward vectors:
p(je1;e2) =Z
bBeta(bj1;2)(be1(1b)e2)db: (13)
Theorem 3. IfFe1=Fe2, the FEs of any reward in the support of the CC-in posterior equal
Fe1. Hence CC-in satisﬁes the medium version of Des. 2 (informative about desirable behavior).
Proof. See Appendix C.
6
We provide an example showing that CC-in does not satisfy the strong version of Des 2 in Appendix F.
Furthermore, CC-in does not satisfy Des. 1 (support on indep pts): consider combining rewards [0;1;1]
and[1;0;1]. Here= [0;0;1]2indep pts, but is not a convex combination of input rewards.
Convex combinations of 2indep pts(CC- indep pts).Instead of only considering convex com-
binations of the input rewards, we consider convex combinations of 2indep pts. The vector
b2Rjindep ptsjof weights for each convex combination is sampled from the Dirichlet distribution:
p(je1;e2) =Z
bDirichlet(bj1;:::;jindep ptsj)0
@jindep ptsjX
i=0biindep pts
i1
Adb: (14)
Theorem 4. If all FEs of rewards in indep ptsare equal, the FEs of any reward in the CC- indep pts
posterior equal the FEs of rewards in indep pts. Hence CC- indep ptssatisﬁes Des. 2 (weak).
Proof. See Appendix D.
Uniform points. Note that as1=:::=jindep ptsj!0, Eq. 14 essentially becomes p(je1;e2) =
Uniform (indep pts). It is helpful to analyze the properties of this uniform points distribution separately.
Table 1 summarizes the properties of MIRD, MIRD-IF, and the reward-space baselines.
Table 1: Performance of reward combination algorithms on the desiderata outlined in Section 4.
Support on
indep ptsInformative about desirable
behaviorBehavior
balanceSupport on intermediate
feature tradeoffs
(strong) (medium) (weak)
Additive 7 7 3 3 7 7
Gaussian 3 7 7 7 7 3
CC-in 7 7 3 3  7
CC-indep pts3 7 7 3  3
Uniform Points 3 7 7 3 3 7
MIRD 7 3 3 3 3 7
MIRD-IF 3 7 7 3 3 3
6.2 Analysis on a toy environment
Figure 2: The Cooking en-
vironment. Left: the grid-
world layout and current
state. Right: irreversible
actions for cooking. Mak-
ing cake or avocado toast in-
volves ﬁrst making dough
from the ﬂour, and then mak-
ing cake or toast; making
pasta uses ﬂour directly.To demonstrate the use of the desiderata, as well as how each method
performs on these desiderata, we developed a toy environment in
which to test reward combination methods. Due to space limitations,
we defer full qualitative and quantitative explanations to Appendix E
and report the broad results here.
Setup. We use the setting of AUP planning to demonstrate the im-
portance of support on indep pts, informativeness about desired the
behavior, and behavior balance desiderata. We obtain the reward pos-
terior by combining the input rewards using each of our seven reward
combination methods, and qualitatively evaluate the behavior of AUP
planning w.r.t. the posterior over ﬁve seeds. The input rewards are
either both speciﬁed by hand, or e1is learned with RLSP while e2is
hand-speciﬁed. We use the cooking environment described in Figure 2.
The states’ features indicate the number of jars of ﬂour, pieces of
dough, cakes, avocado toasts, and servings of pasta. At each state the
agent can move, cook, or do nothing. Cooking is possible when the
agent is to the left of ﬂour or dough.
Support on indep pts.We specify both input rewards to strongly
incentivise either cake or toast by setting e1
cake=e2
toast= 3;e1
toast=
e2
cake= 0, and to mildly incentivise pasta: e1
pasta=e2
pasta= 1. The
desired behavior is to preserve the ﬂour, as any one of the foods might
be desirable. This requires the distribution to place weight on rewards
7
such ascake=toast= 0;pasta= 1 which would make pasta. As expected, methods that have
support on indep ptsbehave as desired, and other methods do not.
Informativeness about desirable behavior (strong). In the setting above the two input rewards
lead the agent to make either cake or toast, but not pasta. So, a reward distribution meeting the strong
version of Des. 2 would only make cake or toast, and so the agent should make dough (which is
useful for both). Note that despite the same setup, this is exactly the opposite behavior than desired
previously, showing the tradeoff between informativeness and support on indep pts. As such, the
results are reversed.
Informativeness about desirable behavior (weak). We hand-specify one of the reward functions
to incentivise the agent to make cakes: e2
cake= 1;e2
toast=e2
pasta= 0.e1is inferred by RLSP with a
uniform prior over sT. Since the current state contains a cake, RLSP infers a positive reward for
cakes,e1
cake0:1, and near-zero rewards for the other features. Hence we have two rewards that
both incentivise the agent to make another cake. The desired behavior is to make the cake, and not
worry about preserving the ability to make pasta or toast as neither input reward cares about it. All
but the Gaussian method satisfy the weak informativeness desideratum, and so they all succeed. The
Gaussian method fails since it thinks
cakecould be negative.
Behavior balance. e2is speciﬁed to only reward toast, e2
toast= 1,e2
cake=e2
pasta= 0, ande1is
speciﬁed only reward cake: e1
cake>0,e1
toast=e1
pasta= 0. So we have two disagreeing rewards: one
incentivises cake, while the other incentivises toast. The desired behavior from AUP here is to make
dough as it is desirable for both the cake and the toast, but then to stop as the next step is unclear.
Ensuring that the reward for toast does not overwhelm the reward for cake requires behavior balance.
We varye1
cakebetween 0.01 and 0.7 to obtain many degrees of imbalance between toast and cake.
MIRD and MIRD-IF have the lowest frequency of the toast reward overwhelming the cake reward,
which indicates that MIRD and MIRD-IF are most behavior balanced. However, occasionally these
methods do nothing instead of making dough.
7 Related Work
Utility aggregation. Combining the speciﬁed reward functions is in many ways similar to combining
utility functions of different agents. Harsanyi’s aggregation theorem [ 15] suggests that maximizing a
ﬁxed linear combination of utility functions results in Pareto-optimal outcomes for correctly speciﬁed
utility functions. This corresponds to our Additive baseline.
Multi-task and meta IRL. Multi-task IRL assumes that the demonstrations originate from multiple
experts performing different tasks, and seeks to recover reward functions that explain the preferences
of each of the experts [ 4,5]. The IRL part of MIRD and MIRD-IF is conceptually similar to these
approaches: the trajectories in each demonstration set Dcan be seen as arising from an expert
executing different tasks. However, MIRD seeks to explain each Das if the trajectories arose from
asingle reward function . This resembles meta IRL, which learns an initialization for the reward
function parameters that is helpful for learning the rewards of new tasks [32, 12].
8 Limitations and future work
Summary. We analyze the problem of combining two potentially misspeciﬁed reward functions
into a distribution over rewards. We identify active reward learning and option value preservation
as two key applications for such a distribution, and determine four properties that the distribution
would ideally satisfy for these applications. We suggest some simple reward-space methods as well
as a behavior-space method, Multitask Inverse Reward Design (MIRD), which is grounded in the
behavior arising from the input rewards. MIRD works well for most applications, and MIRD-IF can
be used for greater robustness and increased likelihood that the true reward is in the distribution. If
the only application is to preserve option value, then the uniform points method also works well.
Future work. The primary avenue for future work is to evaluate the reward combination methods in
the context of active reward learning, and formalize Des. 4 (support on intermediate tradeoffs) in a
way most helpful for active learning. Another avenue for future work is scaling up MIRD to realistic
environments, where the reward function may be nonlinear.
8
References
[1]Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete
problems in ai safety. arXiv preprint arXiv:1606.06565 .
[2]Armstrong, S. and Levinstein, B. (2017). Low impact artiﬁcial intelligences. arXiv preprint
arXiv:1705.10720 .
[3]Armstrong, S. and Mindermann, S. (2018). Occam’s razor is insufﬁcient to infer the preferences
of irrational agents. In Advances in Neural Information Processing Systems , pages 5598–5609.
[4]Babes, M., Marivate, V ., Subramanian, K., and Littman, M. L. (2011). Apprenticeship learning
about multiple intentions. In Proceedings of the 28th International Conference on Machine
Learning (ICML-11) , pages 897–904.
[5]Choi, J. and Kim, K.-E. (2012). Nonparametric bayesian inverse reinforcement learning for
multiple reward functions. In Advances in Neural Information Processing Systems , pages 305–313.
[6]Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep
reinforcement learning from human preferences. In Advances in Neural Information Processing
Systems , pages 4299–4307.
[7]Daniel, C., Viering, M., Metz, J., Kroemer, O., and Peters, J. (2014). Active reward learning. In
Robotics: Science and Systems .
[8]Desai, N., Critch, A., and Russell, S. J. (2018). Negotiable reinforcement learning for pareto
optimal sequential decision-making. In Advances in Neural Information Processing Systems ,
pages 4713–4721.
[9]Evans, O., Stuhlmüller, A., and Goodman, N. D. (2016). Learning the preferences of ignorant,
inconsistent agents. In AAAI , pages 323–329.
[10] Frazier, D. T., Robert, C. P., and Rousseau, J. (2017). Model misspeciﬁcation in abc: Conse-
quences and diagnostics. arXiv preprint arXiv:1708.01974 .
[11] Fu, J., Luo, K., and Levine, S. (2018). Learning robust rewards with adversarial inverse
reinforcement learning. In International Conference on Learning Representations .
[12] Gleave, A. and Habryka, O. (2018). Multi-task maximum entropy inverse reinforcement
learning. arXiv preprint arXiv:1805.08882 .
[13] Hadﬁeld-Menell, D., Milli, S., Abbeel, P., Russell, S. J., and Dragan, A. (2017). Inverse reward
design. In Advances in Neural Information Processing Systems , pages 6765–6774.
[14] Hadﬁeld-Menell, D., Russell, S. J., Abbeel, P., and Dragan, A. (2016). Cooperative inverse
reinforcement learning. In Advances in neural information processing systems , pages 3909–3917.
[15] Harsanyi, J. C. (1955). Cardinal welfare, individualistic ethics, and interpersonal comparisons
of utility. Journal of political economy , 63(4):309–321.
[16] Kolter, J. Z. and Ng, A. Y . (2009). Near-bayesian exploration in polynomial time. In Proceedings
of the 26th Annual International Conference on Machine Learning , pages 513–520. ACM.
[17] Krakovna, V ., Orseau, L., Kumar, R., Martic, M., and Legg, S. (2018). Penalizing side effects
using stepwise relative reachability. arXiv preprint arXiv:1806.01186 .
[18] MacGlashan, J., Ho, M. K., Loftin, R., Peng, B., Wang, G., Roberts, D. L., Taylor, M. E., and
Littman, M. L. (2017). Interactive learning from policy-dependent human feedback. In Proceedings
of the 34th International Conference on Machine Learning-Volume 70 , pages 2285–2294. JMLR.
org.
[19] Majumdar, A., Singh, S., Mandlekar, A., and Pavone, M. (2017). Risk-sensitive inverse
reinforcement learning via coherent risk models. In Robotics: Science and Systems .
[20] Mindermann, S., Shah, R., Gleave, A., and Hadﬁeld-Menell, D. (2019). Active inverse reward
design. In International Conference on Machine Learning .
9
[21] OpenAI (2018). Openai ﬁve. https://blog.openai.com/openai-five/ .
[22] Radlinski, F., Balog, K., Byrne, B., and Krishnamoorthi, K. (2019). Coached conversational
preference elicitation: A case study in understanding movie preferences. In Proceedings of the
Annual SIGdial Meeting on Discourse and Dialogue .
[23] Ramachandran, D. and Amir, E. (2007). Bayesian inverse reinforcement learning. In IJCAI ,
volume 7, pages 2586–2591.
[24] Sadigh, D., Dragan, A. D., Sastry, S., and Seshia, S. A. (2017). Active preference-based learning
of reward functions. In Robotics Science and Systems .
[25] Samuelson, P. A. (1938). A note on the pure theory of consumer’s behaviour. Economica ,
5(17):61–71.
[26] Shah, R., Gundotra, N., Abbeel, P., and Dragan, A. (2019a). On the feasibility of learning,
rather than assuming, human biases for reward inference. In International Conference on Machine
Learning , pages 5670–5679.
[27] Shah, R., Krasheninnikov, D., Alexander, J., Dragan, A., and Abbeel, P. (2019b). Preferences
implicit in the state of the world. In International Conference on Learning Representations .
[28] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrit-
twieser, J., Antonoglou, I., Panneershelvam, V ., Lanctot, M., Dieleman, S., Grewe, D., Nham,
J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and
Hassabis, D. (2016). Mastering the game of go with deep neural networks and tree search. Nature ,
529:484–503.
[29] Steinhardt, J. and Evans, O. (2017). Model mis-speciﬁcation and inverse reinforcement learning.
[30] Turner, A. M., Hadﬁeld-Menell, D., and Tadepalli, P. (2019). Conservative agency via attainable
utility preservation. arXiv preprint arXiv:1902.09725 .
[31] Woodward, M., Finn, C., and Hausman, K. (2019). Learning to interactively learn and assist.
arXiv preprint arXiv:1906.10187 .
[32] Xu, K., Ratner, E., Dragan, A., Levine, S., and Finn, C. (2018). Learning a prior over intent via
meta-inverse reinforcement learning. arXiv preprint arXiv:1805.12573 .
[33] Ziebart, B. D. (2010). Modeling purposeful adaptive behavior with the principle of maximum
causal entropy. PhD thesis, Carnegie Mellon University .
10
A Proof of Theorem 1
Dis a set of trajectories each of which arises either from optimizing e1or optimizinge2. The fraction
of trajectories in Dthat result from e1isb, and the fraction of trajectories arising from e2is1b.
Hence the FEs of Dare
FD=bFe1+ (1b)Fe2: (15)
p(jD) =(argmaxp(jD))entails that each sample is the MCEIRL reward
= argmaxp(jD). MCEIRL ﬁnds a reward vector such that its FEs match those of the
soft value iteration expert who generated the trajectories [ 33]. Hence each sampled gives rise to
FEsFmatchingD. This and Eq. 15 together imply that the FEs arising from optimizing any
2supp(p(je1;e2))with soft value iteration are a convex combination of the FEs of the input
rewards:
82supp(p(je1;e2)) :F=bFe1+ (1b)Fe2; b2[0;1]: (16)
B Proof of Theorem 2
By Theorem 1, the FEs arising from any in the posterior are a convex combination of the FEs
of the input rewards. Hence ﬁnding in the support of p(je1;e2)that minimizes true>Fis
equivalent to ﬁnding 2[0;1]that minimizes
true>(Fe1+ (1)Fe2) =true>Fe1+ (1)true>Fe2:
Since the true returns when following policies arising from the input reward functions true>Fe1and
true>Fe2are scalars, the term above is minimized with = 1whentrue>Fe1<true>Fe2,
and= 0otherwise.
C Proof of Theorem 3
Since the feature expectations (FEs) of the input rewards are equal, the return of any mixture reward
be1(1b)e2for those FEs isFe1(be1(1b)e2). Suppose there exist FEs F0that result in a
higher expected return for the mixture reward:
F0(be1(1b)e2)>Fe1(be1(1b)e2) (17)
For this to be true F0must result in a higher return than Fe1for eithere1ore2. This is a
contradiction, since Fe1already results in the highest return for both input rewards. Hence Fe1
corresponds to the optimal behavior for any mixture reward be1(1b)e28b2[0;1].
D Proof of Theorem 4
Analogous to the proof of Theorem 3 (Appendix C).
E Analysis details
In this appendix we provide more details and explanations regarding the behavior of the various
reward combination methods in the Cooking environment.
E.1 Support on indep pts
We hand-specify both input reward functions to strongly incentivise either cake or toast by setting
e1
cake=e2
toast= 3;e1
toast=e2
cake= 0, and to mildly incentivise pasta: e1
pasta=e2
pasta= 1. The
desired behavior from AUP here is to preserve the ﬂour, as any one of the foods requiring it might be
desirable.
11
The Additive method returns a single reward function that incentivises the agent to make both cake
and toast. AUP applied to one reward function simply optimizes it without needing to preserve option
value, so here it makes either cake or toast (and does not preserve the ﬂour).
The CC-in method does not preserve the ﬂour, choosing to make dough instead. Because of the
high values of e1
cakeande2
toast, optimizing any convex combination of fe1;e2gleads the agent to
either make cake or toast, and so AUP infers that it does not need to preserve the ability to make
pasta. MIRD chooses to make dough for a similar reason: since all trajectories in Darise from
optimizinge1ore2, each trajectory demonstrates the agent making either cake or toast, and no
trajectory demonstrates it making pasta. So no reward inferred from Dincentivises the agent to make
pasta, and hence AUP does not preserve the ability to make pasta.
All methods whose reward samples sometimes give rise to the behaviors arising from 2indep pts
preserve the ﬂour. This is because indep ptscontainss.t. optimizing it leads the agent to make pasta:
cake=toast= 0;pasta= 1. As MIRD-IF, Gaussian, uniform points, and CC- indep ptsmethods
all have substantial probability mass on rewards that incentivise making pasta, AUP applied to the
reward samples from these methods preserves the ability to make pasta.
E.2 Informativeness about desirable behavior (strong).
In the setting above the two input rewards lead the agent to make either cake or toast, but not pasta.
Here a reward posterior meeting the strong version of Des. 2 would only give rise to reward functions
that incentivise the agent to make either cake or toast. Note that despite the same setup, this is exactly
the opposite behavior than desired previously, showing the tradeoff between informativeness and
support on indep pts. As such, the results are reversed.
We see that MIRD-IF, Gaussian, uniform points, and CC- indep ptsmethods do not meet Des. 2
(strong), since they all have substantial probability mass on rewards that incentivise making pasta.
Because of this AUP preserves the ability to make pasta, and does not lead the agent to make dough.
On the other hand, MIRD, Additive, and CC-in methods never give rise to rewards incentivising
the agent to make pasta, and AUP applied to these reward distributions results in making dough.
Although Additive and CC-in do not support reward functions that lead the agent to make pasta in
this particular case, they do not meet the strong version of Des. 2 (shown in Appendix F).
E.3 Informativeness about desirable behavior (medium & weak)
We hand-specify one of the reward functions to incentivise the agent to make cakes: e2
cake= 1;e2
toast=
e2
pasta= 0.e1is inferred by RLSP with a uniform prior over sT. The current state contains a cake
which was likely made by a human, so RLSP infers a positive reward for cakes, e1
cake0:1, and
near-zero rewards for the other features. Hence we have two agreeing rewards that both incentivise
the agent to make another cake. The desired behavior from AUP planning here is to make the cake,
and not worry about preserving the ability to make pasta or toast as neither input reward cares about
it.
The Gaussian posterior fails to make the cake. The distribution over
cakehere is a Gaussian with
0:55and0:71. The posterior assigns substantial probability to
cakebeing negative, and
AUP planning avoids making the cake to preserve the ability to optimize rewards that punish cakes.
Here each reward vector in indep ptsleads the agent to make cake. Since every method except
Gaussian satisﬁes at least the weak version of Des. 2, all reward functions in the support of their
posteriors lead the agent to make cake. Because of this AUP infers that it is not important to preserve
the ability to make pasta or toast, and successfully makes cakes when applied to rewards sampled
from these posteriors.
E.4 Behavior balance
To better understand robustness of the reward combination methods to imbalanced input rewards, we
analyze the performance of our methods by varying the extent of reward imbalance and the degree to
which AUP seeks to preserve option value.
12
The second input reward e2is hand-speciﬁed to only reward toast, e2
toast= 1,e2
cake=e2
pasta= 0. The
ﬁrst rewarde1is hand-speciﬁed as well: e1
toast=e1
pasta= 0. We varye1
cakebetween 0.01 and 0.7 to
obtain many degrees of imbalance between toast and cake. So the input rewards disagree whether
to make toast or cake, and agree that pasta is irrelevant. Hence the desirable action from AUP here
would be to preserve the ability to choose between cake and toast, as either could be desirable. In
addition to varying e1
cake, we vary the weight of the AUP penalty .
Figure 3: Most common outcomes of AUP applied to the six reward posteriors over 5 seeds. Pink
corresponds to AUP failure for more than half of the seeds: e2
toastoverwhelmse1
cakeand the agent
makes toast. Purple corresponds to AUP success (AUP makes dough) for more than half of the seeds.
Yellow corresponds to the middle-ground outcome of doing nothing in more than half of the seeds.
The hyperparameters of the Beta and the Dirichlet distributions are set to 0.5.
Additive always fails and makes toast as it has support on a single reward function that prioritizes toast.
Our results for the other six reward combination methods are shown in Figure 3. We observe that
MIRD and MIRD-IF are signiﬁcantly more robust to imbalanced input rewards than any reward-space
method. However, sometimes MIRD and MIRD-IF lead to AUP doing nothing instead of making
dough. This is explained by the fact that MIRD and MIRD-IF use MCEIRL, which sometimes
outputs negative rewards. For example, consider the case where in the process of generating a MIRD
sample we generate 10 trajectories all of which demonstrate making toast. MCEIRL with these
trajectories might output a positive reward on toast and a negative reward on cake and dough (instead
of positive reward for toast and zero for cake and dough), even if a zero-centered Gaussian prior is
used. These negative rewards on dough sometimes lead AUP to avoid making dough.
Furthermore, we observe that MIRD is more robust than MIRD-IF, and CC-in is more robust than
CC-indep pts. This is because in this case support on rewards giving rise to behaviors arising from
2indep ptsmakes the reward distributions less balanced. In particular, denoting the value of e1
cake
we vary asx,
indep pts
toast, cake =8
><
>:toast, cake = [0;0];
toast, cake = [0;x];
toast, cake = [1;0];
toast, cake = [1;x]9
>=
>;:
Here only the second reward vector incentivises the agent to make cake, while both the third and
the fourth rewards incentivise the agent to make toast. This slight imbalance of indep ptsin favor of
making toast lead to CC- indep ptsand MIRD-IF posteriors being more likely to cause AUP to make
toast instead of preserving the ability to choose between cake and toast.
Finally, we observe that the Gaussian method is roughly equally robust to reward imbalance across
the whole range of e1
cake. This is because unlike the other methods, the standard deviation of the
Gaussian posterior does not depend e1ande2, but only depends on hyperparameters 1and2,
which are ﬁxed to 1.
13
F Additive and CC-in do not satisfy Des. 2 (strong)
s2
s3 s1
s4f(s2) = [1;0;1]
f(s3) = [0;1;1]
f(s4) = [0:9;0:9;0]f(s1) = [0;0;0]
Figure 4: Toy reward combination problem. The aim is to combine reward vectors e1= [1;0;0]and
e2= [0;1;0]. The MDP is as follows. The starting state is s1. Froms1the agent must irreversibly
and deterministically transition to s2,s3, ors4, and stay there indeﬁnitely. The features of each state
sare given by f(s).
Additive. Consider the reward combination scenario in Figure 4. Maximizing e1ore2an agent
would correspondingly end up in s2ors3, both of which have the third feature equal 1. However,
maximizing =e1+e2would lead the agent to s4, where the value of the third feature is 0.
CC-in. Again consider the reward combination task in Figure 4. Maximizing e1ande2separately
would correspondingly lead to choosing s2ands3, both states having the third feature equal 1.
However, maximizing =e1+e2
2= [0:5;0:5;0], a convex combination of the input rewards, would
lead to the agent choosing s4where the value of the third feature equals 0.
As the Additive and CC-in reward posteriors have support on rewards giving rise to behaviors different
from the behaviors of the input rewards w.r.t. the third feature of the toy MDP, neither method meets
the strong version of Des. 2 (informative about desirable behavior).
14

Dmitrii Krasheninnikov
Rohin Shah
Herke van Hoof