Not a paper, but I find Chris Olah’s interview on the 80,000 Hours podcast super inspiring

This podcast covers a significant chunk of work in understanding neural networks, including <@circuits@>(@Circuits Thread@) and <@multimodal neurons@>(@Multimodal Neurons in Artificial Neural Networks@), as well as high-level thoughts such as [advantages of neural net interpretability over neuroscience](http://colah.github.io/notes/interp-v-neuro/) and <@why larger models may be more interpretable@>(@Chris Olah’s views on AGI safety@). Some interesting points I haven’t made in this newsletter before:

1. Interpretability as a field is fractured into several different mini-paradigms. The author’s paradigm might be described as “mechanistic interpretability”, where you try to “fully understand” the neural network from the ground up. An ML-based paradigm is interested in defining good “interpretability metrics” that can then be optimized. An HCI-based paradigm is interested in developing techniques that show good results based on user evaluations (e.g. people can better predict network outputs).

2. Scaling up mechanistic interpretability does seem possible, because (a) as models get larger their features plausibly get crisper and easier to understand, and (b) there are motifs (such as equivariance in curve circuits) that allow you to reduce the number of neurons you have to understand by over an order of magnitude. However, neurons can be _polysemantic_, where they encode multiple features at once; this could pose a significant challenge for mechanistic interpretability. (While current features encoded in polysemantic neurons will probably become crisper as models scale up, we might expect that the scaled up models will have new polysemantic neurons that encode multiple more abstract features.)

3. One aesthetically pleasing aspect of the mechanistic interpretability approach is that, in the world where we succeed, humans could plausibly “keep up” with the neural nets and understand these advanced concepts that the networks have, rather than living happy lives but being unable to comprehend what is going on in the world around them. See also [Using Artificial Intelligence to Augment Human Intelligence](https://distill.pub/2017/aia/).

You may also want to check out [this followup podcast](https://80000hours.org/podcast/episodes/chris-olah-unconventional-career-path/) in which Chris talks about his unconventional career path.


What are you optimizing for? Aligning Recommender Systems with Human Values.

What are you optimizing for?
Aligning Recommender Systems with Human Values
Jonathan Stray1Steven Adler1Dylan Hadﬁeld-Menell2 3
Abstract
We describe cases where real recommender sys-
tems were modiﬁed in the service of various hu-
man values such as diversity, fairness, well-being,
time well spent, and factual accuracy. From this
we identify the current practice of values engi-
neering: the creation of classiﬁers from human-
created data with value-based labels. This has
worked in practice for a variety of issues, but
problems are addressed one at a time, and users
and other stakeholders have seldom been involved.
Instead, we look to AI alignment work for ap-
proaches that could learn complex values directly
from stakeholders, and identify four major direc-
tions: useful measures of alignment, participatory
design and operation, interactive value learning,
and informed deliberative judgments.
1. Introduction
Recommender systems are AI systems that present users
with a tailored set of items based on factors such as past
user behavior, user attributes, and features of the underly-
ing items. They rank huge numbers of items in order to
determine which content each user sees or interacts with on
social media feeds, video platforms, new aggregators, and
in online stores.
Classically, recommender designers have framed their goal
as providing “relevant” items to each user. As these tech-
nologies have rolled out into society, it has become clear that
the appropriate goals are often highly context-dependent,
nuanced, and consequential. Previous work has argued for
the incorporation of additional objectives including the pro-
motion of accurate information (St ¨ocker, 2019), fairness
(Barocas et al., 2019), and diversity of content (Castells
1Partnership on Artiﬁcial Intelligence, San Francisco,
California, USA2Department of Electrical Engineering and
Computer Science, UC Berkeley, California, USA3Center
for Human-Compatible Artiﬁcial Intelligence, Berkeley, Cal-
ifornia, USA. Correspondence to: Dylan Hadﬁeld-Menell
<dhm@eecs.berkeley.edu >.et al., 2015), and the reduction of addiction (Hasan et al.,
2018)(Andreassen, 2015) and polarisation (Benkler et al.,
2018). These represent values-driven decisions about what
‘we,’ the relevant stakeholders, do and don’t want recom-
mended and how items should be ranked.
There are further, less-explored opportunities for values-
driven design. A speciﬁcation for a recommender system
could include, e.g., that a news recommendation system to
support values like informedness, autonomy, inclusiveness,
participation, decentralization, representation, deliberation,
and tolerance (Helberger, 2019) or that social media recom-
menders “provide positive incentives for encouraging more
constructive conversation” (Wagner, 2019). Recommenders
for books, music and video might want to promote cultural
exploration and ensure that emerging artists gain exposure
(Mehrotra et al., 2018). An online store may want its product
recommendations to encourage the sale of products with a
smaller carbon footprint (Rolnick et al., 2019) or otherwise
support social and environmental goals. Recommenders
have already been built around high-level goals like improv-
ing user well-being (Khwaja et al., 2019)(Stray, 2020) and
even proposed as an aid to self-actualization (Knijnenburg
et al., 2016).
In this paper, we propose that recommender system design
can draw from the study of the value alignment problem :
the problem of ensuring that an AI system’s behavior aligns
with the values of the principal on whose behalf it acts
(Hadﬁeld-Menell & Hadﬁeld, 2019). In the correspond-
ingrecommender alignment problem , the objective is to
align recommendations with the goals of users, system de-
signers, and society as a whole. In Section 2, we ground
the recommender alignment problem with case studies of
how the builders of large recommendation systems have
responded to domain-speciﬁc challenges. We observe a
common three phase approach to alignment: 1) relevant
categories of content (e.g., clickbait) are identiﬁed ; 2) these
categories are operationalized as evolving labeled datasets;
and 3) models trained off this data are used to adjust the
system recommendations. In Section 3, we draw from AI
alignment work to propose high-level approaches to the de-
sign of recommendation systems whose output embodies
human values in a more nuanced, participatory and adapt-
Aligning Recommender Systems
able way. We do not attempt to catalog the ethical issues
surrounding recommenders (see e.g. (Milano et al., 2019a))
or to say how various values should be prioritized. Rather,
we are interested in developing affordances that allow high-
level, inclusive control of what a recommendation system
recommends.
2. Case Studies
This section aims to ground our discussion of higher-level
approaches to recommender alignment in concrete examples
of diverse ways that large commercial recommender systems
have been modiﬁed to be “better.”
2.1. Identifying unwanted content
Clickbait is one of the earliest recognized examples of rec-
ommender misalignment. Early recommender systems were
often optimized to show the user the items they would be
most likely to click on, which rewarded items with attention-
grabbing titles but not necessarily quality content. In re-
sponse, system designers identiﬁed alternate metrics that
are harder to game and incorporated them into system train-
ing. For example, “dwell time”, the amount of time a user
spends on a page before coming back to recommender re-
sults, has been used as ranking and personalization signal at
many companies including Yahoo (Yi et al., 2014) and Face-
book(Khalid El-Arini & Joyce Tang, 2014). Other simple
metrics, such at the click-to-share ratio on Facebook posts
(Khalid El-Arini & Joyce Tang, 2014), have also been used
in practice to suppress clickbait.
However, hand-designed metrics are typically unable to
capture the complexity of even relatively simple and un-
controversial concepts. Most large production systems
today (including Flipboard (Cora, 2017) and Facebook
(Peysakhovich & Hendrix, 2016)) instead train classiﬁers
on human-labelled clickbait and use them to measure and
decrease the prevalence of clickbait within the system. Rec-
ommenders that use these learned classiﬁers are more ef-
fective than those that use only hand-designed metrics. As
we discuss below, we attribute this to the combination of
a greater quantity of human feedback with a more ﬂexible
model.
There are a number of other types of content that one might
not want to recommend broadly, such as hateful or harassing
posts, sexual content, posts that advocate violence, or mis-
information. These are too complex to detect with simple
metrics derived from user behavior. Their precise deﬁnition
is politically contested, and the decision to remove or down-
rank this content must balance potential harm, local laws,
and freedom of expression in complex, context-sensitive
ways (York & Zuckerman, 2019). Hateful or harassing
posts are challenging to identify by machine because ofthe nuances of language and context but sometimes signiﬁ-
cant is available, for example if a comment occurs within
a larger online discussion. A “toxicity” model created by
Jigsaw, trained on human-labelled examples, has been tested
at Twitter as part of a “healthy conversations” metric (Wag-
ner, 2019), and used by the New York Times for comment
triage prior to human review (Etim, 2017). However, such
classiﬁers can misunderstand the signiﬁcance of utterances
in race- and gender-biased ways (Salganik & Lee, 2020).
Mis- and disinformation is a much harder problem for auto-
mated detection because it may require open-ended research,
e.g. verifying a video by identifying and speaking to a wit-
ness (Silverman, 2020). Nonetheless, major platforms in-
cluding Facebook (Tessa Lyons, 2018) now employ trained
models in various aspects of their counter-misinformation
efforts. These typically operate at the level of classifying
domains, users, or near-duplicates of previously debunked
items (Cora, 2017) (Facebook, 2020). The ﬁnal source of
truthfulness is human ratings of individual articles, photos,
and videos. At Facebook this information is provided both
by a network of professional fact-checking organizations
and by users who ﬂag content as “false news” (Thompson,
2018).
2.2. Promoting desirable outcomes
More recently, recommender system builders have at-
tempted to identify content that furthers particular values-
based goals. Goals addressed in practice include healthy
public conversations, exposure for little-known creators or
sellers, and the well-being of social media users. Rather than
just identifying more or less desirable items, these richer
conceptions of recommender alignment envision the support
of positive values. Recommender systems could support
democratic values in a variety of ways (Helberger, 2019),
and encourage well-being (Khwaja et al., 2019), personal
self-actualization (Knijnenburg et al., 2016), and human
ﬂourishing (Rodriguez & Watkins, 2009).
Although the empirical evidence for recommender systems’
role in creating “ﬁlter bubbles” is weak (Bruns, 2019), most
discussions of recommenders and the public sphere hold
diversity of viewpoint as an essential value (Helberger et al.,
2018). Such studies typically consider diversity with respect
to marginalized communities or left-right politics, but there
are many other kinds of content diversity that recommender
systems could promote such as topic or cultural diversity,
and various concepts of representative fairness (Gao & Shah,
2020).
Spotify is notable for elaborating on the fairness and diver-
sity issues faced by a music platform. Their recommenda-
tion system is essentially a two-sided market, since artists
must be matched with users in a way that satisﬁes both, or
neither will stay on the platform. Recommender systems
Aligning Recommender Systems
(and human cultural markets in general) often suffer from
“superstar economics” where a small number of stars control
the majority of the attention (Mehrotra et al., 2018). The
company has also discussed the representation of different
demographics, and the value of musical exploration, lead-
ing to a domain-speciﬁc deﬁnition: a playlist is “fair” if it
contains songs from artists at different levels of popularity
(Lalmas-Roelleke, 2019). This is a concrete example of
the management of a multi-stakeholder recommendation
system, a central topic in recommender alignment which is
starting to be more explored (Abdollahpouri et al., 2019).
Especially when ﬁltering social media and news content,
recommenders are key mediators in public discussion. Twit-
ter has attempted to create “healthy conversation” metrics
with the goal to “provide positive incentives for encouraging
more constructive conversation” (Wagner, 2019). One con-
ception of these metrics includes shared attention, shared
reality, variety and receptivity, and a prototype has been
built with models trained on human labels (Gadde & Gasca,
2018). The Dutch FG Mediagroep modiﬁed their news rec-
ommender to increase the consumption of ”timely and fresh”
content, as well as overall coverage, deﬁned as the propor-
tion of articles that were recommended to any user(Lu et al.,
2020)
In the context of a mobile health app, researchers at Telefon-
ica have shown that personalizing recommendations based
on the user’s personality type, e.g. recommending social
activities to extroverts, could result in a meaningful im-
provement in well-being outcomes (Khwaja et al., 2019).
Facebook modiﬁed several of its recommendation systems
in 2017-2018 to include consideration of ”meaningful so-
cial interactions,” an explicit proxy for well-being (Stray,
2020). Such an intervention has potentially large effects
because there are so many Facebook users, but no outcome
information has so far been published.
2.3. Characterizing the State-of-the-Art
These case studies illustrate a common pattern in current
recommender efforts: identifying and operationalizing ab-
stract concepts like “healthy conversation” or “clickbait”,
then adjusting the recommendation algorithm to change the
observed prevalence of these concepts.
Identiﬁcation: This is the phase where system designers
become aware of a negative outcome associated with the sys-
tem and identify a concept associated with it. For example,
identifying that users are getting drawn into misrepresented
videos for short periods of time and developing a deﬁnition
of ‘clickbait’.
Operationalization: A concrete procedure is developed
to identify instances of the abstract concept in the recom-
mender system. Although this can be done through manualreview, most systems rely on some form of machine learning
in order to scale. These models are typically trained from
an evolving dataset where labels are provided by human
workers or observable user behavior (e.g., ﬂagging spam,
responses to user surveys).
Adjustment: In this step, system designers modify the
recommender system in order to increase or decrease the
prevalence of the desired concept, as measured by the clas-
siﬁcation process deﬁned in step 2. This can be done in a
number of ways, most commonly: 1) At training: Adding
concept prevalence as an additional training objective in
a multi-objective training setup, as in (Zhao et al., 2019);
2)At launch: Tracking concept prevalence as a top-line
metric and preferentially launching changes that improve
it; 3) At serving: Directly add or remove items from the
recommendation pool or change their rankings based on
their classiﬁcation, possibly after manual review.
3. Higher-level approaches to recommender
alignment
A major problem with current practice is that it is funda-
mentally reactive. Problems are addressed one at a time,
often after they are widespread. This is most troublesome
when addressing one problem causes a new and harder-to-
quantify problem, e.g. moving from optimizing for clicks to
optimizing for time-in-app may reduce clickbait but could
increase addiction. Even when there’s no risk of creating
new problems, this approach simply cannot scale to handle
the interests of billions of people embedded in millions of
communities. Conducting detailed user research, designing
new metrics, creating appropriate data sets, and training clas-
siﬁcation models is an iterative process that takes months if
not years.
Moreover, the bottleneck in the problem identiﬁcation step
implies that only the problems most visible to system design-
ers get addressed. This privileges issues that affect system
designers and their social circles. Even more troubling is
that it privileges the system designers’ moral and ethical
views. We wish to address the problem of encoding appro-
priate values into recommender systems in a more general,
participatory, and scalable way.
3.1. Useful deﬁnitions and measures of alignment
There is no single deﬁnition of value that a recommendation
system should adhere to. This is in part a philosophical
problem, in part the reality of political pluralism, and in part
because value depends tremendously on context (Gabriel,
2020). But there are still many values that are widely shared,
and useful instruments to evaluate human outcomes.
Suites of metrics for “well-being” are used in public policy
(O’Donnell et al.) and many of those most applicable to AI
Aligning Recommender Systems
systems have been collected into an IEEE standard for well-
being assessment (Schiff et al., 2020). Metrics for other
high-level values, such as agency, have been developed
in ﬁelds such as development economics (Alkire, 2008).
These assessments rely on a combination of self-reported
subjective states (“I am tired”), self-reported objective states
(“I went to work today”), and statistics (“The unemployment
rate is 5.4%”) (Stiglitz et al., 2018).
Many other metrics have been developed to evaluate recom-
menders, but that leaves open the question of which mea-
sures are most useful for value alignment in each domain.
Standard evaluation data sets or protocols for recommenders
in a particular domain will greatly accelerate progress on
recommender alignment.
3.2. Participatory recommenders
The operators of recommender systems have often worked
in a reactive mode, responding to issues after they are iden-
tiﬁed, and in the way that makes the most sense to them. It
would be better if users could be effectively involved in the
initial design of the system, had more control over routine
operation, and good ways to escalate new issues.
Most real recommender systems must trade off between
users or consumers, producers or sellers, and the recom-
mender operator (Abdollahpouri et al., 2019). Recommen-
dations may also affect non-users in a variety of ways, such
as product recommendations that have adverse environmen-
tal consequences or driving directions that create congestion
in otherwise calm neighborhoods (Kulynych et al., 2020).
There sorts of issues are related to the classic economic
problems of externalities and public goods, and are ex-
plicitly considered in multi-stakeholder recommendation
research (Milano et al., 2019b). Users also have prefer-
ences around what recommendations other users see, such
as when a someone ﬂags something as misinformation, or
when people work together to up- and down-vote items
(York & Zuckerman, 2019).
The challenge here is in ﬁguring out who the stakeholders
are, how to involve them, and how to resolve their com-
peting demands (Baum, 2020). One participatory design
effort created an online matching system for a volunteer
food delivery service, attempting ”to enable stakeholders
to construct a computational model that represents their
views and to have those models vote on their behalf to cre-
ate algorithmic policy” (Lee et al., 2019). V olunteer drivers,
donors, recipients, and non-proﬁt employees collaboratively
designed the trade-offs between themselves and between
efﬁciency and equity. This was done by interactive elicita-
tion and modeling of individual preferences, followed by a
Borda voting rule to decide the choice for the overall system.
The authors report that this new algorithm both improved
distributive outcomes and was perceived as fairer.3.3. Interactive value learning
The recommenders with the most troubling side effects of-
ten have passive, low-agency, low-bandwidth interfaces
like inﬁnitely scrolling content feeds or autoplay. In
these systems, user traces are evaluated by a combina-
tion of hand-coded and learned metrics. It is often un-
clear, even to system designers, what ’values’ these metrics
represent, as optimization-driven behavior can be counter-
intuitive (Krakovna, 2018).
Direct user control of how values are enacted would provide
both agency and transparency. This implies that recom-
menders must learn values interactively. There is a general
sentiment that ofﬂine recommender benchmarks are decreas-
ingly meaningful (Rohde et al., 2018), which further sug-
gests that static values training data sets are inadequate.
Rather than depending on surveys for user feedback, we
could be thinking about new kinds of ongoing interaction
protocols. For example, probabilistic models like Inverse
Reward Design (Hadﬁeld-Menell et al., 2017) can be used to
regularize recommender metrics and infer consistent com-
binations of metrics that generalize intent (Ratner et al.,
2018). Interfaces designed around efﬁcient imitation learn-
ing (Brown et al., 2020) and active reward learning (Bıyık
et al., 2020) might be a path towards more accessible, nu-
anced and predictable control of recommendation behavior.
3.4. Design around informed, deliberative judgment
Converging arguments in AI alignment (Tarleton, 2010), po-
litical philosophy (Daniels, 2020), and bioethics (Molewijk
et al., 2008) suggest that informed, deliberative, and perhaps
retrospective evaluations are of a higher quality than imme-
diate judgements. Many people report that they watch more
TV than they retrospectively endorse (Frey et al., 2007) and
a similar type of behavior happens with some social media
users (Andreassen, 2015). The notion of ”time well spent”
gets at similar ideas.
This is closely related to the common problem of focusing
on short vs. long term outcomes, which is an issue with the
use of metrics in AI generally (Thomas & Uminsky, 2020).
One solution is to solicit informed, retrospective feedback.
For example, one could show the user a summary of their
recommendations and usage patterns in the past month, ask
them to spend some time reﬂecting on it, and then rate how
happy they are with this or how well it matches their goals.
Versions of this idea have seen some use in the tech industry,
sometimes known as ”the regret test” (Eyal, 2018).
4. Conclusion
Recommendations systems are a cornerstone of the internet
economy. They allow people to effectively interact with the
Aligning Recommender Systems
gigantic amount of information found in modern internet
platforms. As recommendation systems proliferate, it is
important to consider the impact these systems have and
the values that drive ranking decisions. A crucial step in
this process is to understand how these values are currently
designed and updated. As our case studies illustrate, this
can be largely described with a three step process where
1) outcomes are identiﬁed and reiﬁed as concepts; 2) the
concepts are operationalized with a hand-coded or learned
metric; and 3) this metric is used to adjust recommenda-
tion behavior. Looking to the future, we are excited about
the potential for recommendation systems to better align
with human values through the incorporation of well-being
metrics, participatory approaches to objective design, in-
teractive value learning, and optimizing for informed and
deliberative preferences.
References
Abdollahpouri, H., Adomavicius, G., Burke, R., Guy, I.,
Jannach, D., Kamishima, T., Krasnodebski, J., and Piz-
zato, L. Beyond Personalization: Research Directions
in Multistakeholder Recommendation. arXiv preprint
arXiv:1905.01986 , 2019.
Alkire, S. Concepts and Measures of Agency. Ox-
ford Poverty & Human Development Initiative , 2008.
URL https://ora.ox.ac.uk/objects/uuid:
cdecbaca-447c-43b7-8e3f-851517b5ff97 .
Andreassen, C. S. Online Social Network Site Addiction:
A Comprehensive Review. Current Addiction Reports , 2
(2):175–184, 2015.
Barocas, S., Hardt, M., and Narayanan, A. Fairness and ma-
chine learning, 2019. URL https://fairmlbook.
org/ .
Baum, S. D. Social choice ethics in artiﬁcial intelligence.
AI and Society , 35(1):165–176, 2020.
Benkler, Y ., Faris, R., and Roberts, H. Network propa-
ganda: Manipulation, disinformation, and radicalization
in American politics . Oxford University Press, 2018.
Bıyık, E., Huynh, N., Kochenderfer, M. J., and Sadigh, D.
Active preference-based gaussian process regression for
reward learning. arXiv preprint arXiv:2005.02575 , 2020.
Brown, D. S., Coleman, R., Srinivasan, R., and Niekum,
S. Safe imitation learning via fast bayesian reward infer-
ence from preferences. arXiv preprint arXiv:2002.09089 ,
2020.
Bruns, A. Are Filter Bubbles Real? Polity, 2019. ISBN
978-1509536443.Castells, P., Hurley, N. J., and Vargas, S. Novelty and
diversity in recommender systems. In Recommender
systems handbook , pp. 881–918. Springer, 2015.
Cora, M. V . Detecting Trustworthy Domains. Flipboard ,
2017. URL https://engineering.flipboard.
com/2017/04/domainranking .
Daniels, N. Reﬂective equilibrium. In Zalta, E. N. (ed.),
The Stanford Encyclopedia of Philosophy . Metaphysics
Research Lab, Stanford University, summer 2020 edition,
2020.
Etim, B. The Times Sharply Increases Articles Open
for Comments, Using Google’s Technology, June
2017. URL https://www.nytimes.com/2017/
06/13/insider/have-a-comment-leave-a-
comment.html .
Eyal, N. Want to Design User Behavior? Pass
the ‘Regret Test’ First, 2018. URL https:
//www.psychologytoday.com/us/blog/
automatic-you/201803/want-design-
user-behavior-pass-the-regret-test-
first .
Facebook. Using AI to detect COVID-19 misin-
formation and exploitative content, 2020. URL
https://ai.facebook.com/blog/using-
ai-to-detect-covid-19-misinformation-
and-exploitative-content .
Frey, B. S., Benesch, C., and Stutzer, A. Does Watching TV
Make Us Happy? Journal of Economic Psychology , 28
(3):283–313, 2007.
Gabriel, I. Artiﬁcial intelligence, values and alignment.
arXiv preprint arXiv:2001.09768 , 2020.
Gadde, V . and Gasca, D. Measuring healthy conversation.
Twitter, 2018. URL https://blog.twitter.
com/en{_}us/topics/company/2018/
measuring{_}healthy{_}conversation.
html .
Gao, R. and Shah, C. Toward creating a fairer ranking
in search engine results. Information Processing and
Management , 57(1), 2020.
Hadﬁeld-Menell, D. and Hadﬁeld, G. K. Incomplete con-
tracting and AI alignment. AIES 2019 - Proceedings
of the 2019 AAAI/ACM Conference on AI, Ethics, and
Society , pp. 417–422, 2019.
Hadﬁeld-Menell, D., Milli, S., Abbeel, P., Russell, S. J.,
and Dragan, A. Inverse reward design. In Advances
in Neural Information Processing Systems 30 , pp. 6765–
6774, 2017.
Aligning Recommender Systems
Hasan, M. R., Jha, A. K., and Liu, Y . Excessive use of online
video streaming services: Impact of recommender system
use, psychological factors, and motives. Computers in
Human Behavior , 80:220–228, 2018.
Helberger, N. On the Democratic Role of News Recom-
menders. Digital Journalism , 7(8):993–1012, 2019.
Helberger, N., Karppinen, K., and D’Acunto, L. Exposure
diversity as a design principle for recommender systems.
Information Communication and Society , 21(2):191–207,
2018.
Khalid El-Arini and Joyce Tang. Click-baiting, 2014.
URLhttps://about.fb.com/news/2014/08/
news-feed-fyi-click-baiting/ .
Khwaja, M., Ferrer, M., Iglesias, J. O., Faisal, A. A., and
Matic, A. Aligning daily activities with personality: to-
wards a recommender system for improving wellbeing.
InProceedings of the 13th ACM Conference on Recom-
mender Systems , pp. 368–372, 2019.
Knijnenburg, B. P., Sivakumar, S., and Wilkinson, D. Rec-
ommender systems for self-actualization. Proceedings
of the 10th ACM Conference on Recommender Systems
(RecSys ’16) , pp. 11–14, 2016.
Krakovna, V . Speciﬁcation gaming examples in ai,
2018. URL https://vkrakovna.wordpress.
com/2018/04/02/specification-gaming-
examples-in-ai/ .
Kulynych, B., Overdorf, R., Troncoso, C., and G ¨urses, S.
Pots: Protective optimization technologies. arXiv preprint
arXiv:1806.02711v6 , 2020.
Lalmas-Roelleke, M. Recommending and searching @
Spotify, 2019. URL https://www.slideshare.
net/mounialalmas/recommending-and-
searching-spotify/21 .
Lee, M. K., Kusbit, D., Kahng, A., Kim, J. T., Yuan, X.,
Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A.,
and Procaccia, A. D. Webuildai: Participatory framework
for algorithmic governance. Proceedings of the ACM on
Human-Computer Interaction (CSCW) , 3, 2019.
Lu, F., Dumitrache, A., and Graus, D. Beyond Optimizing
for Clicks: Incorporating Editorial Values in News Rec-
ommendation. arXiv preprint arXiv:2004.09980 , 2020.
Mehrotra, R., McInerney, J., Bouchard, H., Lalmas, M.,
and Diaz, F. Towards a fair marketplace: Counterfactual
evaluation of the trade-off between relevance, fairness &
satisfaction in recommendation systems. International
Conference on Information and Knowledge Management,
Proceedings , pp. 2243–2252, 2018.Milano, S., Taddeo, M., and Floridi, L. Recommender
Systems and their Ethical Challenges. 2019a. URL
https://ssrn.com/abstract=3378581 .
Milano, S., Taddeo, M., and Floridi, L. Ethical aspects of
multi-stakeholder recommendation systems. 2019b. URL
https://ssrn.com/abstract=3378581 .
Molewijk, A. C., Abma, T., Stolper, M., and Widdershoven,
G. Teaching ethics in the clinic. The theory and practice
of moral case deliberation. Journal of Medical Ethics , 34
(2):120–124, 2008.
O’Donnell, G., Deaton, A., Durand, M., Halpern, D., and
Layard, R. Wellbeing and Policy. Legatum Institute .
Peysakhovich, A. and Hendrix, K. Further Reducing Click-
bait in Feed. Facebook, 2016. URL https://about.
fb.com/news/2016/08/news-feed-fyi-
further-reducing-clickbait-in-feed/ .
Ratner, E., Hadﬁeld-Mennell, D., and Dragan, A. D. Sim-
plifying reward design through divide-and-conquer. In
Robotics: Science and Systems , 2018.
Rodriguez, M. A. and Watkins, J. H. Faith in the algorithm,
part 2: Computational eudaemonics. In International
Conference on Knowledge-Based and Intelligent Infor-
mation and Engineering Systems (KES) , pp. 813–820,
Berlin, 2009. Springer.
Rohde, D., Bonner, S., Dunlop, T., Vasile, F., and Karat-
zoglou, A. RecoGym: A Reinforcement Learning Envi-
ronment for the problem of Product Recommendation in
Online Advertising. arXiv preprint arXiv:1808.00720 ,
2018.
Rolnick, D., Donti, P. L., Kaack, L. H., Kochanski, K.,
Lacoste, A., Sankaran, K., Ross, A. S., Milojevic-Dupont,
N., Jaques, N., Waldman-Brown, A., et al. Tackling
climate change with machine learning. arXiv preprint
arXiv:1906.05433 , 2019.
Salganik, M. J. and Lee, R. C. To Apply Ma-
chine Learning Responsibly, We Use It In Mod-
eration. New York Times Open , 2020. URL
https://open.nytimes.com/to-apply-
machine-learning-responsibly-we-use-
it-in-moderation-d001f49e0644 .
Schiff, D., Ayesh, A., Musikanski, L., and Havens, J. C.
Ieee 7010: A new standard for assessing the well-being
implications of artiﬁcial intelligence. arXiv preprint
arXiv:2005.06620 , 2020.
Silverman, C. (ed.). The Veriﬁcation Handbook for Disinfor-
mation And Media Manipulation . European Journalism
Center, 2020. URL https://datajournalism.
com/read/handbook/verification-3 .
Aligning Recommender Systems
Stiglitz, J. E., Fitoussi, J.-P., and Durand, M. Beyond
GDP . OECD, 2018. URL https://www.oecd-
ilibrary.org/content/publication/
9789264307292-en .
St¨ocker, C. How Facebook and Google Accidentally Cre-
ated a Perfect Ecosystem for Targeted Disinformation. In
Grimme, C., Preuss, M., Takes, F. W., and Waldherr, A.
(eds.), Disinformation in Open Online Media , pp. 129–
149. Springer, 2019.
Stray, J. Aligning AI to Human Values means Picking the
Right Metrics. Partnership on AI, 2020. URL https:
//medium.com/@PartnershipAI/aligning-
ai-to-human-values-means-picking-the-
right-metrics-855859e6f047 .
Tarleton, N. Coherent Extrapolated V olition: A Meta-Level
Approach to Machine Ethics. Machine Intelligence Re-
search Institute , 2010.
Tessa Lyons. Increasing Our Efforts to Fight False News.
Facebook , 2018. URL https://about.fb.com/
news/2018/06/increasing-our-efforts-
to-fight-false-news/ .
Thomas, R. L. and Uminsky, D. Reliance on Metrics is a
Fundamental Challenge for AI. In Ethics of Data Science
Conference , 2020.
Thompson, N. Facebook Opens Up About
False News. Wired , 2018. URL https:
//www.wired.com/story/exclusive-
facebook-opens-up-about-false-news/ .
Wagner, K. Inside Twitter’s ambitious plan to change
the way we tweet. Vox, 2019. URL https://www.
vox.com/2019/3/8/18245536/exclusive-
twitter-healthy-conversations-
dunking-research-product-incentives .
Yi, X., Hong, L., Zhong, E., Liu, N. N., and Rajan, S. Be-
yond clicks: Dwell time for personalization. Proceedings
of the 8th ACM Conference on Recommender Systems
(RecSys ’14) , pp. 113–120, 2014.
York, J. and Zuckerman, E. Moderating the Public Sphere.
In Jørgensen, R. F. (ed.), Human Rights in the Age of
Platforms , chapter 6. MIT Press, 2019.
Zhao, Z., Chi, E., Hong, L., Wei, L., Chen, J., Nath, A.,
Andrews, S., Kumthekar, A., Sathiamoorthy, M., and
Yi, X. Recommending what video to watch next: A
Multitask Ranking System. In Proceedings of the 13th
ACM Conference on Recommender Systems (RecSys ’19) ,
pp. 43–51, 2019.

Jonathan Stray
Steven Adler
Dylan Hadfield-Menell

While the previous blog post focused on societal-level approaches to recommender systems, this paper looks at what can be done at a technical level. By analyzing existing case studies of improvements to recommender systems (some of which we’ve <@seen before@>(@Aligning AI to Human Values means Picking the Right Metrics@)), the authors identify a typical approach taken in industry today.

First, engineers identify a problem with an already-deployed recommendation engine, perhaps from user feedback, or through monitoring. Second, they develop a concrete procedure to identify instances of this problem in the recommendations -- a typical approach is to curate a dataset and train an ML classifier to identify these instances, though it is also possible to use manual review. Finally, the recommender system is adjusted to avoid the problem, for example by adding a term to the objective when training the recommender system, or by filtering its outputs based on the classifier’s output.

The authors then propose four high-level technical approaches to recommender alignment:
1. Develop better measures of what we want out of a recommendation engine, for example, an operationalization of “well-being” rather than “engagement”.
2. Allow users to collaboratively design the recommendation engine (called _participatory design_). Rather than have a company decide on how to trade off between different objectives, allow the users to settle upon the appropriate tradeoffs themselves.
3. Interactively learn about the user’s values. While this could look like building better controls as suggested in the previous post, it could also involve e.g. using <@Inverse Reward Design@> to maintain appropriate uncertainty over what the user cares about.
4. Design around “time well spent”, as evaluated by users on reflection or after consideration, rather than revealed preferences or immediate judgments. For example, we could show users a summary of their activity over the past month and ask how happy they are about it.

Both this paper and the previous post seem like meaningful progress in ideas for making recommendation engines better (though as a caveat, I don’t follow this space and so don’t know to what extent this has been said before). I’m glad that we’re getting to the point of actually proposing technical solutions; I hope to see more papers implementing such solutions (we’ve seen one from Twitter <@recently@>(@From Optimizing Engagement to Measuring Value@)).

[Aligning AI Optimization to Community Well-Being](https://link.springer.com/article/10.1007/s42413-020-00086-3) (an expanded version of <@Aligning AI to Human Values means Picking the Right Metrics@>)


Active reward learning

Active Reward Learning
Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, Jan Petersy
Institut f ¨ur Intelligente Autonome Systeme, Technische Universit ¨at Darmstadt, 64289, Germany
yMax-Planck-Institut f ¨ur Intelligente Systeme, T ¨ubingen, 72076, Germany
Email:fdaniel, kroemer, peters g@ias.tu-darmstadt.de
Abstract —While reward functions are an essential component
of many robot learning methods, deﬁning such functions remains
a hard problem in many practical applications. For tasks such
as grasping, there are no reliable success measures available.
Deﬁning reward functions by hand requires extensive task knowl-
edge and often leads to undesired emergent behavior. Instead,
we propose to learn the reward function through active learning,
querying human expert knowledge for a subset of the agent’s
rollouts. We introduce a framework, wherein a traditional learn-
ing algorithm interplays with the reward learning component,
such that the evolution of the action learner guides the queries
of the reward learner. We demonstrate results of our method on
a robot grasping task and show that the learned reward function
generalizes to a similar task.
I. I NTRODUCTION
An important goal of Reinforcement Learning (RL) is to
yield more autonomous robots. However, RL methods require
reward functions to guide the agent towards a desired be-
havior. Such reward functions are usually hand coded and,
unfortunately, deﬁning rewards for real robot tasks manually is
challenging even for relatively well-understood problems such
as grasping. Thus, hand coding reward functions is shifting
the problem of requiring an expert to design a hard coded
controller to requiring a hard coded reward function.
Despite the variety of grasp stability measures that has been
developed [34], it has been shown that the resulting grasps
are outperformed by grasps learned from kinesthetic teach-
in [3, 23]. This example shows that even experts will often
design reward functions that are not effective in practice, or
lead to undesired emergent behavior [27].
To avoid specifying reward functions, Inverse RL (IRL)
extracts a reward function from demonstrations [30, 31, 36].
For many tasks, such demonstrations can be obtained through
methods such as kinesthetic teach-in or tele-operation. Unfor-
tunately some skills, such as dynamic skills, are often hard to
demonstrate (for example, teaching a robot to throw a basket
ball). Moreover, both methods require sufﬁcient proﬁciency of
the demonstrator which may lead to the demonstrator having
to truly become an expert in performing the task ﬁrst.
While it may be difﬁcult to analytically design a reward
function or to give demonstrations, it is often easy for an expert
to rate an agent’s executions of a task. Thus, a promising
alternative is to use the human not as an expert in performing
the task, but as an expert in evaluating task executions.
Based on this insight, preference based algorithms allow the
expert to rank executions and learn a controller based on these
rankings [2, 6]. The generally used approach in ranking is to let
Fig. 1: The Robot-Grasping Task. While grasping is one of the most
researched robotic tasks, ﬁnding a good reward function still proves
difﬁcult.
the expert rank the previously best sample against the current
sample, which is an intriguing idea, as humans are better at
giving relative judgments than absolute judgments. However,
this approach only provides a single bit of information. The
technical term describing how much information human sub-
jects can transmit for a given input stimuli is called channel
capacity [25]. The channel capacity is a measure for how many
input stimuli subjects can distinguish between. In a review of
several experiments, Miller [25] concludes that humans have
a general channel capacity somewhere between 1:6and3:9
bits for unidimensional stimuli. Adding more dimensions to
the stimuli further increases this value.
Experiments have also been performed to ﬁnd out whether
humans can transmit more information when labelling stimuli
according to predescribed categories or when being allowed to
rate on a scale [15]. While there was no signiﬁcant difference,
subjects performed better when rating on a scale.
Based on these insights, we propose an alternative frame-
work in which the human expert can assign numerical values
to observed demonstrations. Furthermore, numerical rewards
allow to indicate strong preferences over demonstrations.
Unfortunately, as already shown by Thomaz and Breazeal [35]
and by Cakmak and Thomaz [5], humans have considerable
noise in their ratings of actions. To deal with this mismatch
between good reward functions for policy search (PS) methods
and noisy human rating systems, we propose to learn a prob-
abilistic model of the reward function. In this paper, we take
advantage of the Gaussian Process (GP) regression framework
as well as the Bayesian Optimization (BO) literature. Having
access to BO methods, allows us to efﬁciently minimize
Rewards  R = f(o)Outcomes o = 𝜙(𝞃)Rewards R ∼ p(R|o,t)Trajectories 𝞃 ∼ p(𝞃 |s, 𝞈)Context sControl Parameters 𝞈 ∼ 𝜋(𝞈|s)Query occasionallyExpert Ratings t ∼ p(t|𝞃)Vanilla RLActive Reward LearningBayesian OptmiziationControl Parameters 𝞈=argmax u(R|s,𝞈,t)Query alwaysNo Reward LearningSemi Supervised Reward LearningSupervised Reward LearningFig. 2: Sketch of the elements of different possible approaches. The left column shows the ‘vanilla’ RL where a reward function is assumed.
The middle column shows the proposed active reward learning approach, which shares the policy learning component with vanilla RL but
models a probabilistic reward model that gets updated by asking for expert ratings sometimes. The right column shows the BO approach,
where actions are chosen that maximize the utility function (for example one of the acquisition functions presented in II-A1 and requires an
expert rating for each of the chosen actions.
the number of expert interactions, which is essential when
designing methods for ‘autonomous’ agents. We evaluate the
proposed method on a series of simulated and one real robot
task and we also show that a reward function which is learned
for one task (grasping a box) can be used to learn a new task
(grasping a pestle).
II. M ETHOD
While there are many approaches to RL, Policy Search (PS)
is especially well suited for real robot tasks. It does not rely
on exhaustive sampling of the state-action space and, as a
result, many recent advances in robot learning rely on policy
search [22, 20, 26]. We consider the contextual episodic PS
case with continuous contexts s2S and continuous control
parameters!2
. We use the term ‘context’ instead of the
more general term ‘state’ to indicate the initial state of the
robot and the environment. The distribution over contexts
is denoted by ()Whereas the traditional RL notation
uses actionsa, we instead write control parameters !to
clarify that we directly optimize for the parameters of a
controller which can, for example, be a Movement Primitive
as discussed in Section III-B. Executing a rollout with control
parameters!results in a trajectory p(js;!), where the
trajectory encodes both, the robot’s state transitions as well
as relevant environment state transitions. The agent starts with
an initial control policy 0(!js)in iteration 0and performs
a predetermined number of rollouts. After each iteration, the
agent updates its policy to maximize the expected reward
Es;!;[R()] =ZZZ
R()p(s;!;) ddsd!;(1)
withp(s;!;) =p(js;!)(!js)(s):Parameter-based
PS has been shown to be well suited for complex robot
tasks [9], as it abstracts complexity while retaining sufﬁcient
ﬂexibility in combination with suitable movement primitive
representations.
In the original RL setting a reward function is assumed to be
known that evaluates the agents behavior and assigns rewards
to the agent’s actions. While not always explicitly stated, the
reward function for real robot tasks often depends not on the
whole trajectory of the robot, but on features ()of the
trajectory. We refer to the result of the feature extraction fromtrajectories as the outcomeso=(). We assume that the
reward depends only on the features of the trajectories. Such
outcomes could, for example, describe the minimal distance
to goal positions or the accumulative energy consumption.
As the problem of specifying good features ()is beyond
the scope of this paper, we assume the features to be known,
as is usually assumed when designing reward functions.
As outcomes are of much lower dimensionality than trajec-
tories, we can use outcomes to efﬁciently model the reward
function ^R(o)from a training set D=fo1:n;R1:ngusing
regression. Using the feature representation, the term R()
in Eq. 1 is replaced by R(o=())to become
Es;!;[R(o)]=ZZZ
R(o=())p(s;!;) ddsd!:(2)
Obviously, it is not sufﬁcient to build the reward function
model on samples from the initial policy 0(!js), as the
agent is likely to exhibit poor performance in the early stages
of learning and the reward learner would not observe good
samples. Therefore, we need to couple the process of learning
a good policy (!js)and a good reward function ^R(o), such
that they are developed interdependently. However, in such
a coupled active learning process, we need to know which
samples to query from our expert, as we want to minimize
expert interaction.
Modelling a probabilistic reward model p(Rjo;D), instead
of a deterministic reward function R(o), allows us to leverage
the information about the certainty of our estimate to control
the amount of human interaction required, as we only need to
interact with the human expert if our model of the reward is
not certain enough. We describe the details of this process in
Section II-A.
When using a probabilistic model of the reward, we have
to replace the reward term in Eq. 1 with
R(o) =Ep(Rjo)[R] =Z
p(Rjo)RdR;
as most PS methods work on the expectation of the reward.
Using the probabilistic model of the reward function the
PS method continuously learns how to achieve high reward
outcomes, while the reward model learner adapts the accuracy
of its model.
Input: Information loss tolerance , improvement threshold ,
acquisition function u
Initializeusing a single Gaussian with random mean. GP with
zero mean prior.
while not converged
Set sample policy:
q(!js) =old(!js)
Sample: collect samples from the sample policy
fsip(s);!iq(!jsi);oig;i2f1;:::;Ng
Deﬁne Bellman Error Function
(s;!) =R(o)V(s)
Minimize the dual function
[;] = arg min[;]g(;)
Determine base line
V(s) =T(s)
Policy update:
Calculate weighting
p(si;!i)_q(si;!i) exp
1
(si;!i)
Estimate distribution (!js)by
weighted maximum likelihood estimates
Reward model update:
FindNominee = true
while FindNominee
Nominate outcome:
o+= arg maxu(o)
if
o+=2D
^
(o+)=>
Demonstrate Corresponding Trajectory +
Query Expert Reward R+
D=D[fo+;R+g
else
FindNominee = false
Update reward model p(Rjo;D)
Optimize GP-hyper parameters
Output: Policy(!js), reward model p(Rjo;D)
TABLE I: We show the algorithmic form of active reward learning
with REPS. We specify the information loss tolerance as well
as an initial sampling policy and an improvement threshold . In
each iteration, the algorithm ﬁrst samples from the sampling policy,
minimizes the dual function to ﬁnd values for Tandand
then computes the next policy. After each policy search iteration,
the reward function learner chooses whether to demonstrate samples
to the expert according to the acquisition function. The parameters
andare parameters of the dual function problem of REPS and
can be optimized through standard optimization algorithms [9].
A. Active Reward Learning
Our goal is to ﬁnd a model p(Rjo;D)that predicts the
reward given an observed outcome and training data D, which
is obtained from an expert. When modelling the reward, we
have to take into account that the expert can only give noisy
samples of his implicit reward function and we also have
to model this observation noise. Thus, we need to solve the
regression problem
R(o) =f(o) +; N (0;);
where we assume zero mean Gaussian noise. Such a regression
problem can, for example, be solved with Gaussian Process
(GP) regression
f(o)GP (m(o);k(o;o0));
wherem(o)is the mean function and k(o;o0)is the covari-
ance function of the GP. For the remainder of this paper weuse the standard squared exponential covariance function
k(o;o0) =2
0exp
jjoo0jj2
22
1
:
The set of hyper parameters =f0;1;gis found through
optimization [29]. Given a training set D=fo1:n;R1:ng,
we can write down the covariance matrix between previously
observed rewards and outcomes
K=2
64k(o1;o1)::: k (o1;on)
.........
k(on;o1)::: k (on;on)3
75+I:
Assuming a zero mean prior, the joint Gaussian probability of
the training samples in Dand the reward prediction R+of a
new unrated observation is given by
R1:n
R+
N
0;
K ^k
kk(o+;o+)
;
with ^k = [k(o1;o+):::k(on;o+)]Tand k =
[k(o+;o1):::k(o+;on)]. The predictive posterior reward
p(R+jo;D)of a new outcome o+is then given by a Gaussian
p(R+jo;D)N
(o+);2(o+)
;
with mean and variance
(o+) =kTK1R1:n;
2(o+) =k(o+;o+)kTK1k;
by conditioning the GP on the observed outcome o+. Using
the GP, we can represent both our expected reward (o+),
which is provided to the policy learner, and the variance of
the reward2(o+)which is essential to the active learning
component. The reward variance 2(o+)depends on the
distance of the outcome o+to all outcomes in the training set
Dand the observation noise , which is a hyper parameter
that we optimize for. Using the predictive variance, we can
employ one of many readily available BO methods to ﬁnd the
maximum of the reward function.
1) Optimizing the Reward Model: The goal of BO is to
optimize a function under uncertainty. Acquisition functions
(AFs), are utility functions which maximize their function
value at locations of the input space which are likely to maxi-
mize the original problem. AFs usually encode an exploration-
exploitation trade-off, such that they do not only query samples
in known high value regions but also in regions that have not
been sufﬁciently explored before. While using GPs to model
the reward function allows us access to the BO toolbox in
general, we deviate from the standard BO approach in two
points. First, as our GP models a relationship of outcomes
to rewards instead of context-actions to rewards, we cannot
sample arbitrary outcomes ^oto improve our estimate. To do
so, we would require access to p(js;!)such that we can
request the agent to perform actions that result in trajectories
^which yield the outcome ^o=(^). Second, we would
need to guarantee that the outcomes requested by the AF to
improve the reward model are physically possible. However,
this transition model is unknown and, thus, we have to revert
to the previously observed outcomes that have been generated
during the agent’s learning process so far.
Furthermore, we need to balance the improvement of our
current estimate of the reward function and the number of
queries that we request from the expert, i.e., we want to ﬁnd a
trade-off between ﬁnding a good reward function and learning
the task with minimal human input.
2) Sample Efﬁciency: In the traditional BO framework, the
goal is to ﬁnd a global maximum. Sampling of the function
can be stopped when the improvement of the function is
marginal, for example when the predicted variance around
the optimum is very low. However, in the proposed scenario
where a policy (!js)and a reward model p(Rjo)are learned
simultaneously and obtaining training samples of the reward
model is very expensive, the problem of deciding when to
improve the reward model becomes crucial.
The reward model relies on the policy (!js)to provide
outcomes in interesting, i.e., high reward regions, and the pol-
icy relies on the reward model p(Rjo)to guide its exploration
towards such regions of interest. Thus, the reward model needs
sufﬁcient training data to approximately predict the reward
function in early stages and a higher density of training points
once the agents policy starts to converge to a solution. At the
same time, we want to minimize the number of queries to the
expert over the learning process.
In order to balance this trade-off, we propose an acquisition
algorithm which, according to a selected acquisition function
u(o), ﬁrst samples the best observed sample outcome from the
history of the agent’s outcomes on+1= arg maxou(ojD).
In this sample based search, we also include all outcomes
that have been demonstrated to the expert. If the acquisition
function is maximized by an already demonstrated outcome,
we stop and do not query any samples in this iteration. Maxi-
mizing the AF by an already observed outcome is unique to the
sample based case, as in the continuous case, a point around
an observed outcome would usually have more variance and,
thus, maximize the AF. With our approach we achieve a sparse
sampling behavior that requires less expert interactions. If,
however, the outcome that maximizes uhas not yet been rated
by the expert, we need to decide whether querying the outcome
is beneﬁcial. For example, we may have already converged to
a good estimate of the reward function, and new outcomes
improve on the mean reward solely due to the observation
noise. In this case we do not want to query the expert. Thus,
we decide whether to query an outcome by thresholding the
ratio of predictive variance and estimated observation noise
(o)=p > ; whereis a tuning parameter which allows
us to trade off the accuracy of the ﬁnal solution with the
query frequency by adapting the available AFs to explicitly
take the estimated observation noise into account. While this
adaptation of the AFs introduces a new parameter to be tuned,
our experiments show that the use of this technique results in
less human interactions while maintaining high performance
and tuning of the parameter is straightforward.
If we decide to query the sample, we update the GP andsearch for the new maximum of the acquisition function,
otherwise we stop and do not update the GP further in this
iteration.
The general information ﬂow of our method is as follows.
We start with an uninformed, i.e., zero mean GP for p(Rjo)
and we initialize the PS method with an initial policy 0(!js).
The PS learner then starts performing one iteration of rollouts,
which we also call episodes. After each iteration of rollouts,
rewards for the outcomes of the resulting trajectories o=()
are requested from the reward learner Rp(Rjo). The
reward learner then decides whether to ask for expert ratings
for any of the outcomes to update its model. In that case,
the agent repeats the corresponding episode to present the
outcome to the expert. Finally, the reward learner returns the
mean estimate of the reward to the PS learner, which uses the
rewards to update its policy and start the next iteration.
III. B ACKGROUND
In this section we provide compact background information
on components of the proposed algorithms that are necessary
to give a complete picture of the proposed approach.
A. Relative Entropy Policy Search
We pair our reward learning algorithm with the recently
proposed relative entropy policy search (REPS) [28]. REPS
is a natural choice for a PS method to be combined with an
active learning component as it has been shown to work well
on real robot problems [9] and is designed to ‘stay close to the
data’. Thus, previous expert queries will remain informative. A
distinctive feature of Relative Entropy Policy Search (REPS),
is that its successive policies vary smoothly and do not jump
in the parameter space or the context space. This behavior
is a beneﬁcial characteristic to increase compliance with our
proposed active learning approach, as we are only able to
predict correct rewards within observed regions of the param-
eter space. To constrain the change in the policy, REPS limits
the Kullback-Leibler divergence between a sample distribution
q(s;!)and the next distribution (!js)(s)
X
s;!(s)(!js) log(s)(!js)
q(s;!): (3)
For the complete optimization problem and its solution we
refer to the original work [28].
B. Dynamic Movement Primitives
A popular use case for PS methods is to ﬁnd good pa-
rameters of trajectory generators such as Dynamic Movement
Primitives (DMPs) [17]. The resulting desired trajectories can
then be tracked by a linear feedback controller. DMPs model
trajectories using an exponentially decreasing phase function
and a nonlinear forcing function. The forcing function excites
a spring damper system that depends on the phase and is
guaranteed to reach a desired goal position, which is one of
the parameters of a DMP. The forcing function is modelled
through a set of weighted basis functions !	. Using the
weights!of the basis functions 	as parameters, we can
learn parametrized joint trajectories, and an increasing number
of basis functions results in an increased ﬂexibility of the
trajectory. We use DMPs for our simulated robot experiments.
C. Bayesian Optimization for RL
An alternative approach to learning control parameters !
using PS methods is to model p(Rjs;!)directly and to
use Bayesian Optimization (BO) instead of the PS learner.
However, the approach proposed in this paper introduces a
layer of abstraction which allows us to learn a mapping of
only a low-dimensional input space to the reward, as we only
have to map from outcomes to reward as opposed to map from
state-action to reward. As BO methods are global methods,
they are more susceptible to the curse of dimensionality than
PS methods which are local methods. As a result, the mapping
p(Rjs;!)which the standard BO solution uses is considerably
more difﬁcult to learn than the mapping of the modular
approach that we are proposing.
The BO approach would also require expert ratings for every
sample, while the proposed approach requires only occasional
human feedback.
D. Acquisition Functions
In the following we present four Acquisition Function (AF)
schemes taken from Hoffman et al. [16] that we used to
optimize the model of the reward function.
a) Probability of Improvement: The Probability of Im-
provement (PI) [16] in its original formulation greedily
searches for the optimal value of the input parameter that
maximizes the function. An adapted version of PI balances the
greedy optimization with an exploration-exploitation trade-off
parameter. The adapted version is given by
PI(o) = (o)f(o)
(o)
;
whereois the best sample in the training set Dand()
is the normal cumulative density function. The exploration-
exploitation trade-off parameter has to be chosen manually.
b) Expected Improvement: Instead of ﬁnding a point that
maximizes the PI, the Expected Improvement (EI) [16], tries
to ﬁnd a point that maximizes the magnitude of improvement.
Thus, it does not only try to improve local maxima but also
considers maxima in different regions and is less greedy in
the search of an optimal reward R.
EI(o) = ((o)f(o)))  (M) +(o)(M);
if(o)>0and zero otherwise, where ()is the normal
probability density function. Mis given by
M=(o)f(o)
(o):
The EI acquisition function shares the tuning factor for
the exploitation-exploration trade-off with the PI, where a
suggested value is = 0:01[16].c) Upper Conﬁdence Bound: The Upper Conﬁdence
Bound directly uses the mean and standard deviation of the
reward function at the sample location to deﬁne the acquisition
function. An adapted version of the UCB function [33] is given
by
GP-UCB (o) =(o) +pv
n(o);
where recommended values v= 1 and
n=
2 log(nd=2+22=3)withd=dim(o)and2(0;1)
are given by Srinivas et al. [33].
d) GP Hedge: As each of the above acquisition functions
lead to a characteristic and distinct sampling behavior, it is
often not clear which acquisition function should be used.
Portfolio methods, such as the GP-Hedge [16], evaluate several
acquisition functions before deciding for a sample location.
Given a portfolio with Jdifferent acquisition functions, the
probability of selecting acquisition function jfor the sample
n+ 1is given by the softmax
p(j) =exp(gj
n)PJ
i=1exp(gj
n);
where >0is the temperature of the soft-max distribution.
The gains vector gis initialized to zero before taking the
ﬁrst sample g1:J
0= 0 , and is then updated with the cu-
mulative reward gained by the selected acquisition function,
i.e.,gj
n+1=gj
n+(oj
n+1), whereoj
n+1is the sample point
nominated by acquisition function j. For all other gains the
value does not change, i.e., gi6=j
n+1=gi
n.
IV. E VALUATIONS
In this section we show evaluations of the proposed active
reward learning approach. For all simulation experiments,
unless stated otherwise, we tested each setting ten times.
A. Five Link Reaching Task
A simulated planar robot consisting of ﬁve links connected
by rotary joints was controlled in joint space using Dynamic
Movement Primitives (DMP) [17], as described in Section
III-B. If not stated otherwise, we used 20basis functions
per joint, resulting in a total of 100 parameters that had to
be learned. We evaluated our approach on a reaching task,
where an analytical reward function was readily available.
The hand coded reward function was given by R(pr) =
1000100jjprpgjj;whereprwas the position of the robot’s
end effector and pgwas the desired target position. However,
we increased the difﬁculty of the task by not supplying the
outcome features in task space, but rather in joint space,
i.e., the GP had to model the forward kinematics to predict
the reward, making the problem both non-convex and high-
dimensional (ﬁve dimensional mapping).
To allow extensive and consistent evaluation of all parame-
ters of the presented approach, we programmed a noisy expert,
which returned the reward with additive white noise (standard
deviation was 20). In Fig. 7 we present results that compare
the coded noisy expert approach to actually querying a human
expert and show that the behavior is comparable. The human
(a) Pestle and the paper box
that is ﬁlled with metal bars.
(b) Failed grasp, not robust
against perturbations.
(c) Mediocre grasp, stable but
incorrect orientation.
(d) Good grasp, stable with in-
tended orientation.
Fig. 3: Examples of different grasps and their categorization. Grasps count as failed if the object is either not picked up at all or if small
perturbations would make the object drop. Grasps that are stable but do not keep the original orientation of the object count as OK but not
successful grasps. Grasps that are both stable and keep the original orientation count as successful grasps.
0 200 400 600 8001000
900
800
700Reward
Rollouts
0 200 400 600 800
Rollouts
# User Inputs150
100
50
0Vanilla RL
UCB
PI
EI
HedgeFive Link Via Point Task
Fig. 4: We evaluated our approach on a programmed, but noisy
expert to emulate human expert input. Vanilla RL REPS queries the
programmed expert for every sample, while our approach builds a
model of the reward function and only queries the expert occasionally.
expert could give rewards on a vertical bar through a graphical
interface.
1) Evaluation of Acquisition Functions: The evaluation of
the available AFs given in Fig. 4 shows that even though there
was only limited difference in the asymptotic performance,
there was considerable difference in the sample efﬁciency.
Especially the GP-UCB AF asks for many user queries. While
PI had the lowest asymptotic performance, as it is the most
greedy of the presented AFs, it also required the lowest number
of user queries. This behavior makes it an interesting candidate
when trying to minimize human interactions.
2) Evaluation of Uncertainty Threshold: To optimally trade
off the number of queries and the agents performance we
need to set the uncertainty threshold trade-off parameter <
(o)=p. This parameter expresses how certain we require
our algorithm to be that a proposed query is not explained by
the estimated observation noise. If we choose to be greater
than1, we require our estimate to improve on the observation
noise. Fig. 5 show the effects of adjusting . The performance
remained stable up to = 1:3and started degrading with
= 2. Changing the order of magnitude of resulted in a
failure to learn the task. While the effects of setting = 1:5on
the performance were moderate, the number of queries were
reduced by about 50% when compared to = 1:3.
3) Evaluation of Sparse Sampling: In order to minimize
human interaction, we stop improving the GP in each iteration
if the outcome that maximizes the AF has already been queried
before, instead of selecting the second best outcome according
to the AF. The results in Fig. 6 show, that sparse sampling
leads to equally good asymptotic performance but requires
considerably less expert interactions.
4) Evaluation Direct Learning: The premise of this paper
is that while BO can be used to efﬁciently learn the mapping
1000
800
600
4000 200 400 600 800Query Frequency
0 200 400 600 80001020304050
Rollouts Rollouts
# User InputsReward
λ= 5.0
λ= 1.5
λ= 1.3
λ= 1.1Fig. 5: We evaluated the impact of the threshold factor that
inﬂuences when and how often we sample. Lower values of led to
better converged performance but required more user interaction.
0 200 400 600 800
Rollouts1000
900
800
700
600
5000 200 400 600 800
Rollouts
# User Inputs150
100
50
0Sparse Sampling
Vanilla RL: REPS
PI Sparse
PI Reward
Fig. 6: Results of comparing our acquisition algorithm to the
standard acquisition algorithm. Our algorithm collects sparse user
queries and does not ask for any user queries after a PS iteration if
the outcome that maximizes the AF has already been queried before.
of outcome to reward, it would be too sample intensive to
directly learn the mapping from parameter space to outcome.
We validated this premise by comparing our joint approach to
directly learning the reward from the control parameters (i.e.,
from the basis function weights !). The results in Fig. 8 show
that BO did not converge to a good solution within 50expert
queries. Both methods were using PI.
5) Comparison to Inverse Reinforcement Learning: We
compared our algorithm to the Maximum Entropy IRL ap-
proach [36] on a via point task in two different scenarios.
Trajectories generated by DMPs had to pass one or two via
points, respectively (20 dimensional action space for each).
0 200 400 600800
Human Expert
Programmed Expert
RolloutsHuman Expert ScenarioReward
0 200 400 600800
Rollouts10203040
# User Inputs
5006007008009001000
Fig. 7: We validated our approach of using a noisy programmed
expert as substitute for a human expert on the simulated tasks. The
results show that both experts yielded similar learning behavior.
# QueriesRewardDirect Learning Standard Error
0 20 40 60 80 1002006001000
Active Reward Learning
Bayesian Optimization
Fig. 8: Comparison of ARL and directly learning the reward from
the control parameter !using BO. In this ﬁgure we plot the
standard error as opposed to the standard deviation. ARL performs
signiﬁcantly better after 50 queries ( p<0:05).
#UI = 5 #UI = 10 #UI = 20
ARL 1VP 995:22:477 999:20:25 999:20:25
IRL 1VP 9833:306 977 :36:334 9843:036
ARL 2VPs 970:713:1 9981:13 999:10:172
IRL 2VPs 994:11:874 9960:4736 9960:555
TABLE II: Comparison of IRL and ARL. We compare the methods
on two via point (VP) tasks with one and two VPs. The columns
show the achieved performance (mean and standard deviation) after
ﬁve, ten or 20 user interactions (UIs).
For this comparison only, we relaxed our assumption that no
demonstrations are available and provided the IRL approach
with (imperfect) demonstrations that passed close to the via
point (at most 0:1units away). In Table II, we show the mean
and standard deviation of the best policy reached for either
ﬁve, ten or 20 user interactions (UI). In our approach, UIs
are ratings while in IRL UIs are demonstrations. The results
show that our approach yields competitive results while not
requiring access to demonstrations.
6) Alternative Policy Learners: While we used REPS for
most of our experiments, the proposed framework is not lim-
ited to a speciﬁc RL method. To investigate the compatibility
with other methods, we compared our framework using REPS
with our framework in combination with a Bayesian Policy
Gradient (BPG) method [13] on a via point task with one via
point (20 dimensional action space). BPG models the gradient
of the policy using a GP and uses the natural gradient in the
policy update. The results in Fig. 9 show that both methods
were able to learn the task in combination with our approach.
B. Robot Grasping
We used the results from Section IV-A to set the parameters
for the real robot experiment (we use PI and set = 3). We
learned the policy (!js)with 15samples per iteration for
a total of 10iterations and we repeated the experiment three
times. The control parameters of the policy were the 15joints
of the ﬁve ﬁnger DLR hand, which is mounted to a 7DOFs
REPS
BPG
0 200 600 1000 0 200 600 100056789101112
Rollouts RolloutsReward
#iUseriInputsBayesianiPolicyiGradient
1000
500
0
-500
-1000
0
Fig. 9: Performance of REPS and BPG on a via point task.KUKA lightweight arm as shown in Fig. 1. We considered
the task of blind grasping as described in [8], where no object
information was available and we did not have visual feedback,
i.e., we did not have information about the contact points.
Instead, we calculated the forces in the ﬁnger tips through the
joint torques and and the hand kinematics. We used the ﬁnger
tip force magnitudes as outcome features which were used by
the GP to model the reward function.
Evaluating the real robot experiments presented the problem
of a success metric. As we did not have a ‘correct’ reward
function that we could evaluate the learned reward function
against, we resorted to introducing three label categories
which were only used for the evaluation after ﬁnishing the
experiments. The scheme presented in Fig. 3 labels grasps as
failures with a reward of 1, if the object was not lifted at all
or slipped when slightly perturbed. Grasps that were stable but
did not keep the intended orientation were given a reward of
0. Finally, grasps that lifted the object and kept the orientation
of the object were assigned a reward of 1. These labels and
reward values were not used during the learning of either the
policy or the reward model but only used to present results.
During the learning phase, the human expert assigned grasp
ratings in the range of 1000 .
1) Learn to Grasp Unknown Object: The object to be
grasped was a cardboard box ﬁlled with metal weights such
that the robot cannot grasp the object with very unstable grasps
or by deforming the paper box. The box, shown in Fig. 3a, was
of size 7:5cm x 5:5cm x 2cm and ﬁlled with two metal bars
with a combined weight of 350g. The results of three trials
presented in Fig. 10 show that the robot learned to perform
a successful grasp of the object in all three trials, while only
requesting six queries in the ﬁrst and twelve queries in the
second trial. In the last trial, the robot’s performance ﬁrst
increased quickly but dropped after 80episodes (or rollouts),
coinciding with a sudden increase of user queries, such that
the ﬁnal number of queries in the last trials was 27. The
reason for this unusual behavior was a malfunction of the
distal joint of the thumb, which rendered the grasping scheme
dysfunctional. As the learner could not reproduce outcomes
that led to good rewards, it resorted to ﬁnding a different grasp
strategy. At the same time, since the GP was presented with
new outcome samples in previously unobserved regions of the
outcome space, it requested new user queries to model the
reward function in the new region of interest.
We compared our learned reward function to a (naive) hand
coded reward function based on the same features. The hand
coded reward function aimed to reach a total force magnitude
over all ﬁngers. Using the programmed reward, the robot was
able to reliably pick up the object after the ﬁrst two trials and
in ten out of 15 grasps at the end of the third trial (We expect
the robot would have also learned to pick it up reliably in the
last trial with more iterations). However, the robot did not pick
up the object in a way that kept the original orientation of the
object. Encoding such behavior through only force features by
hand is challenging. The performance curve of the hand coded
reward function shows a slight dip, which is possible as we
Real Robot Results and QueriesGrasp Rating
Queries
-11
525
Rollouts2040 120140 60 80 100
Rollouts2040 120140 60 80 100
Trial 1
Trial 3Trial 2
Average
Hand CodedFig. 10: The real robot successfully learned good grasps in all of the
three trials. The grasp rating scheme is described in Fig. 3. In the last
trial, one joint failed, represented by a spike in expert queries. The
robot successfully adapted the reward function to recover from the
hardware failure. We also show the average performance of a naive
hand coded reward function.
-11
20 40 60 80 100 120 140Real Robot Transfer
RolloutsGrasp RatingMean and Standard Deviation0
Fig. 11: Performance of one trial on the real robot system. The robot
uses the reward function learned in trial one of the original task to
learn to grasp a new, unknown object (a wooden pestle as shown in
Fig. 3a). For this trial, we did not allow any expert queries. The robot
successfully learned to grasp the object.
are not plotting the internal reward but the reward assigned
according to the grading scheme introduced in Fig. 3.
2) Transfer Learned Reward Function to New Object: As
we based our reward model only on the ﬁnger tip forces, it
is modular and can also be used for (reasonably) different
objects. To test these generalization capabilities, we started a
new trial with a different object (a pestle as shown in Fig.
3a) and initialized the GP with the training data from the ﬁrst
trial of the previous task. For this experiment we only used
ratings from the previous trial, and did not ask for additional
expert ratings. The pestle that we used for this task was similar
in dimensions but different in shape, such that the agent had
to learn different joint conﬁgurations to achieve similar ﬁnger
forces that optimized the reward function. The dimensions of
the pestle were 18cm length, 1:5cm radius on the thin end and
2:25cm radius on the thick end. The results in Fig. 11 show
that the agent was able to learn to reliably perform a robust
grasp on the pestle.
V. R ELATED WORK
The term reward shaping has been used to describe efforts to
adapt rewards functions such that the resulting policy remains
invariant but learning speed is increased [27, 10, 4, 32] or
that learning on a new task is accelerated [21]. Aside from
reward shaping, Dorigo and Colombetti [11] have used the
term robot shaping to describe efforts of teaching different
tasks to a robotic agent. Derived from this terminology the
TAMER framework [19] uses the term interactive shaping.
In TAMER the agent receives reinforcements from a human,
guiding the learning process of the agent, but there is no activecomponent. The Advise framework [14] also uses the term
shaping to describe the process of modelling an oracle and
are actively requesting labels for an agent’s action (good or
bad). This approach is tailored for discrete settings and the
feedback frequency is pre-deﬁned.
In preference learning algorithms, the expert is usually
requested to rank the current best against a new execution
and the reward function is inferred from these rankings [2, 6].
Akrour et al. [1] can also deal with noisy rankings. Chu
and Ghahramani [7] introduced the use of GPs to preference
rankings. In preference based approaches, the expert is limited
to transmit one bit of information and cannot express strong
preferences, as, supposedly, human experts cannot give numer-
ical rewards that are sufﬁciently accurate. Our contribution
is to show how a rating based approach with explicit noise
model can be used in real robot continuous state-action space
problems taking advantage of the stronger guidance through
strong preferences (large differences in assigned rewards).
The problem of expert noise has also been addressed in
Bayesian RL and IRL. In Bayesian RL, Engel et al. [12]
has proposed to model the value function through a GP
and Ghavamzadeh and Engel [13] has proposed to model the
policy gradient through a GP. Especially the policy gradient
method is also well suited to work in combination with our
approached framework. If demonstrations are available, IRL
is a viable alternative. Ziebart et al. [36] have relaxed the
assumptions on the optimality of demonstrations such that a
reward function can be extracted from noisy demonstrations.
In a combination of IRL and preference learning, Jain et al.
[18] have proposed the iterative improvement of trajectories.
In their approach, the expert can choose to rank trajectories
or to demonstrate a new trajectory that does not have to be
optimal but only to improve on the current trajectory. This
approach cannot directly be used on learning tasks such as
grasping, as a forward model is required. Alternatively, Lopes
et al. [24] propose a framework where IRL is combined with
active learning such that the agent can decide when and where
to ask for demonstrations.
VI. C ONCLUSION & F UTURE WORK
We presented a general framework for actively learning the
reward function from human experts while learning the agent’s
policy with any PS or policy gradient method. Our experiments
showed that the learned reward function outperforms naive
hand coded reward functions, generalizes to similar tasks
and that the approach is robust to sudden changes in the
environment, for example when a mechanical failure occurs.
In future work we plan to extend on the ground-laying work of
this paper and investigate possible synergies between speciﬁc
PS methods and the reward learning framework to improve on
the performance of the generic AFs.
ACKNOWLEDGMENTS
The authors want to thank for the support of the European
Union projects # FP7-ICT-270327 (Complacs) and # FP7-ICT-
2013-10 (3rd Hand).
REFERENCES
[1] Riad Akrour, Marc Schoenauer, and Michele Sebag.
Preference-based policy learning. In Machine Learning
and Knowledge Discovery in Databases . 2011.
[2] Riad Akrour, Marc Schoenauer, and Michele Sebag.
Interactive robot education. In European Conference on
Machine Learning Workshop , 2013.
[3] Ravi Balasubramanian, Ling Xu, Peter D Brook,
Joshua R Smith, and Yoky Matsuoka. Physical human
interactive guidance: Identifying grasping principles from
human-planned grasps. IEEE Transactions on Robotics ,
2012.
[4] Jeshua Bratman, Satinder Singh, Jonathan Sorg, and
Richard Lewis. Strong mitigation: Nesting search for
good policies within search for good reward. In Interna-
tional Conference on Autonomous Agents and Multiagent
Systems , 2012.
[5] Maya Cakmak and Andrea L Thomaz. Designing robot
learners that ask good questions. In International con-
ference on Human-Robot Interaction , 2012.
[6] Weiwei Cheng, Johannes F ¨urnkranz, Eyke H ¨ullermeier,
and Sang-Hyeun Park. Preference-based policy iteration:
Leveraging preference learning for reinforcement learn-
ing. In Machine Learning and Knowledge Discovery in
Databases . 2011.
[7] Wei Chu and Zoubin Ghahramani. Preference learning
with Gaussian processes. In International Conference on
Machine Learning , 2005.
[8] Hao Dang and Peter K Allen. Learning grasp stability.
InInternational Conference on Robotics and Automation ,
2012.
[9] Christian Daniel, Gerhard Neumann, and Jan Peters.
Learning Sequential Motor Tasks. In International Con-
ference on Robotics and Automation , 2013.
[10] Sam Devlin and Daniel Kudenko. Dynamic potential-
based reward shaping. In International Conference on
Autonomous Agents and Multiagent Systems , 2012.
[11] Marco Dorigo and Marco Colombetti. Robot shaping:
Developing autonomous agents through learning. Artiﬁ-
cial intelligence , 1994.
[12] Yaakov Engel, Shie Mannor, and Ron Meir. Reinforce-
ment learning with Gaussian processes. In International
Conference on Machine Learning , 2005.
[13] Mohammad Ghavamzadeh and Yaakov Engel. Bayesian
policy gradient algorithms. Advances in Neural Informa-
tion Processing Systems , 2007.
[14] Shane Grifﬁth, Kaushik Subramanian, Jonathan Scholz,
Charles Isbell, and Andrea L Thomaz. Policy shaping:
Integrating human feedback with reinforcement learning.
InAdvances in Neural Information Processing Systems ,
2013.
[15] Harold W Hake and WR Garner. The effect of present-
ing various numbers of discrete steps on scale reading
accuracy. Journal of Experimental Psychology , 1951.
[16] Matthew D Hoffman, Eric Brochu, and Nando de Freitas.Portfolio allocation for Bayesian optimization. In Con-
ference on Uncertainty in Artiﬁcial Intelligence , 2011.
[17] Auke Ijspeert, Jun Nakanishi, and Stefan Schaal. Learn-
ing Attractor Landscapes for Learning Motor Primitives.
InAdvances in Neural Information Processing Systems .
2003.
[18] Ashesh Jain, Brian Wojcik, Thorsten Joachims, and
Ashutosh Saxena. Learning trajectory preferences for
manipulators via iterative improvement. In Advances in
Neural Information Processing Systems , 2013.
[19] W Bradley Knox and Peter Stone. Interactively shaping
agents via human reinforcement: The TAMER frame-
work. In International Conference on Knowledge Cap-
ture, 2009.
[20] Jens Kober, Betty J. Mohler, and Jan Peters. Learning
perceptual coupling for motor primitives. In Intelligent
Robots and Systems , 2008.
[21] George Konidaris and Andrew Barto. Autonomous
shaping: Knowledge transfer in reinforcement learning.
InInternational Conference on Machine Learning , 2006.
[22] Petar Kormushev, Sylvain Calinon, and Darwin Caldwell.
Robot motor skill coordination with EM-based reinforce-
ment learning. In Intelligent Robots and Systems , 2010.
[23] Oliver Kroemer, Renaud Detry, Justus Piater, and Jan
Peters. Combining active learning and reactive control
for robot grasping. Robotics and Autonomous Systems ,
2010.
[24] Manuel Lopes, Francisco Melo, and Luis Montesano.
Active learning for reward estimation in inverse rein-
forcement learning. In Machine Learning and Knowledge
Discovery in Databases . 2009.
[25] George A Miller. The magical number seven, plus or
minus two: some limits on our capacity for processing
information. Psychological Review , 1956.
[26] Andrew Ng and Adam Coates. Autonomous inverted he-
licopter ﬂight via reinforcement learning. Experimental
Robotics IX , 1998.
[27] Andrew Ng, Daishi Harada, and Stuart Russell. Policy
invariance under reward transformations: Theory and
application to reward shaping. In Internation Conference
on Machine Learning , 1999.
[28] Jan Peters, Katharina M ¨ulling, and Yasemin Altun. Rel-
ative entropy policy search. In National Conference on
Artiﬁcial Intelligence , 2010.
[29] Carl E. Rasmussen and Christopher K. I. Williams.
Gaussian Processes for Machine Learning . 2006.
[30] N. Ratliff, A. Bagnell, and M. Zinkevich. Maximum Mar-
gin Planning. In International Conference on Machine
Learning , 2006.
[31] Nathan Ratliff, David Silver, and Andrew Bagnell.
Learning to Search: Functional Gradient Techniques for
Imitation Learning. Autonomous Robots , 2009.
[32] Satinder Singh, Richard L Lewis, Andrew G Barto, and
Jonathan Sorg. Intrinsically motivated reinforcement
learning: An evolutionary perspective. Autonomous Men-
tal Development , 2010.
[33] Niranjan Srinivas, Andreas Krause, Sham M Kakade,
and Matthias Seeger. Gaussian process optimization in
the bandit setting: No regret and experimental design.
Internation Conference on Machine Learning , 2010.
[34] Ra ´ul Su ´arez Feij ´oo, Jordi Cornell `a Medrano, and
M´aximo Roa Garz ´on. Grasp quality measures. 2012.
[35] Andrea L Thomaz and Cynthia Breazeal. Teachable
robots: Understanding human teaching behavior to build
more effective robot learners. Artiﬁcial Intelligence ,
2008.
[36] Brian Ziebart, Andrew Maas, Andrew Bagnell, and
Anind Dey. Maximum entropy inverse reinforcement
learning. In Conference on Artiﬁcial Intelligence , 2008.

Reward learning on continuous control tasks, when using Relative Entropy Policy Search (REPS) for the policy. (This paper is from 2014, before deep learning was applied to robotics tasks.) Their setup is similar to deep RL from human preferences, where they have a reward predictor, which they then use to train a policy, which then generates trajectories and sends it back to the reward predictor. The reward predictor chooses a few trajectories and asks the expert about them; the expert provides a rating (a number), not a preference. This is then used to update the reward predictor.

It's another example to compare to deep RL from human preferences.

Deep Reinforcement Learning from Human Preferences