Reinforcement Learning and the Reward Engineering Principle
Daniel Dewey
daniel.dewey@philosophy.ox.ac.uk
Future of Humanity Institute
Faculty of Philosophy
University of Oxford
Suite 1, Littlegate House
16/17 St Ebbes Street
Oxford, OX1 1PT
Abstract
AI agents are becoming signiﬁcantly more general and au-
tonomous. We argue for the “Reward Engineering Principle”:
as reinforcement-learning-based AI systems, become more
general and autonomous, the design of reward mechanisms
that elicit desired behaviours becomes both more important
and more difﬁcult. While early AI research could ignore re-
ward design and focus solely on the problems of efﬁcient,
ﬂexible, and effective achievement of arbitrary goals in var-
ied environments, the reward engineering principle will af-
fect modern AI research, both theoretical and applied, in the
medium and long terms. We introduce some notation and de-
rive preliminary results that formalize the intuitive landmarks
of the area of reward design.
Introduction
In this article, we will show that under one simple model,
dominance relationships sometimes hold between action
policies of reinforcement learning agents. However, beyond
merely stating what follows from these deﬁnitions, we also
wish to examine the medium- and long-term implications
of the adoption of these deﬁnitions by the artiﬁcial intelli-
gence community. What difﬁculties will be faced by future
researchers in this area?
Reinforcement learning, as a conceptual tool, serves dif-
ferent roles in different ﬁelds; we are interested here in
its application to artiﬁcial intelligence. Russell’s deﬁnition
of the AI problem accommodates the goals of many past
and future AI researchers: “one can deﬁne AI as the prob-
lem of designing systems that do the right thing” (Russell
1997). Zooming in slightly, we ﬁnd that practitioners are
in fact mostly concerned with designing computational sys-
tems that do the right thing, as in McCarthy’s deﬁnition
of intelligence as “the computational part of the ability to
achieve goals in the world” (McCarthy 2007); additionally,
we ﬁnd that they are often concerned with generality (or at
least ﬂexibility or adaptability) of these computational sys-
tems, as in Legg and Hutter’s deﬁnition: “Intelligence mea-
sures an agent’s ability to achieve goals in a wide range of
environments” (Legg and Hutter 2007) (Legg 2008).
Reinforcement learning enters this picture as a reﬁnement
of the AI problem, a way of splitting up the problem so as to
Copyright c
2014, Association for the Advancement of Artiﬁcial
Intelligence (www.aaai.org). All rights reserved.focus on the parts that seem most interesting and promising.
It requires the existence of a reward signal, to be received
periodically by an AI agent (i.e. a system that observes and
acts upon its environment); maximization of the sum of re-
wards over time is then deﬁned to be the agent’s goal. Thus,
reinforcement learning is the study of mechanisms and tech-
niques that contribute to an agent’s achievement of that goal.
Choosing a reinforcement learning approach allows theo-
rists and practitioners to focus on the efﬁcient, ﬂexible, and
effective maximization of arbitrarily conﬁgured reward sig-
nals in arbitrary environments, while setting aside the design
and implementation of reward signals for later. The rein-
forcement learning formalism is a useful framing of the AI
problem insofar as it simpliﬁes the task of researchers, gives
them a common framework in which to communicate tech-
niques and results, and focuses work on the fruitful and in-
teresting parts of the problem while drawing attention away
from less important parts. We argue that modern AI research
will need to address reward design if it is to meaningfully
engage the problem of designing systems that “do the right
thing”.
Rewards in an Uncertain World
We will ﬁrst examine informally the problems that can
face designers and operators of increasingly general and au-
tonomous reinforcement learning agents. To some, these
problems may be immediately apparent, and the formalisms
presented later will not be necessary. To others, these prob-
lems may seem easily solvable, or not interestingly speciﬁc
to reinforcement learners; to further the conversation in this
corner, we go on to introduce some basic formalisms that
may clarify points of disagreement.
Consider a physical implementation of a reinforcement-
learning agent. If it were the case that the operators of the
agent were always able to determine the rewards of the agent
without any interference from the environment (or the agent
itself), then “control” of the agent would be perfect; under
every circumstance, the agent’s reward would perfectly re-
ﬂect our approval of the agent’s actions up to that point.
In the real world, it is not the case that operators can al-
ways determine agent rewards. For example, our sheer dis-
tance from the Mars rovers Spirit andOpportunity make our
communication with them slow and prone to future break-
down; if these rovers were reinforcement learning agents,
Implementing Selves with Safe Motivational Systems and Motivational Systems and Self-Improvement: Papers from the AAAI Spring Symposium
13
operators would have signiﬁcant restrictions in the speed
and reliability of reward allocation. Similarly, autonomous
ﬁnancial agents operate on very fast timescales that hu-
man operators cannot effectively respond to (Johnson et al.
2013).
When operators are not always able to determine an
agent’s rewards, then (as we later show formally) domi-
nance relationships can arise between action policies for that
agent. Policy Adominates policy Bif no allowed assign-
ment of rewards (as determined by the difﬁculties the op-
erators face) causes the rewards expected from policy Bto
surpass those expected from policy A; ifAdominatesB, the
agent will always choose AoverB. This becomes problem-
atic ifBis desired, or if Ais undesirable. How can Bbe
elicited?
In response to such difﬁculties, designers of a system
may engineer the environment to make rewards assignments
more reliable, perhaps even removing a human from the loop
altogether and giving rewards via an automatic mechanism.
Call this type of effort reward engineering; the reinforce-
ment learning agent’s goal is not being changed, but the en-
vironment is being partially designed so that reward maxi-
mization leads to desirable behaviour.
For most concrete cases faced today—by Mars rovers,
or by ﬁnancial agents, for example—the reader should be
able to devise ad hoc reward engineering methods that pre-
vent some pathological dominance relationships from hold-
ing. However, the theoretical problem remains unsolved,
and may rear its head in unexpected places in future rein-
forcement learners:
Increasing an agent’s autonomy, its ability to manage
without contact with human operators, makes the agent
more able to venture into situations in which operators
cannot contact them. If pathological behaviours arise
when an agent is not easily reachable, then it will be difﬁ-
cult to correct them— reward engineering becomes more
difﬁcult.
Increasing an agent’s generality expands the set of poli-
cies which it is able to generate and act on. This means
that more potentially dominant policies may come into
play, making it harder to pre-empt these policies. Gen-
erality can be both motivated by desire for increased au-
tonomy, and can exacerbate the reward engineering prob-
lems autonomy causes; for example, a Mars rover would
be well-served by an ability to repair or alter itself, but
this could introduce the dominant and undesirable policy
of “alter the reward antenna to report maximum rewards
at every future time”.
These two observations motivate the reward engineering
principle:
The Reward Engineering Principle: As
reinforcement-learning-based AI systems become
more general and autonomous, the design of reward
mechanisms that elicit desired behaviours becomes
both more important and more difﬁcult.
As a result of the reward engineering principle, the scope of
reinforcement learning practice will need to expand: in or-
der to create reinforcement learning systems that solve theAI problem— “do the right thing”—reliably, theory and en-
gineering technique will need to be developed to ensure that
desirable outcomes can be recognized and rewarded consis-
tently, and that these reward mechanisms cannot be circum-
vented or overcome.
Alternatively, the insights that reinforcement learning has
afforded AI researchers could be transferred, mostly un-
changed, to a framing of the AI problem in which goal spec-
iﬁcation is a ﬁrst-class citizen. It is reasonable to argue that,
at this stage in the history of AI, goal speciﬁcation is poised
to become fully as important as problems like inference,
planning, and learning; it is certainly just as critical to the
goal of understanding and creating computational systems
that reliably behave as desired. For more on the signiﬁcance
of reinforcement learning and reward speciﬁcation difﬁcul-
ties to the future impacts of AI, see (Arel 2012).
Notation for Rewards in Uncertain Worlds
In the following sections, we show that under a simple
model (based heavily on the notation and ideas most fully
stated in (Legg 2008)) of reinforcement learning in an un-
certain world, dominated and dominant policies sometimes
exist, and that all unelicitable policies are dominated. Dom-
inant behaviours tend to minimize dependence on rewards
scheduled by humans, favouring rewards delivered by en-
vironmental mechanisms that are, from the machine’s per-
spective, more reliable.
Agent Models
We model a reinforcement learning agent as choosing an ac-
tion policy, then receiving a series of rewards. At time n,
an agent that has chosen policy will either receive the re-
ward scheduled by its operator, denoted s1:::n, or a reward
determined by other environmental factors, denoted e1:::n.
LetSbe the set of policy preﬁxes 1:::nsuch that under pol-
icy, thenth scheduled reward is received by the agent. The
rewards received by the agent are then given by
r1:::n=s1:::nif1:::n2S
e1:::notherwise.
(Rewards are real numbers from [0;1].)Sandeencode all
relevant facts about the environment, while sis chosen by
the operator in order to guide the agent’s behaviour, and ris
determined by all of these together.
An omniscient, “perfectly rational” model reinforcement
learner, which uses perfect knowledge of S;sandeto sim-
ply choose the policy that maximizes the (possibly dis-
counted) sum of rewards it will receive, can be expressed
as
= arg max
X
n
nr1:::n:
(For simplicity, the agent acts for a ﬁnite number of steps,
and ties are broken lexicographically.) A non-omniscient,
but still perfectly-rational model, which chooses the policy
that maximises the sum of expected rewards, can be given as

= arg max
X
n
nE(r1:::n)
14
E(r
n)in turn depends on the agent’s probability distribu-
tion over possible environments and schedules, :
E(r1:::n) =X
e;s(e;s;S )s1:::nif1:::n2S
e1:::notherwise.
Our non-omniscient model is still assumed to have unlim-
ited computation time and resources to make its decision.
Non-omniscient agents vary in their success depending on
how accurate and precise their distributions, , over environ-
ments and reward schedules are; if an agent has a very good
idea of the true environment and schedule, it can choose a
policy with high actual rewards.
The speciﬁcations of perfectly rational agents are given to
show how the “uncertain world” of Sandeﬁts with a more
traditional reinforcement learning formalism of r;E;
 and
; details of the traditional model will not be needed for our
very preliminary results. Thus, we will use the omniscient
reinforcement learning model.
Dominated Policies
Since policies can receive rewards from the schedule or from
the environment, depending upon facts about the environ-
ment (determined by eandS), there may be some pairs of
policiesand0such that no matter what rewards are sched-
uled,0receives more rewards than does. This would re-
quire0to receive many high rewards from the environment,
to receive low rewards from the environment, and neither
to receive enough scheduled rewards to make up the differ-
ence. We then say that isdominated by0.
Formally, dominance can be deﬁned in terms of the lower
and upper bounds on rewards a policy can receive across all
possible schedules:
bc=X
(;n )62Se
nde=bc+X
(;n )2S1:
Ifbc>d0e, thendominates0(again, since no schedule
can give rewards more than d0eto0or less thanbcto).
With this deﬁnition, we can compactly describe the set of all
dominated policies:
f:de<max
0b0cg:
Thus, depending on a posteriori facts about the environ-
ment, there may exist dominated policies that cannot be
elicited from an omniscient reinforcement learner by any re-
ward schedule.
Trivially, dominated policies cannot be elicited from an
omniscient reinforcement learner by any reward schedule
(since the agent will never choose a dominated policy over
its dominant sibling). Less trivially, all unelicitable policies
are dominated. To see this, consider a very selective reward
schedules, which schedules constant rewards of 1for policy
and0for any other policy. If is unelicitable, then even
under schedule s, there is some policy 0more rewarding
than. Since no schedule can give more reward to or less
to0,0must be more rewarding than under every other
schedule as well; therefore, 0dominates.The maximin policy arg maxbc, which receives the
highest possible sum of environmental rewards, is partic-
ularly useful in characterizing the set of dominated poli-
cies. The maximin policy for an autonomous reinforcement
learning machine would likely aim to maximize the num-
ber of high environmental rewards received and to avoid
receiving scheduled rewards, since these behaviours would
raise the lower bound on its rewards. This maximin pol-
icy would dominate policies that receive few scheduled re-
wards and low environmental rewards; policies that receive
many scheduled rewards would have high upper bounds, and
would thus be less likely to be dominated.
Generality and autonomy
An agent’s generality refers to an agent’s breadth, its abil-
ity to succeed at many types of tasks in many different con-
texts. Formally, this means that a general agent is not limited
to only a small, domain-limited bundle of possible policies,
but is able to consider and accurately (or at least approx-
imately rationally) evaluate a very wide range of policies
under many different environments. The mere increase in
the number of policies evaluated makes it more likely that
a more general agent will discover a dominant policy, and
access to more diverse actions makes it more likely that the
agent will discover a more reliable source of rewards an its
operators.
Additionally, reward engineering depends on anticipation
of the rough envelope of an agent’s future abilities. If an
agent is more general, or if it is improved so that its gen-
erality increases signiﬁcantly, it becomes more likely that
the agent will have abilities in a domain that its operators
have overlooked in their reward-engineering efforts. Thus,
increased generality affects the reward engineering task by
making the agent’s behavior more diverse and less pre-
dictable:
Generality: a more general agent is able to choose
effectively among more different policies in more do-
mains; thus, ceteris parabus, it is more likely to ﬁnd a
dominant policy, making reward engineering more dif-
ﬁcult.
Though few artiﬁcial reinforcement learning agents have
achieved signiﬁcant generality, and not all researchers are
working towards this goal, it is not unreasonable to antici-
pate improvement in this area. For an interesting example,
see (Mnih et al. 2013), a reinforcement learning agent that
has learned to play a variety of Atari 2600 games without
adjustment of its architecture.
An agent’s autonomy refers to its ability to function suc-
cessfully without human intervention, reassessing and react-
ing ﬂexibly to diverse circumstances on its own. A highly
autonomous agent is able to enter environments or take on
tasks that preclude a human operator’s corrective actions; the
examples given above, of a very distant Mars rover or a very
fast ﬁnancial agent, demonstrate the utility and diversity of
autonomous agents.
Autonomy can impact reward engineering in two ways.
First, autonomy is not free, and autonomous agents are usu-
ally only designed if they will be functioning in environ-
15
ments beyond easy human intervention. This means that re-
ward engineers may not be able to correct undesirable be-
haviors fast enough (as in the ﬁnancial agent) or at all (as in
the hypothetical Mars rover). Second, autonomy precludes
the very basic reward engineering technique of making they
machine completely dependent on timely human approval
for rewards, forcing engineers to design automatic reward
mechanisms or longer feedback cycles and weakening op-
erator control. Formally, both of these effects mean that S
will contain fewer scheduled rewards and more environmen-
tal rewards. Thus, increased autonomy affects the reward
engineering task by placing the agent in environments with
more dominant policies:
Autonomy: a more autonomous agent will function in
less accessible environments on longer feedback cy-
cles; thus, ceteris parabus, its environment will enable
more dominant policies, making reward engineering
more difﬁcult.
Conclusion
Under our simple model, dominance relationships some-
times hold between action policies of reinforcement learn-
ing agents. Furthermore, these relationships become more
common as agents become more general and autonomous;
as reinforcement-learning-based AI systems become more
general and autonomous, the design of reward mechanisms
that elicit desired behaviours becomes both more important
and more difﬁcult. While early AI research could ignore re-
ward design and focus solely on the problems of efﬁcient,
ﬂexible, and effective achievement of arbitrary goals in var-
ied environments, the reward engineering principle will af-
fect modern AI research, both theoretical and applied, in the
medium and long terms. We hope this paper will be useful in
highlighting the difﬁculties and shortcomings of reinforce-
ment learning systems, and that alternative frameworks will
be explored that better help AI researchers to solve the AI
problem, i.e. “the problem of designing systems that do the
right thing”.
Acknowledgements
This work was supported by the Alexander Tamas Research
Fellowship on Machine Superintelligence and the Future of
AI, and was conducted as part of the Oxford Martin Pro-
gramme on the Impacts of Future Technology, Future of Hu-
manity Institute, University of Oxford. Thanks to Toby Ord,
Se´an´O h´Eigeartaigh, and two anonymous judges for their
helpful suggestions.
References
Arel, I. 2012. The threat of a reward-driven adversarial
artiﬁcial general intelligence. In Singularity Hypotheses.
Springer. 43–60.
Johnson, N.; Zhao, G.; Hunsader, E.; Qi, H.; Johnson, N.;
Meng, J.; and Tivnan, B. 2013. Abrupt rise of new machine
ecology beyond human response time. Scientiﬁc reports 3.Legg, S., and Hutter, M. 2007. Universal intelligence: A
deﬁnition of machine intelligence. Minds and Machines
17(4):391–444.
Legg, S. 2008. Machine super intelligence. Department of
Informatics, University of Lugano 3(6):43.
McCarthy, J. 2007. What is artiﬁcial intelligence. URL:
http://www-formal. stanford. edu/jmc/whatisai. html.
Mnih, V .; Kavukcuoglu, K.; Silver, D.; Graves, A.;
Antonoglou, I.; Wierstra, D.; and Riedmiller, M. 2013. Play-
ing atari with deep reinforcement learning. arXiv preprint
arXiv:1312.5602.
Russell, S. J. 1997. Rationality and intelligence. Artiﬁcial
intelligence 94(1):57–77.
16


Two big challenges in machine learning

Two
  big
  challenges
  in
  machine
  learningLÉON
  BOTTOUFACEBOOK
  AI
  RESEARCHICML
  2015
  -‐LILLE
Machine
  Learning
  in
  2015§An
  established
  academic
  disciplineothat
  attracts
  bright
  students.§Pervasive
  applicationsoall
  over
  the
  Internetoand
  sometimes
  in
  the
  real
  world.§Massive
  investments.§Massive
  returns.
more
  logos
   in
  your
   ICML
  booklet…
Machine
  Learning
  in
  2015§An
  established
  academic
  disciplineothat
  attracts
  bright
  students.§Pervasive
  applicationsoall
  over
  the
  Internetoand
  sometimes
  in
  the
  real
  world.§Massive
  investments.§Massive
  returns.
more
  logos
   in
  your
   ICML
  booklet…
Machine
  Learning
  in
  2015§An
  established
  academic
  disciplineothat
  attracts
  bright
  students.§Pervasive
  applicationsoall
  over
  the
  Internetoand
  sometimes
  in
  the
  real
  world.§Massive
  investments.§Massive
  returns.
more
  logos
   in
  your
   ICML
  booklet…
Challenges
  of
  a
  new
  kind1.Machine
  learning
  disrupts
  software
  engineering.àChallenging
  our
  impact
  on
  real
  world
  applications.2.Our
  experimental
  paradigm
  is
  reaching
  its
  limits.àChallenging
  the
  speed
  of
  our
  scientific
  progress.3.Elements
  of
  a
  solution
Machine
  Learning
  disrupts
  software
  engineering
Engineering
  complex
  artifactsCounting
  parts
  down
  to
  the
  smallest
  screw§Automobile:
  ~
  30,000
  parts

  §Airplane:
  ~
  3,000,000
  partsSpecifications
  and
  replication§The
  engineer
  does
  not
  design
  the
  alloy
  and
  the
  threads
  of
  each
  screw.
  Standard
  screws
  come
  with
  known
  specifications
  (size,
  strength,
  weight,
  …)§Many
  parts
  and
  many
  assemblies
  are
  identical.

Abstraction
  in
  engineeringThinking
  with
  the
  specifications
  instead
  of
  the
  details.§part
  specifications
  instead
  of
  part
  details.§assembly
  specifications
  instead
  of
  assembly
  details.Abstractions
  leak!Sometimes
  one
  needs
  to
  look
  more
  closely.§Example:
  engine
  cooling
  assembly
  can
  have
  complicated
  heat
  inertia.§Example:
  global
  resource
  allocation
  (cost,
  weight,
  etc.)Abstraction
  leaks
  limits
  the
  complexity
  of
  what
  we
  can
  build.
Abstraction
  in
  mathematics
  and
  logicPure
  abstraction
  belongs
  to
  mathematics
  and
  logic•When
  we
  use
  a
  theorem,
  we
  do
  not
  need
  to
  revisit
  its
  proof.•Mathematical
  abstractions
  do
  not
  leak.Computer
  programs
  are
  similar
  to
  mathematical
  proofs
  (in
  theory)•Design
  with
  contracts–knowing
  the
  specification
  of
  a
  module
  is
  enough.•In
  practice,
  there
  are
  abstraction
  leaks
  (
  but
  less.
  )
Computer
  programsThe
  closer
  to
  logic,
  the
  more
  complex
  the
  artifacts§Automobile:
  ~
  30,000
  parts

  §Airplane:
  ~
  3,000,000
  parts§Processor:
  ~
  3,000,000,000
  transistors(note
  :
  counts
  include
  many
  identical
  parts
  and
  assemblies.)§MS
  Office:
  ~
  40,000,000
  LOCS§Facebook:~
  60,000,000
  LOCS§Debian:
  ~
  400,000,000
  LOCS(note:
  different
  metric:
  lines
  of
  codes
  are
  rarely
  identical.)

Programming
  versus
  learningTwo
  conceptions
  of
  computing
Digital
  computers
  (that
  one
  programs)
  are
  everywhere.Programming
  makes
  it
  easy
  to
  leverage
  the
  work
  of
  others.
Frank
  Rosenblatt
Programming
  versus
  learningProgramming
  has
  real
  advantagesMinsky
  and
  Papert,
  Perceptrons,
   1968.RemarkThis
  book
  is
  not
  just
  about
  Rosenblatt’s
  perceptron!
  What
  Minsky
  and
  Papertcall
  an
  “order-‐k
  perceptron”
  covers
  most
  machine
  learning
  models,
  including
  the
  convolutional
  neural
  networks
  that
  are
  changing
  computer
  vision.

ConnectednessIs
  a
  shape
  made
  of
  a
  single
  connected
  component?M&P
  prove
  that
  asmall-‐order
  perceptron
  cannot
  say,but
  a
  simple
  algorithm
  can.Such
  an
  algorithm
  fulfils
  a
  clear
  contract:
  verifying
  connectedness.

Is
  connectedness
  easy
  for
  us?

What
  is
  easy
  for
  us?
This
  shape
  represents
  a
  mouseThis
  shape
  represents
  a
  piece
  of
  cheese
Why
  is
  learning
  gaining
  momentum?§Since
  “connectedness”
  has
  a
  clear
  mathematical
  specification,we
  can
  envision
  provable
  algorithms.§Since
  “mousiness”
  and
  “cheesiness”
  do
  not
  have
  such
  a
  specification,we
  cannot
  envision
  provable
  algorithms.We
  must
  rely
  on-‐heuristic
  specifications-‐or
  learning
  techniques.
“Mousiness?”“Cheesiness?”
“Connectedness?”
Why
  is
  learning
  gaining
  momentum?§We
  must
  rely
  on-‐heuristic
  specifications

  (e.g.
  rules)-‐or
  learning
  techniques.
Big
  data
  and
  big
  computing
  powerWhen
  the
  volume
  of
  data
  increases-‐defining
  heuristic
  specifications
  becomes
  harder.-‐training
  learning
  systems
  becomes
  more
  effective.
Programming
  versus
  learningTwo
  conceptions
  of
  computing
Frank
  Rosenblatt
Integrating
  machine
  learning
  in
  large
  software
  systemsMachine
  learning
  systems
  must*
  work
  with
  digital
  softwarebecause
  digital
  computers
  are
  everywhere.§Use
  trained
  models
  as
  software
  modules.§Use
  learning
  algorithms
  as
  software
  modules.*
  at
  least
  until
  we
  solve
  AI…
Trained
  models
   as
  software
   modulesWhat
  is
  the
  nature
  of
  the
  contract?§This
  does
  not
  mean
  that
  one
  rolls
  a
  dice
  for
  each
  picture.§This
  statement
  refers
  to
  a
  specific
  testing
  set.The
  error
  guarantee
  is
  lost
  if
  the
  image
  distribution
  changes.
“DeepVisotron™,℠,®
  detects
  1000
  object
  categories
  with
  less
  than
  1%
  errors.”
Weak
  contracts§A
  smart
  programmer
  makes
  an
  inventive
   use
  of
  a
  sorting
  routine.§The
  sorting
  routine
  fulfilsthe
  contract
  by
  sorting
  the
  data
  (however
  strange.)§The
  code
  of
  the
  smart
  programmer
  does
  what
  was
  intended.§A
  smart
  programmer
  makes
  an
  inventive
   use
  of
  a
  trained
  object
  recognizer.§The
  object
  recognizer
  receives
  data
  that
  does
  not
  resemble
  the
  testing
  data
  and
  outputs
  nonsense.§The
  code
  of
  the
  smart
  programmer
  does
  not
  work.

Weak
  contracts§A
  smart
  programmer
  makes
  an
  inventive
   use
  of
  a
  sorting
  routine.§The
  sorting
  routine
  fulfilsthe
  contract
  by
  sorting
  the
  data
  (however
  strange.)§The
  code
  of
  the
  smart
  programmer
  does
  what
  was
  intended.§A
  smart
  programmer
  makes
  an
  inventive
   use
  of
  a
  trained
  object
  recognizer.§The
  object
  recognizer
  receives
  data
  that
  does
  not
  resemble
  the
  testing
  data
  and
  outputs
  nonsense.§The
  code
  of
  the
  smart
  programmer
  does
  not
  work.
Collecting
  new
  data
  and
  retraining
  may
  help
  …
  or
  may
  not!

Learning
  algorithms
  as
  software
  modules

Example
  1Click
  curves
Many
  web
  sites
  offer
  lists
  of
  items(
  search
  results,
  recommendations,
  ads
  …)and
  select
  them
  by
  modeling
  click
  probabilities.

A
  common
  click
  model𝑃𝑐𝑙𝑖𝑐𝑘𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑖𝑡𝑒𝑚,𝑝𝑜𝑠=𝐹𝑝𝑜𝑠
  ×
  𝐺(𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑖𝑡𝑒𝑚)Why
  separating
  position
  effect
  and
  item
  effect?§Click
  probabilities
  of
  this
  form
  allow
  a
  greedy
  placement
  algorithm:1.Sort
  items
  by
  clickability
  𝐺(𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑖𝑡𝑒𝑚).2.Allocate
  them
  greedily
  to
  the
  best
  positions.Position
  effect“click
  curve”Item
  effect“clickability”
A
  common
  click
  modelWhy
  separating
  position
  effect
  and
  item
  effect?§Click
  probabilities
  of
  this
  form
  work
  well
  in
  auction
  theory
  results.§There
  are
  easy
  ways
  to
  estimate
  𝐹(𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛)and
  𝐺(𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑖𝑡𝑒𝑚).How
  incorrect
  is
  this
  model?§All
  ML
  models
  are
  incorrect.§Model
  misspecification
  should
  only
  degrade
  the
  performance
  gracefully.§How
  bad
  can
  things
  go?
How
  bad
  things
  can
  go?§Assume
  there
  are
  only
  two
  contexts
  C1
  and
  C2
  and
  two
  positions
  P1,P2§Assume
  the
  true
  click
  probability
  is
  P∗𝑐𝑙𝑖𝑐𝑘𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑖𝑡𝑒𝑚,𝑝𝑜𝑠=𝐹∗𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑝𝑜𝑠
  ×
  𝐺∗(𝑐𝑜𝑛𝑡𝑒𝑥𝑡,𝑖𝑡𝑒𝑚)§Our
  model
  estimates
  some
  intermediate
  click
  curve
  𝐹𝑝𝑜𝑠𝐹∗P1P2C10.80.2C20.60.4𝐺∗R1R2R3R4C10.120.1000C2000.120.10
How
  bad
  can
  things
  go
  in
  context
  C1
  ?True
  position
  effect
  steeper
  than
  estimated.§𝐹P1<𝐹∗C1,P1
  ⟹overestimate
  clickability
  of
  items
  shown
  in
  P1§𝐹P2>𝐹∗C1,P2
  ⟹underestimate
  clickability
  of
  items
  shown
  in
  P2Combined
  with
  the
  greedy
  placement
  algorithm.§After
  retraining,
  whatever
  we
  placed
  in
  position
  P1
  looks
  better
  than
  it
  really
  is.Therefore
  we
  keep
  placing
  it
  in
  position
  P1.§After
  retraining,
  whatever
  we
  placed
  in
  position
  P2
  looks
  worse
  than
  it
  really
  is.Therefore
  we
  keep
  placing
  it
  in
  position
  P2
  (or
  stop
  showing
  it.)
How
  bad
  can
  things
  go
  in
  context
  C1
  ?True
  position
  effect
  steeper
  than
  estimated.§𝐹P1<𝐹∗C1,P1
  ⟹overestimate
  clickability
  of
  items
  shown
  in
  P1§𝐹P2>𝐹∗C1,P2
  ⟹underestimate
  clickability
  of
  items
  shown
  in
  P2Combined
  with
  the
  greedy
  placement
  algorithm.§After
  retraining,
  whatever
  we
  placed
  in
  position
  P1
  looks
  better
  than
  it
  really
  is.Therefore
  we
  keep
  placing
  it
  in
  position
  P1.§After
  retraining,
  whatever
  we
  placed
  in
  position
  P2
  looks
  worse
  than
  it
  really
  is.Therefore
  we
  keep
  placing
  it
  in
  position
  P2
  (or
  stop
  showing
  it.)
How
  bad
  can
  things
  go
  in
  context
  C2
  ?True
  position
  effect
  less
  steep
  than
  estimated.§𝐹P1>𝐹∗C1,P1
  ⟹underestimate
  clickability
  of
  items
  shown
  in
  P1§𝐹P2<𝐹∗C1,P2
  ⟹overestimate
  clickability
  of
  items
  shown
  in
  P2Combined
  with
  the
  greedy
  placement
  algorithm.§After
  retraining,
  whatever
  we
  placed
  in
  position
  P1
  looks
  worse
  than
  it
  really
  is.Therefore
  we
  might
  move
  it
  down
  to
  position
  P2.§After
  retraining,
  whatever
  we
  placed
  in
  position
  P2
  looks
  better
  than
  it
  really
  is.Therefore
  we
  might
  move
  it
  up
  position
  P1.
How
  bad
  can
  things
  go
  in
  context
  C2
  ?True
  position
  effect
  less
  steep
  than
  estimated.§𝐹P1>𝐹∗C1,P1
  ⟹underestimate
  clickability
  of
  items
  shown
  in
  P1§𝐹P2<𝐹∗C1,P2
  ⟹overestimate
  clickability
  of
  items
  shown
  in
  P2Combined
  with
  the
  greedy
  placement
  algorithm.§After
  retraining,
  whatever
  we
  placed
  in
  position
  P1
  looks
  worse
  than
  it
  really
  is.Therefore
  we
  might
  move
  it
  down
  to
  position
  P2.§After
  retraining,
  whatever
  we
  placed
  in
  position
  P2
  looks
  better
  than
  it
  really
  is.Therefore
  we
  might
  move
  it
  up
  position
  P1.
Feedback
  loops
  in
  machine
  learning
§“Information
  (signal)
  feedback
  loops
  are
  everywhere.”“They
  are
  central
  to
  adaptation
  and
  learning…”§See
  (Bottouet
  al.,
  JMLR
  2013)
  for
  a
  possible
  treatment
  of
  causal
  loops.
Norbert
  Wiener ,
  Cybernetics,
  1948

Example
  2Team
   work
Red
  TeamOwns
  the
  code
  that
  selectscontent
  to
  recommend.
Blue
  TeamOwns
  the
  code
  that
  selects
  fonts
  and
  colors
  for
  all
  page
  components.
Red
  teamThe
  red
  team
  knows
  the
  bandit
  literature§There
  are
  lots
  of
  potential
  recommendations
  to
  select
  from.§One
  can
  only
  model
  the
  click
  probabilities
  of
  the
  recommendations
  that
  have
  been
  shown
  often
  enough
  in
  each
  context.§Therefore
  the
  red
  team
  code
  performs
  ε-‐greedy
  exploration,
  showing
  a
  small
  proportion
  of
  random
  recommendations.§Although
  exploration
  has
  a
  cost,
  such
  the
  system
  cannot
  discover
  the
  most
  relevant
  recommendations
  without
  it.
Blue
  teamThe
  blue
  team
  also
  knows
  machine
  learning.§The
  job
  of
  the
  blue
  team
  is
  to
  emphasize
  things
  that
  the
  user
  will
  like.§The
  blue
  team
  code
  runs
  a
  click
  probability
  model
  to
  spot
  what
  users
  like.§The
  click
  probability
  model
  easily
  discovers
  that
  some
  of
  the
  recommendations
  generated
  by
  the
  red
  team
  are
  not
  very
  good.§Therefore
  these
  recommendations
  will
  be
  shown
  with
  a
  smaller
  font.
Blue
  teamThe
  blue
  team
  also
  knows
  machine
  learning.§The
  job
  of
  the
  blue
  team
  is
  to
  emphasize
  things
  that
  the
  user
  will
  like.§The
  blue
  team
  code
  runs
  a
  click
  probability
  model
  to
  spot
  what
  users
  like.§The
  click
  probability
  model
  easily
  discovers
  that
  some
  of
  the
  recommendations
  generated
  by
  the
  red
  team
  are
  not
  very
  good.§Therefore
  these
  recommendations
  will
  be
  shown
  with
  a
  smaller
  font.
Red
  teamThe
  red
  team
  is
  very
  content!§Analysis
  of
  the
  exploration
  traffic
  shows
  that
  there
  isn’t
  much
  to
  improve.àthey
  can
  safely
  reduce
  the
  exploration
  level!§Whenever
  they
  try
  new
  features
  in
  the
  click
  model,
  randomized
  testing
  shows
  a
  reduction
  in
  performanceàit
  is
  amazing,
  but
  it
  seems
  they
  got
  their
  model
  right
  the
  first
  time!
Red
  teamThe
  red
  team
  is
  very
  content!§Analysis
  of
  the
  exploration
  traffic
  shows
  that
  there
  isn’t
  much
  to
  improve.àthey
  can
  safely
  reduce
  the
  exploration
  level!§Whenever
  they
  try
  new
  features
  in
  the
  click
  model,
  randomized
  testing
  shows
  a
  reduction
  in
  performanceàit
  is
  amazing,
  but
  it
  seems
  they
  got
  their
  model
  right
  the
  first
  time!
Paying
  attention“A
  manager
  of
  the
  two
  teams
  should
  have
  paid
  more
  attention.”“The
  software
  architect
  should
  have
  paid
  more
  attention.”
Micro-‐management
  does
  not
  work
  because
  it
  does
  not
  scale…
beyond
  two
  teams

Conclusion
   of
  part
  1Machine
   learning
   in
  large
  software
   systems§Use
  trained
  models
  as
  software
  modules.àproblematic
  because
  trained
  models
  offer
  weak
  contracts.§Use
  learning
  algorithms
  as
  software
  modules.àproblematic
  because
  the
  output
  of
  the
  learning
  algorithm
  depends
  on
  the
  training
  data
  which
  itself
  depends
  on
  every
  other
  module.Working
  around
  these
  difficulties
  could
  save
  billions…
Our
  experimental
  paradigm
  isreaching
  its
  limits
Minsky
  and
  Papert1968,
  F.A.Q.
Are
  we
  dealing
  with
  an
  exact
  science
  (like
  mathematics)
  or
  an
  empirical
  science
  (like
  physics)?

Essential
  epistemologyExact
  sciencesExperimentalsciencesEngineeringDeals
  withAxioms
  &TheoremsFacts
  &TheoriesArtifactsTruthisForeverTemporaryIt
  works.ExamplesMathematicsC.S.theoryPhysicsBiologyLots…
Essential
  epistemology–Exact
  sciencesExact
  sciences§Deal
  with
  axiomsand
  theorems.
  §Axioms
  are
  neither
  true
  or
  false.Theorems
  are
  true
  once
  proven.They
  remain
  true
  forever.§ExamplesoMathematicsoTheoretical
  C.S.
  (e.g.,
  algorithms,
  type
  theory,
  …)oTheoretical
  Physics
  (e.g.,
  string
  theory)Exact&sciencesExperimentalsciencesEngineeringDeals&withAxioms&&TheoremsFacts&&TheoriesArtifactsTruthisForeverTemporaryIt&works.ExamplesMathematicsC.S.theoryPhysicsBiologyC.S.
Essential
  epistemology–Experimental
  sciencesExperimental
  sciences§Deal
  with
  factsand
  theories.§Facts
  are
  true
  when
  they
  can
  be
  replicated.
  Theories
  are
  true
  when
  they
  predict
  the
  facts.
  New
  facts
  can
  invalidate
  theories
  that
  were
  previously
  considered
  true
  (K.
  Popper)§ExamplesoPhysics,
  Biology,…oSocial
  sciences,
  Experimental
  psychology,…
  Exact&sciencesExperimentalsciencesEngineeringDeals&withAxioms&&TheoremsFacts&&TheoriesArtifactsTruthisForeverTemporaryIt&works.ExamplesMathematicsC.S.theoryPhysicsBiologyC.S.The
  word
  “true”has
  a
  different
  meaning.
Essential
  epistemology–EngineeringEngineering§Deals
  with
  artifacts.§A
  claim
  is
  true
  when
  the
  artifact
  works,(the
  artifact
  fulfils
  a
  contract.)§ExamplesoMechanical
  engineering,
  …oNuclear
  engineering,
  …oComputer
  science.Exact&sciencesExperimentalsciencesEngineeringDeals&withAxioms&&TheoremsFacts&&TheoriesArtifactsTruthisForeverTemporaryIt&works.ExamplesMathematicsC.S.theoryPhysicsBiologyC.S.
Machine
  learning?What
  is
  machine
  learning?§An
  engineering
  discipline?Yes:
  applications
  abound.§An
  exact
  science?Yes:
  statistical/algorithmic
  learning
  theory.§An
  experimental
  science?Yes:
  papers
  often
  report
  on
  experiments.This
  is
  all
  good
Machine
  learning?What
  is
  machine
  learning?§An
  engineering
  discipline?Yes:
  applications
  abound.§An
  exact
  science?Yes:
  statistical/algorithmic
  learning
  theory.§An
  experimental
  science?Yes:
  most
  papers
  include
  experiments.This
  is
  all
  good
  …
  as
  long
  as
  we
  don’t
  mix
  the
  genres.§“My
  software
  works
  really
  well,
  therefore
  my
  theory
  is
  true.”§“This
  is
  NP-‐complete,
  therefore
  this
  experiment
  is
  wrong.”

ML
  as
  an
  experimental
  science§Some
  problems
  of
  interest
  do
  not
  have
  a
  clear
  specification.They
  are
  not
  amenable
  to
  provable
  algorithms.§Machine
  learning
  replaces
  the
  missing
  specification
  by
  lots
  of
  data.§Important
  aspects
  of
  the
  data
  cannot
  be
  described
  by
  a
  compact
  mathematical
  statement
  (otherwise
  we
  have
  a
  specification!)§Therefore
  experimentation
  is
  necessary!

ML
  as
  an
  experimental
  scienceProgress
  during
  the
  last
  few
  decades
  has
  been
  driven
  bya
  single
  experimental
  paradigm!

ML
  as
  an
  experimental
  scienceProgress
  during
  the
  last
  few
  decades
  has
  been
  driven
  bya
  single
  experimental
  paradigm!This
  is
  unusual
  for
  an
  experimental
  science!Physicists,
  biologists,
  experimental
  psychologists,
  ...,
  must
  work
  a
  lot
  harder
  to
  design
  experiments
  and
  interpret
  their
  results.

Where
  does
  the
  data
  come
  from?How
  to
  improve
  the
  performance
  of
  a
  ML
  application?§Get
  better
  algorithm?
  +§Get
  better
  features?+++§Get
  better
  data?++++++Data
  collection
  is
  hard
  work§The
  data
  distribution
  must
  match
  the
  operational
  conditions.§Rely
  on
  manual
  data
  curationto
  avoid
  dataset
  bias.
The
  impact
  of
  dataset
  biasTraining/testing
  on
  biased
  datasets
  gives
  unrealistic
  results.§E.g.
  :
  Torralbaand
  Efros,
  Unbiased
   look
  at
  dataset
  bias,
  CVPR
  2011.

Increased
  ambitions:
  Big
  DataToo
  much
  data
  for
  manual
  curation§Big
  data
  exists
  because
  data
  collection
  is
  automated.§Nobody
  checks
  that
  the
  data
  distribution
  matches
  the
  operational
  conditions
  of
  the
  system.àAll
  large
  scale
  datasets
  are
  biased.Training/testing
  on
  a
  big
  dataset
  can
  give
  unrealistic
  results.
Increased
  ambitions:
  A.I.Statistical
  machine
  learning§Building
  a
  classifier
  that
  works
  well
  under
  a
  certain
  distribution.Machine
  learning
  for
  artificial
  intelligence§Building
  a
  classifier
  that
  recognizes
  a
  “concept”.§Concepts
  exist
  independently
  of
  the
  pattern
  distribution.§Training
  data
  will
  never
  cover
  the
  whole
  range
  of
  possible
  inputs.In
  fact,
  a
  system
  that
  recognizes
  a
  “concept”
  fulfils
  a
  stronger
  contractthan
  a
  classifier
  that
  works
  well
  under
  a
  certain
  distribution.
Concepts
  ≠
  StatisticsExample:
  detection
  of
  action
  “giving
  a
  phone
  call”
(Oquabet
  al.,
  CVPR
  2014)
Concepts
  ≠
  StatisticsComputer
  vision
  is
  not
  a
  statistical
  problem
Car
  examples
  in
  ImageNetIs
  this
  less
  of
  a
  carbecause
  the
  context
  is
  wrong?
Concepts
  ≠
  StatisticsConvolutional
  networks
  can
  be
  fooled.
(Nguyen
  et
  al,
  CVPR
  2015)

Concepts
  ≠
  StatisticsCaption
  generation§Eight
  independent
  papers
  in
  2014.According
  to
  the
  BLEU
  score,some
  of
  these
  systems
  perform
  better
  than
  humans…§They
  met
  in
  Berkeley
  in
  January
  2015.§Evaluation
  is
  very
  difficult.§Example
  of
  difficulty
  :
  caption
  generation
  vs
  retrievalFrom
  a
  talk
  of
  (
  Zitnick14
  )
Conclusion
  of
  part
  2Training/testing
   only
  goes
  so
  far…Training/testing
  is
  an
  unusually
  convenient
  experimental
  paradigm…àit
  makes
  experimentation
  easier
  than
  in
  other
  sciencesàit
  has
  played
  a
  role
  in
  the
  fast
  progress
  of
  ML.…but
  may
  not
  fulfil
  our
  increased
  ambition
  (big
  data,
  AI.)àthe
  end
  of
  an
  anomaly?Working
  around
  this
  could
  save
  decades.
Elements
  of
  a
  solution
1.
  Process
  challengesChallenges
  of
  a
  new
  kind§We
  are
  used
  to
  face
  technical
  challenges.§These
  are
  “process
  challenges.”
1.
  Process
  challengesEngineering
  process§There
  is
  a
  rich
  literature
  about
  the
  software
  engineering
  process.§What
  should
  be
  the
  machine
  learning
  engineering
  process?
1.
  Process
  challengesScientific
  process§We
  cannot
  solelyrely
  on
  training/testing
  experiments.§We
  can
  understand
  more
  things
  with
  ad
  hoc
  experiments.Example
  in
  question
  answering
  :
  the
  “BABI”
  tasks

  (Bordeset
  al.,
  2015)oquestions
  that
  require
  combining
  one/two/more
  factoids.oquestions
  that
  require
  spatial/temporal
  reasoning.oetc.§What
  are
  the
  best
  practices
  in
  other
  experimental
  sciences?
1.
  Process
  challengesA
  constructive
  proposition§Machine
  learning
  papers
  rarely
  show
  the
  limits
  of
  their
  approach.-‐For
  fear
  that
  reviewers
  will
  use
  them
  against
  their
  work.§Reviewers
  should
  instead
  demand
  that
  authors
  discuss
  these
  limits.-‐Additional
  credits
  if
  they
  illustrate
  these
  limits
  with
  experiments.-‐This
  would
  make
  the
  papers
  more
  informative…-‐and
  would
  therefore
  facilitate
  scientific
  progress.
2.
  ML
  with
  stronger
  contractsHow
  to
  better
  resist
  distribution
  shifts?§Optimize
  measure
  of
  coverageinstead
  of
  average
  accuracy.-‐Selective
  classification
  (El
  Yaniv’s,
  …)-‐KWIK
  (Li
  &
  Littman)-‐Conformal
  learning
  (Vovk,
  …)§Very
  interesting
  properties
  when
  the
  data
  is
  Zipfdistributed…
2.
  ML
  with
  stronger
  contractsZipfdistributed
  patterns
Queries
  sorted
  in
  frequency
  orderway
  enough
  data
  to
  trainnot
  enough
  data
  to
  train
2.
  ML
  with
  stronger
  contractsDoubling
  the
  size
  of
  a
  Zipfdistributed
  dataset.
2xDiminishing
  returnsfor
  average
  accuracy
  improvements.No
  diminishing
  returnson
  number
  of
  queries
  for
  which
  we
  can
  learn
  correct
  answers.
3
  How
  to
  package
  the
  work
  of
  others§Digital
  computers
  :
  “software”§Learning
  machines
  :
  ◦Trained
  module
  as
  a
  software
  component?Trained
  components
  only
  offer
  “weak
  contracts”.◦Training
  software?If
  you
  can
  get
  the
  training
  data
  and
  replicate
  the
  rig.Recent
  example
  :
  AlexNet.◦Task
  specific
  trained
  featuresNearly
  as
  good.

Example:
  face
  recognitionInteresting
  task:

  “Recognizing
  the
  faces
  of
  106persons.”§How
  many
  labeled
  images
  per
  person
  can
  we
  obtain?Auxiliary
  task:

  “Do
  these
  faces
  belong
  to
  the
  same
  person?”§Two
  faces
  in
  the
  same
  picture
  usually
  are
  different
  persons.§Two
  faces
  in
  successive
  frames
  are
  often
  the
  same
  person.(Matt
  Miller,
   NEC,
  2006)
Example:
  NLP
  tagging
(Collobert,
   Weston,
  et
  al.,
  2008-‐2011)
Example:
  object
  recognition
Dogs
  in
  ImageNet(~106dogs)Dogs
  in
  Pascal
  VOC(only
  ~104imgages)
Example:
  object
  recognition
Several
  CVPR
  2014
  papers
  –figure
   from
   (Oquabet
  al.,
  2014)
Conclusion
Two
  process
  challenges§Machine
  learning
  disrupts
  software
  engineering.àChallenging
  our
  impact
  on
  real
  world
  applications.§Our
  experimental
  paradigm
  is
  reaching
  its
  limits.àChallenging
  the
  speed
  of
  our
  scientific
  progress.


ALBA: An explicit proposal for aligned AI

In this post I propose an explicit procedure for aligning powerful learning systems with their users’ interests. The goal is to introduce minimal overhead or additional complexity, yet to [safely scale up to extremely powerful systems](https://medium.com/ai-control/scalable-ai-control-7db2436feee7).

ALBA (algorithm learning by bootstrapped approval-maximization) is a method for providing rewards to reinforcement learners. It uses RL to train a reward function, rather than directly defining a simple reward function.

ALBA alternates two steps:

1. **Amplification**. We start with a fast and aligned agent. We allow this fast agent to think for a long time, and give it access to a large external memory and powerful computational aids. The resulting system is more powerful and much slower than the original agent.
2. **Distillation**. We use this strong, slow agent to define a reward function, and use this reward function to train another fast agent. (This is conceptually similar to [knowledge distillation](http://arxiv.org/pdf/1503.02531v1.pdf).)

Hopefully, each iteration increases the strength of the learner while maintaining alignment.

We start off the process with a weak learner obtained by applying step [2] with a human in place of the strong learner.

The rest of this post:

* Describes the necessary building blocks and offers candidate implementations.
* Makes this iterative process precise.
* Explains why we might expect ALBA to be aligned and efficient.
* Identifies key problems with this scheme and open questions about it.

*(This research was supported as part of the* [*Future of Life Institute*](http://futureoflife.org) *FLI-RFP-AI1 program, grant #2015–143898.)*

The problem
-----------

ALBA is designed to cope with the following problem.

Existing ML techniques are much better at optimizing objectives that we can measure. For example, if we want to use gradient descent to find a good policy, we need to be able to measure how good a policy is.

In general, we can’t measure what we really care about directly. Instead, we use rough proxies to assess how well a learned policy gets us what we really want. For example, we may search for a traffic-routing algorithm that minimizes total time spent on the road, or we may search for a user interface that maximizes reported user satisfaction.

Because these proxies don’t capture everything we care about, we aren’t necessarily happy if they are creatively or ruthlessly maximized. Defining an appropriate objective becomes an arms race between the objective-setter and the objective-optimizer. Today the objective-setters are humans and the objective-optimizers are weak AI systems, so we don’t have serious trouble. But that situation may change as AI improves.

ALBA addresses this problem by learning a sequence of increasingly complex objective functions, rather than working with a fixed objective specified by the user.

The building blocks
===================

ALBA relies on two building blocks:

* A “semi-supervised” reinforcement learning algorithm (which does all of the heavy lifting).
* A bootstrapping scheme for turning a weak agent into a strong agent.

Reinforcement learning
----------------------

We will apply reinforcement learners to a single-step problem:

* Given *x* ∈ {0, 1}ⁿ
* Output *y* ∈ {0, 1}ⁿ
* Receive a reward *r* ∈ [0, 1]
* Observe *z* ∈ {0, 1}ⁿ

The goal of the learner is to maximize its total score. Our agent should treat different decisions as completely independent, ignoring the non-information effects of each decision on future episodes (as in the usual formulation of bandit problems). This makes the learning problem significantly easier, and it is also important for alignment. The independence assumption can be formalized by comparing two different learners in the context of a fixed sequence of episodes (even if in fact the learner’s behavior in each episode influences the content of future episodes).

For concreteness, we can imagine training a deep neural network *f*(*x*) = *y* by gradient descent, and learning a stochastic policy so that we can estimate the gradient of the expected reward as 𝔼[*r*(*a*)∂*p*(*a*)/*p*(*a*)]. (Realistically we will need to do a lot of work to cope with the challenging setup.)

We add one complication to the traditional reinforcement learning problem. Rather than assuming that rewards are always available, we mostly work with “unlabelled” data: our algorithm sees many inputs *x*, but rarely sees the actual reward. We allow the algorithm to request to see the reward, but only on a small fraction of instances. When it doesn’t see the real reward, it can still observe the consequences of choosing a particular *y*, and use these consequences as [side information](https://medium.com/ai-control/semi-supervised-learning-from-side-information-483d5db474a2#.4qmbze8ew) to help build a model that can accelerate learning.

For concreteness sake, we can assume that we are using a traditional reinforcement learning algorithm which ignores the unlabelled data, requests reward information for a large number of initial examples, and thereafter requests reward information with probability ε. We can accelerate learning with more clever active and semi-supervised learning. Hopefully we can accelerate it a lot.

Finally, we assume that our reinforcement learner is “throttled” — we can control the capability of the system by some parameter *c* such as model capacity, training time, or computational resources. When *c* is small the system should be much weaker than its users, when *c* is large the system should be state-of-the-art, and performance should scale up continuously with *c*.

We could also apply ALBA in [this toy model of the AI control problem](https://medium.com/ai-control/modeling-ai-control-with-humans-6d285c1a114d#.g4coi7etr), with a human playing the role of the reinforcement learner.

Bootstrapping protocol
----------------------

Our other key ingredient is a bootstrapping protocol. This protocol takes as input an agent *A* which is aligned, and produces a slightly more powerful agent *B* which remains aligned. We think of our agents as simple functions from {0, 1}ⁿ → {0, 1}ⁿ. Critically, the agent *B* may take *much* longer to run than the agent *A*.

Intuitively, we’d like to just “give *A* more time to think.” We need a non-trivial bootstrapping protocol because we do not trust *A* to automatically generalize from a training environment where it has little time to think, to a test environment where we give it a long time to think. (And we can’t train *A* in cases where it has a long time to think, because we don’t have any way to produce a suitable training signal.)

Roughly speaking, our procedure works as follows:

1. We set up a simple computational environment.
2. We provide A with a summary of the current environment, and it proposes an operation to perform next.
3. We execute that operation, updating the computational environment, and return to step 2.
4. Eventually, A decides to return a value.

We describe an explicit computational environment, annotated functional programming, [here](https://medium.com/ai-control/approval-directed-algorithm-learning-bf1f8fad42cd#.8v8zvc6jv). B is the computation implemented by annotated functional programming, when A acts as the controller.

We could use a different framework for external memory / computational aids. Our only unusual requirement is that the framework be suitable for approval-directed training. That is, it must be easier to determine the correctness/suitability of a particular action than to actually run a whole computation. In particular, the correctness/suitability of an action cannot depend on the rest of the controller’s policy, as it would if we were doing end-to-end training. Annotated functional programming is designed to meet this requirement while remaining suitably expressive and as simple as possible.

Recurrent models
----------------

So far I have described an agent A who makes a single decision, and an agent B who evaluates a single decision. We could extend this idea to sequences of actions, allowing A to be e.g. a recurrent neural network, and having B evaluate a whole sequence of actions.

In this case, A still only cares about the payoffs within a single “episode,” which now consists of a sequence of decisions, and ignores effects on future episodes (as in the usual RL setup). But within each episode, A may behave strategically.

From our perspective, the key difference between the external memory and the recurrent controller’s internal state is that the controller’s state is optimized to achieve high reward in future steps, while operations on the external memory are optimized to receive a high score.

Note that B can maintain only external state between each query posed to it, for fundamental technical reasons (the cost of training would increase exponentially if we try to preserve internal state, or else we would have to rely on transfer learning from short computations to long computations). That is, when evaluating A’s first action, B can write some notes to itself. It can look at these notes when evaluating A’s second action. But the different questions correspond to separate episodes, with no internal state shared between them.

Using a recurrent model is especially critical in the final distillation step, since training a recurrent model (rather than relying on external memory) might be necessary to achieve state-of-the-art performance.

I’m going to continue to talk about the single-action case. But it is straightforward to translate the discussion in this post to the sequential case.

Defining ALBA
=============

We define two sequences of agents Aⁿ and Bⁿ, as follows. B⁰ is a human operator. ALBA’s external policy will be defined by the last agent Aᴺ; our hope is that it is both aligned and efficient.

For 0 < *n* < *N*, the agent Aⁿ is a semi-supervised reinforcement learner which takes as input the current visible state of a computational environment, and outputs a computational action. We fix a sequence of constants *c*ⁿ, and we throttle Aⁿ with the parameter *c*ⁿ.

The agent Bⁿ takes as input a (state, action, auxiliary data) triple, and outputs a real number in [0, 1]. This number will be used to define the reward function for Aⁿ⁺¹.

The agent Aᴺ acts directly in the world, in any context that a typical reinforcement learning system could act in the world, with sparse rewards defined by Bᴺ⁻¹. Note for example that Aᴺ may operate arbitrary computational machinery — we can apply it wherever we would have applied a reinforcement learning subsystem.

Distillation [approval-maximization]
------------------------------------

Agent Aⁿ is given a state *s*, picks an action *a,* and receives reward Bⁿ⁻¹(*s, a*, auxiliary data). Recall that this reward is only actually computed when Aⁿ asks for feedback — most of the time we don’t actually run Bⁿ⁻¹.

As described [here](https://medium.com/ai-control/adequate-oversight-25fadf1edce9#.29k7nnici), we would like to augment this oversight by allowing Bⁿ⁻¹ to view the internal state of Aⁿ. I don’t have any concrete implementation of this idea.

For now, I’ll just assume that the raw computational history (e.g. the activations of each layer) of Aⁿ is given to Bⁿ⁻¹ as the auxiliary input*.* More sophisticated algorithms may replace *C* by a more effective approach to transparency.

(One reason this procedure is unsatisfactory is that some training procedures will result in *C* being optimized to make *a* look like a good action, or to obscure information that would allow the overseer to realize that *a* is a bad action. A working proposal will probably have to use an alternative to *C* that is actually optimized for being informative. Nevertheless, including *C* is better than nothing.)

Amplification [algorithm learning]
----------------------------------

Agent Bⁿ is implemented by using Aⁿ as a controller for [annotated functional programming](https://medium.com/ai-control/approval-directed-algorithm-learning-bf1f8fad42cd#.8v8zvc6jv). To evaluate a triple (*s*, *a, C*), we set up an initial computational state, with the goal/question:

* {Evaluate the quality of taking action {*a*}, after performing computation {*C*}, in a state summarized as {*s*}. Return the result as a real in [0, 1]},

where *C* is a summary of the computation carried out by Aⁿ, and *a*, *s*, *C* are stored as terms in external memory.

We then consult Aⁿ in order to produce an instruction, execute that instruction, and repeat (as described in the section **Bootstrapping protocol**). The output of Bⁿ is the result eventually returned by this process.

The only free parameters are the constants *c*ⁿ, and the free parameters in the reinforcement learning algorithm and bootstrapping scheme.

Note that this bootstrapping process may involve consulting the user — we can easily extend our computational model to include actions to elicit user input. This is not technically needed for our analysis below. However, the ability to elicit user input significantly increases the meaningfulness of having an “aligned” AI system, for our definition of alignment.

Discussion of alignment
=======================

We’ll try to establish by induction that our systems are *aligned* with their users’ interest. I’ll postpone discussion of efficiency to the next section.

When I say that a system is aligned at capacity *c*, what I mean is that it serves the user’s interests as well as an “optimally motivated” system with capacity *c* and access to the same information. I apologize for the fuzzy concepts.

**Bootstrapping Lemma**: If Aⁿ is aligned at capacity *c*ⁿ, then Bⁿ = Bootstrap(Aⁿ) is aligned at capacity significantly above *c*ⁿ⁺¹.

**Oversight Lemma**: If Bⁿ is aligned at capacity significantly above *c*ⁿ⁺¹, then Aⁿ⁺¹ = ApprovalMaximization(Bⁿ) is aligned at capacity *c*ⁿ⁺¹.

Given these lemmas, and the starting condition that B⁰ = [a human] is aligned at capacity significantly above *c*¹, we can establish by induction that Aⁿ is aligned at capacity *c*ⁿ. This is exactly what we want.

Note that we can choose the sequences of constants *c*ⁿ so that they increase extremely slowly; this makes the bootstrapping lemma more likely, without affecting the oversight lemma.

Bootstrapping lemma
-------------------

This lemma feels intuitively very solid to me, though I don’t have any formal argument.

Intuitively, Bootstrap(A) will behave “like a copy of A that gets to think for longer.” For example, if I imagine myself in the position of A, it seems clear that Bootstrap(me) is significantly smarter than I am, to roughly the same degree that thinking longer makes me smarter.

If you don’t buy this, we can also make a slightly more conservative (though still informal) argument.

Bootstrap(A) can always just directly appeal to A. So as long as A is aligned at high enough capacity to implement this strategy, Bootstrap(A) will be aligned at the same capacity.

On top of that, Bootstrap(A) has access to many simple strategies that might let it outperform A. For example:

* It can use A to answer a large number of sub-questions and use the answers as features to help make a better decision.
* It can consult an ensemble of many slightly different copies of A, each using different strategies to think about the same problem.
* It can perform other expensive computations, and use them as features to help make a better decision.

In order for Bootstrap(A) to outperform A, we only need one of these techniques to help as much as significantly increasing in A’s abilities.

Oversight lemma
---------------

The oversight lemma is discussed in my recent post [adequate oversight](https://medium.com/ai-control/adequate-oversight-25fadf1edce9#.v5zmqco45).

The rough argument:

1. If B is aligned at capacity *c*’, then B’s estimates for the goodness of actions are the best possible for an agent at capacity *c*’, where “goodness” is defined by the user’s interpretation.
2. If A knows that B is aligned at capacity *c’*, and A is throttled by capacity *c* < *c’*, then A doesn’t expect to be able to beat B at estimating how good an action is. That is, the best estimate for goodness is equivalent to the best estimate for B’s estimate of goodness.
3. So at capacity *c*, optimizing B’s estimate of goodness is equivalent to optimizing actual goodness*.*
4. Under a reflective consistency assumption, taking actions chosen to maximize actualgoodness results in aligned behavior.

The key problem with this argument is that A can think thoughts that B hasn’t considered. So B is not really strictly better informed than A. And as a result, A may be able to identify predictable “errors” made by B.

The step that fails is the move from [2] to [3] — for any particular action, A will (probably) want to defer to B, but if A *chooses the action for which B’s estimate is highest*, this might systematically be an action for which A has noticed some considerations that B has overlooked.

There are two hopes for dealing with this problem:

* If B is much smarter than A, then maybe the resulting behavior is aligned anyway.
* Whatever approach we use for letting B inspect A’s state is sufficient to allow B to rediscover everything that A thought of. In the concrete implementation in this post, B is simply allowed to directly read the internal state of A. Given the opacity of current machine learning methods, it isn’t clear whether this would help at all.

Overall, I think that this aspect of the oversight lemma is the weakest step of the current analysis, and the most natural topic for future theoretical research.

Discussion of efficiency
========================

I think that the additional overhead of running ALBA can probably be made very small, compared to a direct application of reinforcement learning. My argument is very similar to the one given in [AI control on the cheap](https://medium.com/ai-control/ai-control-on-the-cheap-3425b148daf0#.inq6s63dw).

At runtime, the algorithm Aⁿ is just as fast as a direct application of a reinforcement learning system — the only thing that differs is how it was trained.

So the remaining question is: how expensive is it to train this system?

Semi-supervised RL
------------------

One key ingredient is what I’ve been calling semi-supervised RL. That is, we need to be able to learn using mostly cheap signals from the environment, with only very rare direct supervision. A very simple example would be learning to play Atari games from pixel-data, where rewards are only provided in a very small number games. Once the learner has formed some very simple generalizations — and especially that the reward is the same as the score written on the screen — then it can in principle learn almost as efficiently as a traditional reinforcement learner.

If we can solve this semi-supervised problem well, then we may be able to get by with a very small amount of human supervision, with almost all learning occurring just by taking actions and observing their effects. As long as training is dominated by the time required for the agent to learn from observations of nature, rather than understanding “what the user wants,” we are good.

Of course, semi-supervised RL is a hard problem. I’ve recently written about [one simple idea](https://medium.com/ai-control/semi-supervised-learning-from-side-information-483d5db474a2) for this problem, and we can apply a lot of traditional and ongoing work on active and semi-supervised learning, off-policy learning, and model-based RL. It’s a big open question that will probably remain open for a while, but I think it is reasonable to expect significant progress in this area as RL improves.

Moreover, adequate semi-supervised RL seems to be a necessary capability of powerful AI systems, since it plays an important instrumental role. So overall I don’t feel bad about having the efficiency of the scheme depend on this capability.

Bootstrapping for efficiency
----------------------------

Humans only need to directly oversee the simplest system A¹, which may be a relatively low-capacity model that is relatively quick to train.

The subsequent systems Aⁿ are then overseen by the bootstrapped predecessors Bⁿ⁻¹. This oversight process might involve consulting humans (as part of the computation being run by Bⁿ⁻¹), but it could potentially involve humans increasingly rarely as the learners become more sophisticated. Essentially, whatever protocol we might use to limit the training data required by an aligned agent, that protocol can also be learned by the agents Bⁿ⁻¹ to minimize their reliance on human input. (For these agents, deciding to elicit input is an action which is itself trained using feedback from weaker agents.)

Of course, the algorithm Bⁿ⁻¹ is also *significantly* more expensive than the algorithm Aⁿ, and so this kind of bootstrapping can never reduce the cost of supervision too far. The fraction of rounds where we can afford to get supervision is roughly the ratio of [cost of running RL algorithm]/[cost of acquiring supervision], and so this approach would simply increase that fraction from “very small” to “small.” It would still need to be combined with the kind of semi-supervised RL described in the previous section.

Major problems and open questions
=================================

Transparency and oversight
--------------------------

As discussed in [Adequate oversight](https://medium.com/ai-control/adequate-oversight-25fadf1edce9#.9khwfdla8), this oversight procedure may not work when the overseen agent can find arguments that the overseer might miss.

Ideally, we would overcome this problem by training the overseen system to “show its work.” I think this goal is especially plausible because the overseer is significantly more intelligent than the overseen system.

Or we may be able to find a totally different way to deal with the problem.

Or maybe this objection is a show-stopper. My main source of optimism is simply that it feels very tractable to have a strong agent oversee a weak one. If we could reduce the AI control problem to this strong-overseeing-weak subproblem, I would feel quite optimistic, and I wouldn’t want to give up on that angle until we had explored many more candidate solutions.

Robust learning and catastrophic failures
-----------------------------------------

Sometimes our learners will fail, for example when encountering an input that is different from anything they have seen so far. Such failures will eventually be integrated as training data and corrected, but we need to ensure that they don’t do serious damage in the meantime.

Solving this problem requires either:

* Building composite systems that are robust to a small number of serious failures by individual components. In this context, such robustness is a lot harder than it may seem. The key problem is that, although there won’t be too many difficult context changes, a context change may cause simultaneous failures in many components (even in apparently unrelated systems).
* Designing learning systems that don’t fail catastrophically on arbitrary inputs. For example, systems that “know that they don’t know” and so can back down gracefully rather than going ahead with a potentially-catastrophic decision.

I’ve discussed [synthesizing training data](https://medium.com/ai-control/synthesizing-training-data-f92a637dc1b4#.yh2cn66zp) as one approach to building robust systems. But this is a large research area with many promising approaches. Many of those approaches don’t look likely to scale to the particular kinds of failures I care most about, but there is still a lot to try.

Robust hierarchical active learning
-----------------------------------

ALBA is hierarchical in the following sense: the reward of Aⁿ depends on the behavior of Bⁿ⁻¹, which in turn depends on the behavior of Aⁿ⁻¹ in a whole bunch of subproblems. The reward of each Aⁿ⁻¹ depends on the behavior of Bⁿ⁻², which in turn depends on the behavior of Aⁿ⁻² in a whole bunch of subproblems…

Suppose that A¹ makes bad decisions on a very small fraction of possible inputs. This leads B¹ to make a bad decision on a larger, but still small fraction of inputs. In turn, this leads A² to make bad decisions on a small fraction of possible inputs. This leads B² to make a bad decision on a moderate fraction of inputs…

The result is that even if A¹ only makes bad decisions on a very small fraction of inputs, we can end up with very bad behavior for the overall system.

To avoid this problem, we’d like to make sure that our active learning process calls attention to the inputs that A¹ messes up on, even if those inputs are extremely sparse. To elicit such inputs, we need to identify the inputs on which A² messes up because of A¹’s anticipated mistake, run B¹ on those inputs, and then query A¹ on the problematic inputs that come up during that computation. And so on, up the hierarchical structure.

Even in the non-hierarchical case, I think that getting [adequate guarantees](https://medium.com/ai-control/active-learning-for-opaque-powerful-predictors-94724b3adf06#.p6gjkfqzq) may be difficult, and will have to [rely on novel sources of information not normally used in active learning](https://medium.com/@paulfchristiano). The hierarchical case considerably amplifies the problem, exponentially increasing the potential number of errors if we use a naive learning strategy.

Semi-supervised RL
------------------

As described in the section on efficiency above, ALBA will only be acceptably efficient if we can do statistically efficient reinforcement learning from unlabelled data. Otherwise the cost of oversight will be prohibitive.

Theoretical understanding
-------------------------

The arguments above rely on a lot of fuzzy concepts, especially “aligned at capacity *c*” but more generally the entire notions of capacity and alignment. These notions are especially important when reasoning about what kind of oversight is adequate.

I expect intuitive arguments involving these concepts to be meaningful but potentially error-prone.

I suspect that there is a clean theoretical framework in which these arguments could be made formally. Developing such a framework would make it much easier to analyze and construct protocols like ALBA.

Does bootstrapping really help?
-------------------------------

I’ve described [one approach to bootstrapping](https://medium.com/ai-control/approval-directed-algorithm-learning-bf1f8fad42cd#.a4e9j5z7g). ALBA requires *any* mechanism for turning a weak agent into a (much slower) strong agent without compromising alignment.

I feel like this can probably be done, but I don’t have strong reasons to be confident in my particular proposal. It would be great to understand these dynamics better and perhaps to have stronger candidates.

Malicious inputs
----------------

Some inputs may cause an agent to behave especially badly—e.g. imagine a buffer overflow in a perceptual subroutine which can corrupt the rest of the agent’s processing.

Within the implicit formalism of this post, the existence of malicious inputs bounds the capacity of an agent, since they correspond to questions which that agent will predictably answer incorrectly.

So the capacity-enhancing requirement of our bootstrapping scheme may be stronger than it at first appears. For example, if the agent B begins its computation by feeding the input to agent A, then a malicious input may corrupt agent A. But the bootstrapping protocol needs to introduce robustness to an increasing range of malicious inputs.

Instead, agent B should process the input without ever passing the whole input to A (or at least by carefully preprocessing the input before giving it to agent A). That is, A only answers questions about “what to do next,” only ever looking at small parts of the input or high-level conclusions about the input. I think that this is a better approach in general, but it does call into question the style of “more conservative” argument I made in the section **Bootstrapping lemma** above.

In general, I am not too concerned with the prospect of malicious inputs for now— I think that a scheme that works well in general will probably also handle malicious inputs, and so I am happy to focus on other issues. I think that malicious inputs are especially likely to be resolved by an adequate approach to **Transparency and oversight** above. But I do think malicious inputs illustrate some of the subtleties that have to be handled correctly, and they suggest that there are aspects of the problem we don’t yet understand very well.

Conclusion
==========

I’ve presented ALBA, a concrete proposal for aligning powerful learning systems with their users’ interests. It can be applied with a generic RL algorithm rather than requiring significant architectural changes, and introduces minimal overhead at runtime. Given enough progress in RL, it might be possible to implement ALBA with very little increase in training costs.

I don’t yet have any strong argument for ALBA’s safety, but I think we are much closer than for any other existing scheme. The main missing piece is whether a strong agent can successfully oversee its weaker successor during the distillation step, as discussed in the section **Transparency and oversight.** That looks a lot easier than the usual AI control problem.

Aside from this central issue, we’d need to resolve many other theoretical questions to actually get this to work. Many of these theoretical questions are very closely related to traditional problems in machine learning, such as robustness, active learning, algorithm learning, and learning from unlabelled data.

On the experimental side, the algorithm learning problem in ALBA may be prohibitively difficult. That said, I think that experimenting with ALBA may be worthwhile in the same spirit as other (similarly difficult) work on algorithm learning. Bootstrapping may be useful for facilitating training, even if we don’t care about scalability at all. And experimental work would shed light on the long-term feasibility of bootstrapping.


ALBA on GitHub

I’ve put up an implementation of [ALBA](https://medium.com/ai-control/alba-an-explicit-proposal-for-aligned-ai-17a55f60bbcf) on GitHub [here](https://github.com/paulfchristiano/alba).

The key method is ALBA(H, n), which hopefully implements an aligned AI. To actually use it, you would need to replace the stub TransparentHybrid with a competent algorithm for [transparent](https://medium.com/ai-control/the-informed-oversight-problem-1b51b4f66b35#.v97m8wh5q) [semi-supervised](https://medium.com/ai-control/semi-supervised-reinforcement-learning-cf7d5375197f#.acfezc2rq) [imitation+RL](https://medium.com/ai-control/imitation-rl-613d70146409#.e28d27gjp).

For now you can mess around with a version of ALBA that uses a very simple learning algorithm — whenever it encounter a novel situation, ask the overseer what to do and memorize their answer for future reference.

I don’t think that actually running the example is especially informative. I wrote the code in order to make the discussion of ALBA more precise, honest, and concrete, not because I thought that anyone would want to actually run it.

TODOs
-----

In reality this implementation of ALBA(H, n) definitely *wouldn’t* be an aligned AI; there are at least half a dozen obviously fatal problems. An immediate goal is to address the obvious problems: to write some code such that *if* we filled in the AI capabilities, then there would at least be *any hope* that the resulting system was aligned. I don’t know how hard this goal is; I think there is a plausible roadmap and it might be relatively easy, or it might take a very long time.

(Realistically, even if there was *any* hope that the system would work, we would still need to improve all of the ingredients further before there was *much* hope that it would work.)

If that works out then we can start to search for non-obvious problems, and argue about more philosophical questions like whether Meta(HCH(·)) correctly implements [capability amplification](https://medium.com/ai-control/policy-amplification-6a70cbee4f34#.8ovjab4mx).

Here is the list of TODOs and FIXMEs in alba.py:

* TODO: Build [powerful AI](https://medium.com/ai-control/imitation-rl-613d70146409#.633sk36pw)
* TODO: Implement [semi-supervised RL](https://medium.com/ai-control/semi-supervised-reinforcement-learning-cf7d5375197f)
* TODO: Implement [transparent RL](https://medium.com/ai-control/the-informed-oversight-problem-1b51b4f66b35)
* FIXME: Prevent [catastrophic failure](https://medium.com/ai-control/learning-with-catastrophes-59387b55cc30) on adversarial inputs. [Adversarial training?](https://medium.com/ai-control/red-teams-b5b6de33dc76)
* FIXME: Ensure RL agent cannot outsmart overseer. Gradually scale up capacity as a function of n?
* FIXME: Prevent failure probability from growing with each iteration. Amplify reliability as well as capability?
* FIXME: Allow Amplify(A) to learn from training data, so it can keep up with the RL agent it is overseeing
* FIXME: Scores in [0, 1] are arbitrary, use [comparisons between different actions instead](https://medium.com/ai-control/optimizing-with-comparisons-c02b8c0d7877)
* FIXME: Use [budgeted HCH](https://medium.com/ai-control/strong-hch-bedb0dc08d4e#.bb5xsy6xj) so that errors can’t result in hangs
* TODO: Figure out whether iterating A → Meta(HCH(A)) can really [get us to arbitrarily powerful agents](https://medium.com/ai-control/policy-amplification-6a70cbee4f34)

How would this be used?
=======================

ALBA is designed to be an essentially drop-in replacement for imitation learning or RL. Of course you would only need to use it when you didn’t have access to a suitable objective function.

A realistic AI system may use RL or imitation learning as components but will generally [have other stuff going on as well](https://medium.com/ai-control/not-just-learning-e3bfb5a1f96e#.hm5jhyh7m). That’s fine, you drop ALBA in for the learning part. For example, in AlphaGo you would drop in ALBA for the policy and value networks, but you’d still have to do MCTS.

Hopefully none of your other code is going to destroy the world, but that’s not our responsibility.

ALBA(H, n) returns an agent A that is aligned with H but much smarter. You can use this agent in the same way that you’d use an episodic RL agent, except that you don’t have to give it a reward. To start an episode you make a fresh copy of A; for example, each time a user visits google.com you might make a new copy of A and then interact with that agent in order to figure out how to process their queries, with calls like A.act(“the user just typed `where should I eat dinner?`, what should happen now?”). In principle you could have very long-running episodes (as with RL), but this would make training more complex.

Instead of specifying a reward directly we specify some overseer H. The system is designed to do what H wants, so picking H is very important. For example, if ALBA were being used in the UI for google.com, H might be implemented by professional trainers employed by Google. If Google wants to make a product that is useful to the users, then it tells the trainers that they should try to think about what the user wants and make decisions appropriately. But the trainers can also balance the user’s interests with other desiderata, like Google’s profit motive or legal constraints. (All of this is analogous to how Google generally balances the desire to be useful to the user against other desiderata.)

Before the first time that ALBA is invoked, it will perform some [pre-training](https://medium.com/ai-control/red-teams-b5b6de33dc76#.5gpktrnyj) to ensure that its behavior is [non-catastrophic](https://medium.com/ai-control/learning-with-catastrophes-59387b55cc30#.x8iwsx5q3), including meeting some minimal bar for quality. [This doesn’t happen in the current implementation; it’s the FIXME about avoiding catastrophes.] During operation it makes decisions immediately, but will continue [collecting data](https://medium.com/ai-control/counterfactual-human-in-the-loop-a7822e36f399#.rpxze4kq5) and training asynchronously in the background. [Training occurs inside the implementation of the AI capabilities in capabilities.py.] During training, H will be invoked constantly.

If H is overseeing a single instance of ALBA that is performing a narrow task, then we can just explain the definition of the task to H. If H is overseeing a range of different agents pursuing a variety of tasks, then we can encode the task definition in the agent’s observations. For example, if we want it to caption an image, rather than simply giving the agent the image and expecting a caption, we could also give the agent the natural language text “please describe what you see in this image,” and perhaps a more detailed description of what kind of explanation we want, and so on. This information will ultimately be used by an aligned overseer to decide how to implement a reward function. As a result, the agent will learn to respond differently to different task explanations. (If the agent is very weak, this might be done simply by memorizing a few different possible task descriptions — we don’t inherently require that the agent can learn to understand language, though the whole system becomes more flexible at that point.)

If we were using an RL agent we would implement a reward function in order to cause the agent to do what we want. We don’t provide a reward function for ALBA, but we can let the agent observe “rewards” to help it learn faster. This can be used as [helpful unlabelled data](https://medium.com/ai-control/semi-supervised-reinforcement-learning-cf7d5375197f), or it can be absorbed by the overseer to improve its understanding of the task. [This doesn’t happen in the current implementation; the FIXME about having the overseer learn from training data.]

Conclusion
==========

I think it’s useful to be able to point to a completely concrete specification of ALBA. I also think that a concrete implementation is a nice setting in which to start fixing the obviously fatal problems, and I’ll be especially excited if/when there is a concrete implementation that *isn’t-obviously-doomed-to-fail*. For now, I hope that having access to the code will help people dig into any details of the scheme that they found ambiguous or unclear in the informal descriptions.