Human-Compatible Artificial Intelligence.

Human-Compatible Artiﬁcial Intelligence
Stuart Russell
Computer Science Division, University of California, Berk eley
1

1
Human-Compatible Artiﬁcial
Intelligence
1.1 Introduction
Artiﬁcial intelligence (AI) has as its aim the creation of intelligent mach ines. An entity
is considered to be intelligent, roughly speaking, if it chooses actions that are expected
to achieve its objectives, given what it has perceived.1Applying this deﬁnition to
machines, one can deduce that AI aims to create machines that cho ose actions that
are expected to achieve their objectives, given what they have pe rceived.
Now, what are these objectives? To be sure, they are—–up to now , at least–
—objectives that we put into them; but, nonetheless, they are ob jectives that operate
exactly as if they were the machines’ own and about which they are c ompletely cer-
tain. We might call this the standard model of AI: build optimizing machines, plug in
the objectives, and oﬀ they go. This model prevails not just in AI bu t also in con-
trol theory (minimizing a cost function), operations research (ma ximizing a sum of
rewards), economics (maximizing individual utilities, GDP, quarterly p roﬁts, or social
welfare), and statistics (minimizing a loss function). The standard m odel is a pillar of
twentieth-century technology.
Unfortunately, this standard model is a mistake. It makes no sens e to design ma-
chines that are beneﬁcial to us onlyif we write down our objectives completely and
correctly. If the objective is wrong, we might be lucky and notice th e machine’s sur-
prisingly objectionable behavior and be able to switch it oﬀ in time. Or, if the machine
is more intelligent than us, the problem may be irreversible. The more in telligent the
machine, the worse the outcome for humans: the machine will have a greater ability
to alter the world in ways that are inconsistent with our true object ives and greater
skill in foreseeing and preventing any interference with its plans.
In 1960, after seeing Arthur Samuel’s checker-playing program lea rn to play check-
ers far better than its creator, Norbert Wiener (1960) gave a cle ar warning:
If we use, to achieve our purposes, a mechanical agency with w hose operation we cannot
eﬃciently interfere ...we had better be quite sure that the p urpose put into the machine is
1This deﬁnition can be elaborated and made more precise in var ious ways—particularly with
respect to whether the choosing and expecting occur within t he agent, within the agent’s designer,
or some combination of both. The latter certainly holds for h uman agents, viewing evolution as the
designer. The word “objective” here is also used informally , and does not refer just to end goals. For
most purposes, an adequately general formal deﬁnition of “o bjective” covers preferences over lotteries
over complete state sequences. Moreover, “state” here incl udes mental state as well as the world state
external to the entity.
2Human-Compatible AI
the purpose which we really desire.
Echoes of Wiener’s warning can be discerned in contemporary asser tions that “su-
perintelligent AI” may present an existential risk to humanity. (In t he context of the
standard model, “superintelligent” means having a superhuman cap acity to achieve
givenobjectives.)Concernshavebeenraisedbysuchobserversa sNickBostrom(2014),
Elon Musk (Kumparak, 2014), Bill Gates (2015),2and Stephen Hawking (Osborne,
2017). There is very little chance that as humans we can specify our objectives com-
pletelyandcorrectly,insuchawaythatthepursuitofthoseobjec tivesbymorecapable
machines is guaranteed to result in beneﬁcial outcomes for humans .
Themistakecomesfromtransferringaperfectlyreasonabledeﬁn itionofintelligence
from humans to machines. The deﬁnition is reasonable for humans be cause we are
entitled to pursue our own objectives—–indeed, whose would we pur sue, if not our
own? The deﬁnition of intelligence is unary, in the sense that it applies to an entity by
itself. Machines, on the other hand, are not entitled to pursue the ir own objectives.
A more sensible deﬁnition of AI would have machines pursuing ourobjectives.
Thus,wehaveabinarydeﬁnition: entityAchoosesactionsthatare expectedtoachieve
the objectives of entity B, given what entity A has perceived. In th e unlikely event
that we (entity B) can specify the objectives completely and corre ctly and insert them
into the machine (entity A), then we can recover the original, unary deﬁnition. If
not, then the machine will necessarily be uncertain as to our objectives, while being
obliged to pursue them on our behalf. This uncertainty—with the cou pling between
machines and humans that it entails—is crucial to building AI systems o f arbitrary
intelligence that are provably beneﬁcial to humans. We must, there fore, reconstruct
the foundations of AI along binary rather than unary lines.
1.2 Artiﬁcial Intelligence
The goal of AI research has been to understand the principles und erlying intelligent
behavior and to build those principles into machines that can then exh ibit such behav-
ior. In the 1960s and 1970s, the prevailing theoretical deﬁnition of intelligence was the
capacityfor logicalreasoning,including the ability to deriveplans ofa ctionguaranteed
to achieveaspeciﬁed goal.Apopularvariantwasthe problem-solving paradigm,which
requires ﬁnding a minimum-cost sequence of actions guaranteed to reach a goal state.
More recently, a consensus has emerged in AI around the idea of a r ational agent that
perceives and acts in order to maximize its expected utility. (In Mark ov decision pro-
cesses and reinforcement learning, utility is further decomposed in to a sum of rewards
accrued through the sequence of transitions in the environment s tate.) Subﬁelds such
as logical planning, robotics, and natural-language understanding are special cases of
the general paradigm. AI has incorporated probability theory to h andle uncertainty,
utility theory to deﬁne objectives, and statistical learning to allow m achines to adapt
to new circumstances. These developments have created strong connections to other
disciplines that build on similar concepts, including control theory, ec onomics, opera-
tions research, and statistics.
2Gates wrote, “I am in the camp that is concerned about superin telligence. ...I agree with Elon
Musk and some others on this and don’t understand why some peo ple are not concerned.”
Artiﬁcial Intelligence 3
Inboththelogical-planningandrational-agentviewsofAI,themach ine’sobjective—–
whether in the form of a goal, a utility function, or a reward function —–is speciﬁed
exogenously. In Wiener’s words, this is “the purpose put into the ma chine.” Indeed,
it has been one of the tenets of the ﬁeld that AI systems should be general-purpose –
—i.e., capable of accepting a purpose as input and then achieving it—–r ather than
special-purpose , with their goal implicit in their design. For example, a self-driving car
should accept a destination as input instead of having one ﬁxed dest ination. However,
some aspects of the car’s “driving purpose” are ﬁxed, such as tha t it shouldn’t hit
pedestrians. This is built directly into the car’s steering algorithms ra ther than being
explicit: no self-driving car in existence today “knows” that pedest rians prefer not to
be run over.
Putting a purpose into a machine that optimizes its behavior accordin g to clearly
deﬁned algorithms seems an admirable approach to ensuring that th e machine’s be-
havior furthers our own objectives. But, as Wiener warns, we nee d to put in the right
purpose. We might call this the King Midas problem: Midas got exactly w hat he asked
for—–namely, that everything he touched would turn to gold—–but , too late, he dis-
coveredthe drawbacksofdrinking liquid gold andeating solid gold.The technical term
for putting in the right purpose is value alignment . When it fails, we may inadvertently
imbue machines with objectives counter to our own. Tasked with ﬁnd ing a cure for
cancer as fast as possible, an AI system might elect to use the entir e human popula-
tion as guinea pigs for its experiments. Asked to de-acidify the ocea ns, it might use up
all the oxygen in the atmosphere as a side eﬀect. This is a common cha racteristic of
systems that optimize: variables not included in the objective may be set to extreme
values to help optimize that objective.
Unfortunately, neither AI nor other disciplines built around the opt imization of
objectives have much to say about how to identify the purposes “w e really desire.”
Instead, they assume that objectives are simply implanted into the machine. AI re-
search, in its present form, studies the ability to achieve objective s, not the design of
those objectives. In the 1980s the AI community abandoned the id ea that AI systems
could have deﬁnite knowledge of the state of the world or of the eﬀe cts of actions, and
they embraced uncertainty in these aspects of the problem state ment. It is not at all
clear why, for the most part, they failed to notice that there must also be uncertainty
in the objective. Although some AI problems such as puzzle solving ar e designed to
have well-deﬁned goals, many other problems that were considered at the time, such
as recommending medical treatments, have no precise objectives and ought to reﬂect
the fact that the relevant preferences (of patients, relatives, doctors, insurers, hospital
systems, taxpayers, etc.) are not known initially in each case.
Steve Omohundro (2008) has pointed to a further diﬃculty, obser ving that any
suﬃciently intelligent entity pursuing a ﬁxed, known objective will act to preserve
its own existence (or that of an equivalent successor entity with an identical objec-
tive). This tendency has nothing to do with a self-preservation inst inct or any other
biological notion; it’s just that an entity usually cannot achieve its ob jectives if it’s
dead. Accordingto Omohundro’sargument,a superintelligent mach inethat hasan oﬀ-
switch—–which some, including Alan Turing (1951)himself, haveseen a s our potential
salvation–—will take steps to disable the switch in some way. Thus we m ay face the
4Human-Compatible AI
prospect of superintelligent machines—–their actions by deﬁnition u npredictable and
their imperfectly speciﬁed objectives conﬂicting with our own—–who se motivation to
preserve their existence in order to achieve those objectives may be insuperable.
1.3 1001 Reasons to Pay No Attention
Objections have been raised to these arguments, primarily by rese archers within the
AI community. The objections reﬂect a natural defensive reactio n, coupled perhaps
with a lack of imagination about what a superintelligent machine could do . None hold
water on closer examination. Here are some of the more common one s:
•Don’t worry, we can just switch it oﬀ :3This is often the ﬁrst thing that pops
into a layperson’s head when considering risks from superintelligent A I—–as if a
superintelligent entity would never think of that. It’s rather like say ing that the
risk of losing to Deep Blue or AlphaGo is negligible–—all one has to do is mak e
the right moves.
•Human-level or superhuman AI is impossible :4This is an unusual claim for AI
researchers to make, given that, from Turing onward, they have been fending oﬀ
such claims from philosophers and mathematicians. The claim, which is b acked
by no evidence, appears to concede that if superintelligent AI were possible, it
would be a signiﬁcant risk. It’s as if a bus driver, with all of humanity as his
passengers, said, “Yes, I’m driving toward a cliﬀ–—in fact, I’m press ing the pedal
to the metal. But trust me, we’ll run out of gas before we get there .” The claim
also represents a foolhardy bet against human ingenuity. We’ve mad e such bets
before and lost. On September 11, 1933, renowned physicist Erne st Rutherford
stated, with utter conﬁdence, “Anyone who expects a source of power from the
transformation of these atoms is talking moonshine.” On September 12, 1933,Leo
Szilard invented the neutron-induced nuclear chain reaction. A few years later,
he demonstrated such a reaction in his laboratory at Columbia Univer sity. As he
recalled in a memoir: “We switched everything oﬀ and went home. That night,
there was very little doubt in my mind that the world was headed for gr ief.”
•It’s too soon to worry about it : The right time to worryabout a potentially serious
problem for humanity depends not just on when the problem will occu r but also
on how much time is needed to devise and implement a solution that avoid s the
risk. For example, if we were to detect a large asteroid predicted to collide with
the Earth in 2070, would we say, “It’s too soon to worry!”? And if we c onsider
the global catastrophic risks from climate change predicted to occ ur later in this
century, is it too soon to take action to prevent them? On the cont rary, it may
be too late. The relevant timescale for human-level AI is less predict able, but,
like nuclear ﬁssion, it might arrive considerably sooner than expecte d. Moreover,
the technological path to mitigate the risks is also arguably less clear . These two
3AI researcher Jeﬀ Hawkins, for example, writes, “Some intel ligent machines will be virtual, mean-
ing they will exist and act solely within computer networks. ...It is always possible to turn oﬀ a
computer network, even if painful.” https://www.recode.n et/2015/3/2/11559576/.
4The AI100 report (Stone et al., 2016) includes the following assertion: “Unlike in the mov ies,
there is no race of superhuman robots on the horizon or probab ly even possible.”
1001 Reasons to Pay No Attention 5
aspects in combination do not argue for complacency; instead, the y suggest the
need for hard thinking to occur soon. Wiener (1960) ampliﬁes this po int, writing,
“The individual scientist must work as a part of a process whose time scale is
so long that he himself can only contemplate a very limited sector of it. ...Even
when the individual believes that science contributes to the human e nds which
he has at heart, his belief needs a continual scanning and re-evalua tion which is
only partly possible. For the individual scientist, even the partial ap praisal of this
liaison between the man and the process requires an imaginative forw ard glance
at history which is diﬃcult, exacting, and only limitedly achievable. And if we
adhere simply to the creed of the scientist, that an incomplete know ledge of the
world and of ourselves is better than no knowledge, we can still by no means
always justify the naive assumption that the faster we rush ahead to employ the
new powers for action which are opened up to us, the better it will be . We must
always exert the full strength of our imagination to examine where t he full use of
our new modalities may lead us.”
One variation on the “too soon to worry about it” argument is Andre w Ng’s
statement that it’s “like worrying about overpopulation on Mars.” Th is appeals
to a convenient analogy: Not only is the risk easily managed and far in t he future
but also it’s extremely unlikely that we’d even try to move billions of huma ns
to Mars in the ﬁrst place. The analogy is a false one, however. We’re a lready
devoting huge scientiﬁc and technical resources to creating ever more capable AI
systems. A more apt analogy would be a plan to move the human race t o Mars
with no consideration for what we might breathe, drink, or eat once we arrived.
•It’s a real issue but we cannot solve it until we have superint elligence: One would
not propose developing nuclear reactors and thendeveloping methods to contain
the reaction safely. Indeed, safety should guide how we think abou t reactor de-
sign. It’s worth noting that Szilard almost immediately invented and pa tented a
feedback control system for maintaining a nuclear reaction at the subcritical level
for power generation, despite having absolutely no idea of which elem ents and
reactions could sustain the ﬁssion chain.
By the same token, had racial and gender bias been anticipated as a n issue with
statistical learning systems in the 1950s, when linear regression be gan to be used
for all kinds of applications, the analytical approaches that have b een developed
in recent years could easily have been developed then, and would app ly equally
well to today’s deep learning systems.
Inotherwords,wecanmakeprogressonthebasisofgeneralpro pertiesofsystems–
e.g, systems designed within the standard model–without necessar ily knowing
the details. Moreover, the problem of objective misspeciﬁcation ap plies to all AI
systems developed within the standard model, not just superintellig ent ones.
•Human-level AI isn’t really imminent, in any case : The AI100report,forexample,
assuresus,“Contrarytothe morefantasticpredictionsforAIin the popularpress,
the Study Panel found no cause for concern that AI is an imminent t hreat to
humankind.” This argument simply misstates the reasons for concer n, which are
notpredicatedonimminence. Inhis2014book, Superintelligence: Paths, Dangers,
Strategies , Nick Bostrom, for one, writes, “It is no part of the argument in th is
6Human-Compatible AI
book that we are on the threshold of a big breakthrough in artiﬁcial intelligence,
or that we can predict with any precision when such a development mig ht occur.”
Bostrom’s estimate that superintelligent AI might arrive within this ce ntury is
roughly consistent with my own, and both are considerably more con servative
than those of the typical AI researcher.
•Any machine intelligent enough to cause trouble will be inte lligent enough to have
appropriate and altruistic objectives :5Thisargumentis related toHume’s is-ought
problem and G. E. Moore’s naturalistic fallacy, suggesting that some how the ma-
chine, as a result of its intelligence, will simply perceive what is right give n its
experience of the world. This is implausible; for example, one cannot p erceive,
in the design of a chessboard and chess pieces, the goal of checkm ate; the same
chessboard and pieces can be used for suicide chess, or indeed man y other games
still to be invented. Put another way: Where Bostrom imagines huma ns driven
extinct by a putative robot that turns the planet into a sea of pape rclips, we hu-
mans see this outcome as tragic, whereas the iron-eating bacteriu mThiobacillus
ferrooxidans is thrilled. Who’s to say the bacterium is wrong? The fact that a
machine has been given a ﬁxed objective by humans doesn’t mean tha t it will
automatically take on board as additional objectives other things t hat are impor-
tant to humans. Maximizing the objective may well cause problems fo r humans;
the machine may recognize those problems as problematic for human s; but, by
deﬁnition, they are not problematic within the standard model from the point of
view of the given objective.
•Intelligence is multidimensional, “so ‘smarter than human s’ is a meaningless con-
cept.”: This argument, due to Kevin Kelly (2017), draws on a staple of mode rn
psychology—thefact that a scalarIQdoes notdo justice to the fu ll rangeofcogni-
tive skills that humans possess to varyingdegrees. IQ is indeed a cru de measure of
human intelligence, but it is utterly meaningless for current AI syste ms, because
their capabilities across diﬀerent areas are uncorrelated. How do w e compare the
IQ of Google’s search engine, which cannot play chess, with that of D eep Blue,
which cannot answer search queries? None of this supports the ar gument that
because intelligence is multifaceted, we can ignore the risk from supe rintelligent
machines. If “smarterthan humans” is a meaningless concept, the n “smarterthan
gorillas” is also meaningless, and gorillas therefore have nothing to fe ar from hu-
mans. Clearly, that argument doesn’t hold water. Not only is it logically possible
for one entity to be more capable than another across all the relev ant dimensions
of intelligence, it is also possible for one species to represent an exist ential threat
to another even if the former lacks an appreciation for music and lite rature.
5Rodney Brooks (2017), forexample, asserts thatit’s imposs iblefora program to be “smartenough
that it would be able to invent ways to subvert human society t o achieve goals set for it by humans,
without understanding the ways in which it was causing probl ems for those same humans.” Often, the
argument adds the premise that people of greater intelligen ce tend to have more altruistic objectives,
a view that may be related to the self-conception of those mak ing the argument. Chalmers (2010)
points to Kant’s view that an entity necessarily becomes mor e moral as it becomes more rational,
while noting that nothing in our current understanding of AI supports this view when applied to
machines.
Solutions 7
1.4 Solutions
Can we tackle Wiener’s warning head-on? Can we design AI systems wh ose purposes
don’t conﬂict with ours, so that we’re sure to be happy with how they behave? On
the face of it, this seems hopeless, because it will doubtless prove in feasible to write
down our purposes correctly or imagine all the counterintuitive way s a superintelligent
entity might fulﬁll them.
If we treat superintelligent AI systems as if they wereblack boxes f rom outer space,
then indeed there is no hope. Instead, the approach we seem oblige d to take, if we
are to have any conﬁdence in the outcome, is to deﬁne some formal problem Fand
design AI systems to be F-solvers, such that the closer the AI system comes to solving
Fperfectly, the greater the beneﬁt to humans. In simple terms, th e more intelligent
the machine, the better the outcome for humans: we hope the mac hine’s intelligence
will be applied both to learning our true objectives and to helping us ac hieve them.
If we can work out an appropriate Fthat has this property, we will be able to create
provably beneﬁcial AI. Moreover,
There is, I believe, an approach that may work. Humans can reason ably be de-
scribed as having (mostly implicit and partially formed) preferences o ver their future
lives—–that is, given enough time and unlimited visual aids, a human cou ld express
a preference (or indiﬀerence) when oﬀered a choice between two f uture lives laid out
before him or her in all their aspects. (This idealization ignores the po ssibility that
our minds are composed of subsystems with eﬀectively incompatible p references; if
true, that would limit a machine’s ability to optimally satisfy our prefere nces, but it
doesn’tseemtopreventusfromdesigningmachinesthatavoidcata strophicoutcomes.)
The formal problem Fto be solved by the machine in this case is a game-theoretic
one: to maximize human future-life preferences subject to its initia l uncertainty as
to what they are, in an environment that includes human participant s. Furthermore,
although the future-life preferences are hidden variables, they’r e grounded in a volu-
minous source of evidence, namely, all of the human choices ever ma de.
This formulation sidesteps Wiener’s problem, because we do not put a ﬁxed pur-
pose in the machine according to which it can rank all possible futures . Instead, the
machine knows that it doesn’t know the true preference ranking, s o it naturally acts
cautiously to avoid violating potentially important but unknown prefe rences. (We can
certainly include fairly strong priors on the positive value of life, healt h, etc., to make
the machine more useful more quickly.) The machine may learn more ab out human
preferences as it goes along, of course, but it will never achieve co mplete certainty.
Such a machine will be motivated to ask questions, to seek permission or additional
feedback before undertaking any potentially risky course of actio n, to defer to human
instruction, and to allow itself to be switched oﬀ. These behaviors ar e not built in
via preprogrammed scripts or rules; rather, they fall out as solut ions of the formal
problem F.
As noted in the introduction, this involves a shift from a unary view of AI to a
binary one. The classical view, in which a ﬁxed objective is given to the machine, is
illustrated qualitatively in Figure 1.1. Once the machine has a ﬁxed obje ctive, it will
act to optimize the achievementofthe objective; its behavioris eﬀe ctively independent
8Human-Compatible AI
Fig. 1.1 (a) The classical AI situation in which the human objective i s considered ﬁxed
and known by the machine, depicted as a notional graphical mo del. Given the objective, the
machine’s behavior is (roughly speaking) independent of an y subsequent human behavior,
as depicted in (b). This unary view of AI is tenable only if the human objective can be
completely and correctly stated.
Fig. 1.2 When the human objective is unobserved, machine behavior is no longer inde-
pendent of human behavior, because the latter provides more information about the human
objective.
of the human’s behavior.6
1.4.1 Assistance games
This basic idea is made more precise in the framework of assistance games —originally
known as cooperative inverse reinforcement learning (CIRL) game s in the terminology
of Hadﬁeld-Menell et al.(2017a). The simplest case of an assistance game involves two
agents, one human and the other a robot. It is a game of partial inf ormation, because,
while the human knows the reward function, the robot does not—–e ven though the
robot’s job is to maximize it. It may involve a form of inverse reinforce ment learning
(Russell, 1998; Ng and Russell, 2000) because the robot can learn m ore about human
preferences from the observation of human behavior–—a proces s that is the dual of
reinforcement learning, wherein behavior is learned from rewards a nd punishments.
To illustrate assistance games, I’ll use the paperclip game . It’s a very simple game
in which Harriet the human has an incentive to “signal” to Robbie the ro bot some
information about her preferences. Robbie is able to interpret tha t signal because he
6The independence is not strict because the human’s behavior can provide information about the
state of the world. Thus, a passenger in an automated taxi cou ld tell the taxi that snipers have been
reported on the road it intends to take, picking oﬀ passenger s for fun; but this might aﬀect the taxi’s
behavior only if it already knows that death by gunﬁre is unde sirable for humans.
Solutions 9
H
R R R[2,0] [1,1] [0,2]
[90,0]
[50,50][0,90]$0.90 $1.00 $1.10
Fig. 1.3 The paperclip game. Each branch is labeled [ p,s] denoting the number of paper-
clips and staples manufactured on that branch. Harriet the h uman can choose to make two
paperclips, two staples, or one of each. (The values in green italics are the values for Harriet
if the game ended there, assuming θ=0.45.) Robbie the robot then has a choice to make 90
paperclips, 90 staples, or 50 of each.
can solve the game and therefore he can understand what would ha ve to be true about
Harriet’s preferences in order for her to signal in that way.
The steps of the game are depicted in Figure 1.3. It involvesmaking pa perclips and
staples. Harriet’s preferences are expressed by a payoﬀ functio n that depends on the
number of paperclips and the number of staples produced, with a ce rtain “exchange
rate” between the two. Harriet’s preference parameter θdenotes the relative value (in
dollars) of a paperclip; for example, she might value paperclips at θ=0.45 dollars,
which means staples are worth 1 −θ=0.55 dollars. So, if ppaperclips and sstaples
are produced, Harriet’s payoﬀ will be pθ+s(1−θ) dollars in all. Robbie’s prior is
P(θ) =Uniform (θ;0,1). In the game itself, Harriet goes ﬁrst, and can choose to
make two paperclips, two staples, or one of each. Then Robbie can c hoose to make 90
paperclips, 90 staples, or 50 of each.
Notice that if she were doing this by herself, Harriet would just make two staples,
with a value of $1.10. (See the annotations at the ﬁrst level of the t ree in Figure 1.3.)
But Robbie is watching, and he learns from her choice. What exactly d oes he learn?
Well, that depends on how Harriet makes her choice. How does Harrie t make her
choice? That depends on how Robbie is going to interpret it. One can r esolve this
circularity by ﬁnding a Nash equilibrium. In this case, it is unique and can be found
by applying the iterated-best-response algorithm: pick any strat egy for Harriet; pick
thebeststrategyforRobbie,givenHarriet’sstrategy;picktheb eststrategyforHarriet,
given Robbie’s strategy; and so on. The process unfolds as follows:
1. Start with the greedy strategy for Harriet: make two paperclip s if she prefers
paperclips; make one of each if she is indiﬀerent; make two staples if s he prefers
staples.
2. TherearethreepossibilitiesRobbiehastoconsider,giventhisstr ategyforHarriet:
(a) If Robbie sees Harriet make two paperclips, he infers that she p refers pa-
perclips, so he now believes the value of a paperclip is uniformly distribu ted
between 0.5 and 1.0, with an average of 0.75. In that case, his best p lan is to
make 90 paperclips with an expected value of $67.50 for Harriet.
10Human-Compatible AI
(b) If Robbie sees Harriet make one of each, he infers that she valu es paperclips
and staples at 0.50, so the best choice is to make 50 of each.
(c) If Robbie sees Harriet make two staples, then by the same argu ment as in
(a), he should make 90 staples.
3. Given this strategy for Robbie, Harriet’s best strategy is now so mewhat diﬀerent
from the greedy strategy in step 1. If Robbie is going to respond to her making
one of each by making 50 of each, then she is better oﬀ making one of each not
just if she is exactly indiﬀerent, but if she is anywhere close to indiﬀer ent. In fact,
the optimal policy is now to make one of each if she values paperclips an ywhere
between about 0.446 and 0.554.
4. Given this new strategy for Harriet, Robbie’s strategy remains u nchanged. For
example, if she chooses one of each, he infers that the value of a pa perclip is
uniformly distributed between 0.446 and 0.554, with an average of 0.5 0, so the
best choice is to make 50 of each. Because Robbie’s strategy is the s ame as in
step 2, Harriet’s best response will be the same as in step 3, and we h ave found
the equilibrium.
With her strategy, Harriet is, in eﬀect, teaching Robbie about her p references using a
simple code–—a language, if you like–—that emerges from the equilibriu m analysis.
Note also that Robbie never learns Harriet’s preferences exactly, but he learns enough
to act optimally on her behalf–—i.e., he acts (given his limited options) ju st as he
would, if he did know her preferences exactly. He is provably beneﬁc ial to Harriet
under the assumptions stated, and under the assumption that Ha rriet is playing the
game correctly.
It is possible to prove that provided there are no ties that cause co ordination prob-
lems, ﬁnding an optimal strategy for the robot in an assistance gam e can be done by
solving a single-agent partially observable Markov decision process ( POMDP) whose
state spaceisthe underlyingstate spaceofthe gameplusthe huma npreferenceparam-
etersθ. POMDPs in general are very hard to solve, but the POMDPs that re present
assistance games have additional structure that enables more eﬃ cient algorithms (Ma-
liket al., 2018).
1.4.2 The oﬀ-switch game
Within the same basic framework,one can also show that a robot solv ingan assistance
game will defer to a human and allow itself to be switched oﬀ. This prope rty is illus-
trated in oﬀ-switch game shown in Figure 1.4 (Hadﬁeld-Menell et al., 2017b). Robbie
is now helping Harriet ﬁnd a hotel room for the Ineternational Pape rclip Convention
in Geneva. Robbie can act now—let’s say he can book Harriet into a ver y expensive
hotel near the meeting venue. He is quite unsure how much Harriet w ill like the hotel
and its price; let’s say he has a uniform probability for its net value to H arriet be-
tween−40 and +60, with an average of +10. He could also “switch himself oﬀ”— less
melodramatically, take himself out of the hotel booking process alto gether—which is
deﬁned (without loss of generality) to have value 0 to Harriet. If th ose were his two
choices, he would go ahead and book the hotel, incurring a signiﬁcant risk of making
Harriet unhappy. (If the range were −60 to +40, with average −10, he would switch
himself oﬀ instead.) I’ll give Robbie a third choice, however: explain his p lan, wait,
Solutions 11
+60 –40U = ? U = 0act switch self off
U = 0
U = 0 U = ?wait
switch robot off
switch self off actwaitgo ahead
+60 –40R
H
R
Fig. 1.4 The oﬀ-switch game. R, the robot, can choose to act now, with a highly uncertain
payoﬀ; to switch itself oﬀ; or to defer to H, the human. Hcan switch Roﬀ or let it go ahead.
Rnow has the same choice again. Acting still has an uncertain p ayoﬀ, but now Rknows the
payoﬀ is nonnegative.
and let Harriet switch him oﬀ. Harriet can either switch him oﬀ or let him g o ahead
and book the hotel. What possible good could this do, one might ask, g iven that he
could make both of those choices himself?
The point is that Harriet’s choice—to switch Robbie oﬀ or let him go ahea d—
provides Robbie with information about Harriet’s preferences. I’ll a ssume, for now,
that Harriet is rational, so if Harriet lets Robbie go ahead, it means th e value to
Harriet is positive. Now, as shown in Figure 1.4, Robbie’s belief changes : it is uniform
between 0 and +60, with an average of +30.
So, if one evaluates Robbie’s initial choices from his point of view:
1. Acting now and booking the hotel has an expected value of +10.
2. Switching himself oﬀ has a value of 0.
3. Waiting and letting Harriet switch him oﬀ leads to two possible outcom es:
(a) There is a 40% chance, based on Robbie’s uncertainty about Har riet’s pref-
erences, that she will hate the plan and will switch Robbie oﬀ, with valu e
0.
(b) There is a60%chanceHarrietwill likethe plan and allowRobbie togoa head,
with expected value +30.
Thus, waiting has expected value (0 .4×0) + (0.6×30)= + 18, which is better than
the +10 Robbie epects if he acts now.
The upshot is that Robbie has a positive incentive to defer to Harriet —that is, to
allow himself to be switched oﬀ. This incentive comes directly from Robb ie’s uncer-
tainty about Harriet’s preferences. Robbie is aware that there’s a chance (40% in this
example) that he might be about to do something that will make Harrie t unhappy, in
which casebeing switched oﬀwouldbe preferableto goingahead. Wer eRobbie already
certain about Harriet’s preferences, he would just go ahead and m ake the decision (or
switch himself oﬀ); there would be absolutely nothing to be gained fro m consulting
Harriet, because, according to Robbie’s deﬁnite beliefs, he can alre ady predict exactly
what she is going to decide.
In fact, it is possible to prove the same result in the general case: a s long as Robbie
12Human-Compatible AI
is not completely certain that he’s about to do what Harriet herself w ould do, he is
better oﬀ allowing her to switch him oﬀ. Intuitively, her decision provid es Robbie with
information, and the expected value of information is always nonneg ative. Conversely,
if Robbie is certain about Harriet’s decision, her decision provides no n ew information,
and so Robbie has no incentive to allow her to decide.
Formally, let P(u) be Robbie’s prior probability density over Harriet’s utility for
the proposed action a. Then the value of going ahead with ais
EU(a) =/integraldisplay∞
−∞P(u)·udu=/integraldisplay0
−∞P(u)·udu+/integraldisplay∞
0P(u)·udu.
On the other hand, the value of action d, deferring to Harriet, is composed of two
parts: if u >0 then Harriet lets Robbie go ahead, so the value is u, but ifu <0 then
Harriet switches Robbie oﬀ, so the value is 0:
EU(d) =/integraldisplay0
−∞P(u)·0du+/integraldisplay∞
0P(u)·udu.
Comparing the expressions for EU(a) andEU(d), it follows immediately that
EU(d)≥EU(a)
because the expression for EU(d) has the negative-utility region zeroed out. The two
choices have equal value only when the negative region has zero pro bability—that is,
when Robbie is already certain that Harriet likes the proposed action .
There are some obvious elaborations on the model that are worth e xploring im-
mediately. The ﬁrst elaboration is to impose a cost for Harriet’s time. In that case,
Robbie is less inclined to bother Harriet if the downside risk is small. This is as it
should be. And if Harriet is really grumpy about being interrupted, sh e shouldn’t be
too surprised if Robbie occasionally does things she doesn’t like.
The second elaboration is to allow for some probability of human error —that is,
Harriet might sometimes switch Robbie oﬀ even when his proposed act ion is reason-
able, and she might sometimes let Robbie go ahead even when his propo sed action is
undesirable.Itisstraightforwardtofoldthiserrorprobabilityinto themodel(Hadﬁeld-
Menellet al., 2017b). As one might expect, the solution shows that Robbie is less
inclined to defer to an irrational Harriet who sometimes acts against her own best in-
terests. The more randomly she behaves, the more uncertain Rob bie has to be about
her preferences before deferring to her. Again, this is as it should be: for example, if
Robbie is a self-driving car and Harriet is his naughty two-year-old pa ssenger, Robbie
should not allow Harriet to switch him oﬀ in the middle of the highway.
The oﬀ-switch example suggests some templates for controllable-a gent designs and
provides a simple example of a provably beneﬁcial system in the sense introduced
above.The overallapproachresemblesprincipal–agentproblemsin economics,wherein
the principal (e.g., an employer) needs to incentivize another agent (e.g., an employee)
to behave in ways beneﬁcial to the principal. The key diﬀerence here is that we are
building one of the agents in order to beneﬁt the other. Unlike a huma n employee, the
robot should have no interests of its own whatsoever.
Solutions 13
Assistancegamescanbegeneralizedtoallowforimperfectlyrationa lhumans(Hadﬁeld-
Menellet al., 2017b), humans who don’t know their own preferences (Chan et al.,
2019), multiple human participants, multiple robots, and so on. Scalin g up to complex
environments and high-dimensional perceptual inputs may be poss ible using methods
related to deep inverse RL. By providing a factored or structured action space, as
opposed to the simple atomic actions in the paperclip game, the oppor tunities for
communication can be greatly enhanced. Few of these variations ha ve been explored
so far, but I expect the key property of assistance games to rem ain true: robots that
solve such games will be beneﬁcial (in expectation) to humans (Hadﬁ eld-Menell et al.,
2017a).
While the basic theory of assistance games assumes perfectly ratio nal robots that
can solve the assistance game exactly, this is unlikely to be possible in p ractical situ-
ations. Indeed, one expects to ﬁnd qualitatively diﬀerent phenome na occurring when
the robot is much less capable than, roughly as capable as, or much m ore capable
than the human. There is good reason to hope that in all cases impro ving the robot’s
capability will be beneﬁcial to the human, because it will do a better jo b of learning
human preferences and a better job of satsifying them.
1.4.3 Acting with unknown preferences
Multiattribute utility theory (Keeney and Raiﬀa, 1976) views the wor ld as composed
of a set of attributes {X1,...,X n}, with preferences deﬁned on lotteries over complete
assignments to the attributes. This is clearly an oversimpliﬁcation, b ut it suﬃces for
our purpose in exploring some basic phenomena.
In some cases, a machine’s scope of action is strictly limited. For exam ple, a (non-
Internet-connected) thermostat can only turn the heating on a nd oﬀ, and, to a ﬁrst
approximation, aﬀects the temperature in the house and the owne r’s bank balance.7It
is plausible in this case to imagine that the thermostat might develop a d ecent model
of the user’s preferences over temperature and cost attribute s.
In the great majority of circumstances, however, the AI system ’s knowledge of
human preferences will be extremely incomplete compared to its sco pe of action. How
can it be useful if this is the case? Can it even fetch the coﬀee?
Itturnsoutthat theanswerisyes,ifweunderstand“fetchthec oﬀee”therightway.
“Fetch the coﬀee” does not divide the world into goal states (wher e the human has
coﬀee) and non-goal states. Instead, it says that the human’s c urrent preferences rank
coﬀee states above non-coﬀee states all other things being equal . This idea of goals as
ceteris paribus comparativesiswell-established(vonWright,1972;WellmanandDoyle ,
1991). In this context, it suggests that the machine should act in a minimally invasive
fashion—that is, satisfy the preferences it knows about (coﬀee) without disturbing any
other attributes of the world.
7In reality, it is very diﬃcult to limit the eﬀects of the agent ’s actions to a small set of attributes.
Turning the heat oﬀ may make the occupants more susceptible t o viral infection, and proﬂigate
heating may tip the occupants into bankuptcy, and so on. A dev ice connected to the Internet, with
the ability to send character streams, can aﬀect the entire p lanet through propaganda, online trading,
etc.
14Human-Compatible AI
There remains the question of why the machine should assume that le aving other
attributes unaﬀected is better than disturbing them in somerando m way.One possible
answer is some form of risk aversion, but I suspect this is not enoug h. One has a sense
that a machine that does nothing is better than one that acts rand omly, but this is
certainly not implicit in the standard formulation of MDPs. I think one h as to add
the assumption that the world is not in an arbitrary state; rather, it resembles a state
sampled from the stationarydistribution that resultsfrom the act ionsofhuman agents
operating according to their preferences (Shah et al., 2019). In that case, one expects
a random action to make things worse for the humans.
There is another kind of action that is beneﬁcial to humans even whe n the machine
knows nothing at all about human preferences: an action that simp ly expands the set
of actions available to the human. For example, if Harriet has forgot ten her password,
Robbie can giveher the password,enabling a wider rangeof actions t han Harriet could
otherwise execute.
1.5 Reasons for Optimism
There are some reasons to think this approach may work in practice . First, there
is abundant written and ﬁlmed information about humans doing things (and other
humans reacting). More or less every book ever written contains e vidence on this
topic. Even the oldest clay tablets, tediously recording the exchan ge of N sheep for M
oxen, give information about human preferences between sheep a nd oxen. Technology
tobuild modelsofhumanpreferencesfromthisstorehousewillpres umablybe available
long before superintelligent AI systems are created.
Second, there are strong near-term economic incentives for rob ots to understand
human preferences, which also come into play well before the arriva l of superintel-
ligence. Already, computer systems record one’s preferences fo r an aisle seat or a
vegetarian meal. More sophisticated personal assistants will need to understand their
user’s preferences for cost, luxury, and convenient location whe n booking hotels, and
how these preferences depend on the nature and schedule of the user’s planned activ-
ities. Managing a busy person’s calendar and screening calls and emails requires an
even more sophisticated understanding of the user’s life, as does t he management of
an entire household when entrusted to a domestic robot. For all su ch roles, trust is
essential but easily lost if the machine reveals itself to lack a basic und erstanding of
human preferences. If one poorly designed domestic robot cooks the cat for dinner, not
realizing that its sentimental value outweighs its nutritional value, t he domestic-robot
industry will be out of business.
For companies and governments to adopt the new model of AI, a gr eat deal of
research must be done to replace the entire toolbox of AI methods , all of which have
been developed on the assumption that the objective is known exac tly. There are two
primary issues for each class of task environments: how to relax th e assumption of a
known objective and what form of interaction to assume between t he machine and the
human. For example, problem-solving task environments have an ob jective deﬁned by
a goal test G(s) and a stepwise cost function c(s,a,s′). Perhaps the machine knows a
relaxed predicate G′⊃Gand upper and lower bounds c+andc−on the cost function,
and can ask the human (1) whether any given state ssatisﬁesGand (2) whether one
Obstacles 15
trajectory to sis preferred to another. Design considerations include formal pre cision,
algorithmic complexity, feasibility of the interaction protocol from t he human point of
view, and applicability in real-world circumstances.
The standard model of AI as maximizing objectives does not imply tha t all AI sys-
tems have to solve some particular problem formulation such as an inﬂ uence diagram
or a factored MDP. For example, it is entirely consistent with the sta ndard model to
build an AI system directly as a policy, expressed as a set of condition –action rules
specifying the optimal action in each category of states.8By the same token, I am
not proposing that all AI systems under the new model have to solv e some explicitly
formulated representation of the assistance game. It is importan t to maintain a broad
conception of the approach and how it applies to the design of AI sys tems for any par-
ticular task environment. The crucial elements are (1) acknowledg ement that there is
partial and uncertain information about the true human preferen ces that are relevant
in the task environment; (2) a means for information to ﬂow at run- time from humans
to machines concerning those preferences; and (3) allowance for the human to be a
joint participant in the run-time process.
1.6 Obstacles
Thereareobviousdiﬃculties withanapproachthat expectsmachine stolearnunderly-
ingpreferencesfromobservinghumanbehavior.Theﬁrstis thath umansareirrational,
in the sense that our actions do not reﬂect our preferences. This irrationality arises
in part from our computational limitations relative to the complexity o f the decision
problems we face. For example, if two humans are playing chess and o ne of them loses,
it’s because the loser (and possibly the winner too) made a mistake—– a move that
led inevitably to a forced loss. A machine observing that move and ass uming perfect
rationality on the part of the human might well conclude that the hum anpreferred
to lose. Thus, to avoid reaching such conclusions, the machine must take into account
theactualcognitive mechanisms of humans.
As yet, we do not know enough about human cognitive mechanisms to invert real
human behavior to get at the underlying preferences. One thing th at seems intuitively
clear, however, is that one of our principal methods for coping with the complexity
of the world is to organize our behavior hierarchically. That is, we mak e (defeasible)
commitments to higher-level goals such as “write an essay on a huma n-compatible
approach to AI”; then, rather than considering all possible seque nces of words, from
“aardvarkaardvarkaardvark...”to“zyzzyvazyzzyvazyzzy va...”asachessprogram
would do, we choose among subtasks such as “write the introductio n” and “read more
about preference elicitation.” Eventually, we get down to the choice of words, and then
typing each word involves a sequence of keystrokes, each of which is in turn a sequence
of motor control commands to the muscles of the arms and hands. At any given point,
then, a human is embedded at various particular levels of multiple deep and complex
hierarchies of partially overlapping activities and subgoals. This mean s that for the
8Many applications of control theory work exactly this way: t he control theorist works oﬄine with
a mathematical model of the system and the objective to deriv e acontrol law that is then implemented
in the controller.
16Human-Compatible AI
machine to understand human actions, it probably needs to unders tand a good deal
about what these hierarchies are and how we use them to navigate t he real world.
Machines might try to discover more about human cognitive mechanis ms by an
inductive learning approach. Suppose that in some given state sHarriet’s action a
depends on her preferences θaccording to mechanism h, i.e.,a=h(θ,s). (Here, θ
represents not a single parameter such as the exchange rate bet ween staples and pa-
perclips, but the Harriet’s preferences over future lives, which co uld be a structure of
arbitrarycomplexity.) By observingmanyexamples of sanda, is it possible eventually
to recover handθ? At ﬁrst glance, the answerseems to be no (Armstrong and Minder -
mann, 2019). For example, one cannot distinguish between the follo wing hypotheses
about how Harriet plays chess:
1.hmaximizes the satisfaction of preferences, and θis the desire to win games.
2.hminimizes the satisfaction of preferences, and θis the desire to lose games.
From the outside, Harriet plays perfect chess under either hypot hesis.9If one is merely
concerned with predicting her next move, it doesn’t matter which fo rmulation one
chooses. On the other hand, for a machine whose goal is to help Har riet realize her
preferences, it really does matter! The machine needs to know whic h explanation
holds. From this viewpoint, something is seriously wrong with the seco nd explanation
of behavior. If Harriet’s cognitive mechanism hwere really trying to minimize the
satisfaction of preferences θ, it wouldn’t make sense to call θher preferences. It is,
then, simply a mistake to suppose that handθare separately and independently
deﬁned. I have already argued that the assumption of perfect ra tionality—–i.e., h
is maximization—–is too strong; yet, for it to make sense to say that Harriet has
preferences, hwill have to satisfy (or nearly satisfy) some basic properties assoc iated
with rationality. These might include choosing correctly according to preferences in
situations that are computationally trivial—–for example, choosing b etween vanilla
and bubble-gum ice cream at the beach. Cherniak (1986) presents an in-depth analysis
of these minimal conditions on rationality.
Furtherdiﬃculties ariseifthemachinesucceedsinidentifyingHarriet ’spreferences,
but ﬁnds them to be inconsistent. For example, suppose she prefe rs vanilla to bubble
gum and bubble gum to pistachio, but prefers pistachio to vanilla. In t hat case her
preferences violate the axiom of transitivity and there is no way to m aximally satisfy
her preferences. (That is, whatever ice cream the machine gives h er, there is always
another that she would prefer.) In such cases, the machine could a ttempt to satisfy
Harriet’s preferences up to inconsistency ; for example, if Harriet strictly prefers all
three of the aforementioned ﬂavors to licorice, then it should avoid giving her licorice
ice cream.
Of course, the inconsistency in Harriet’s preferences could be of a far more radical
nature. Many theories of cognition, such as Minsky’s Society of Mind (1986), posit
multiple cognitive subsystems that, in essence, have their own pref erence structures
9Of course, the Harriet who prefers to lose might grumble when she keeps winning, thereby giving
a clue as to which Harriet she is. One response to this is that g rumbling is just more behavior, and
equally subject to multiple interpretations. Another resp onse is to say that Harrietmight feel grumbly
but, in keeping with her minimizing h, would instead jump for joy. This is not to say that there is no
fact of the matter as to whether Harriet is pleased or displea sed with the outcome.
Obstacles 17
and compete for control—–and these seem to be manifested in addic tive and self-
destructive behaviors, among others. Such inconsistencies place limits on the extent
to which the idea of machines helping humans even makes sense.
Also diﬃcult, from a philosophical viewpoint, is the apparent plasticity of human
preferences—the fact that they seem to change over time as the result of experiences.
It is hard to explain how such changes can be made rationally, becaus e they make
one’s future self less likely to satisfy one’s present preferences ab out the future. Yet
plasticity seems fundamentally important to the entire enterprise, because newborn
infants certainly lack the rich, nuanced, culturally informed prefer ence structures of
adults. Indeed, it seems likely that our preferences are at least pa rtially formed by a
process resembling inverse reinforcement learning, whereby we ab sorb preferences that
explain the behavior of those around us. Such a process would tend to give cultures
some degree of autonomy from the otherwise homogenizing eﬀects of our dopamine-
based reward system.
Plasticity also raises the obvious question of which Harriet the machin e should try
to help: Harriet 2020, Harriet 2035, orsome time-averagedHarriet? (See Pettigrew(2020)
for a full treatment of this approach,wherein decisions for individu als who changeover
timearemadeasiftheyweredecisionsmadeonbehalfofmultipledistinc tindividuals.)
Plasticityisalsoproblematicbecauseofthepossibilitythatthemachin emay,bysubtly
inﬂuencing Harriet’s environment, gradually mould her preferences in directions that
make them easier to satisfy, much as certain political forces have b een said to do with
voters in recent decades.
Iam often asked,“Whosevaluesshouldwe alignAI with?” (The quest ionis usually
posed in more accusatory language, as if my secret, Silicon-Valley-h atched plan is
to align all the world’s AI systems with my own white, male, Western, cis gender,
Episcopalian values.) Of course, this is simply a misunderstanding. The kind of AI
system proposed here is not “aligned” with any values, unless you co unt the basic
principle of helping humans realizetheir preferences.For eachof th e billions ofhumans
on Earth, the machine should be able to predict, to the extent that its information
allows, which life that person would prefer.
Now, practical and social constraints will prevent all preference s from being max-
imally satisﬁed simultaneously. We cannot all be Ruler of the Universe. This means
that machines must mediate among conﬂicting preferences—–some thing that philoso-
phers and social scientists have struggled with for millennia. At one e xtreme, each
machine could pay attention only to the preferences of its owner, s ubject to legal con-
straints on its actions. This seems undesirable, as it would have a mac hine belonging
to a misanthrope refuse to aid a severely injured pedestrian so tha t it can bring the
newspaper home more quickly. Moreover, we might ﬁnd ourselves ne eding many more
laws as machines satisfy their owners’ preferences in ways that ar e very annoying to
others even if not strictly illegal. At the other extreme, if machines c onsider equally
the preferences of all humans, they might focus a larger fraction of their energies on
the least fortunate than their their owners might prefer—a state of aﬀairs not con-
ducive to investment in AI. Presumably, some middle ground can be fo und, perhaps
combining a degree of obligation to the machine’s owner with public subs idies that
support contributions to the greatergood. Determining the ideal solution for this issue
18Human-Compatible AI
is an open problem.
Another common question is, “What if machines learn from evil people ?” Here,
thereisarealissue.Itis notthatmachineswilllearntocopyevilactions.Themachine’s
actions need not resemble in any way the actions of those it observe s, any more than
a criminologist’s actions resemble those of the criminals she observes . The machine
is learning about human preferences; it is not adopting those prefe rences as its own
and acting to satisfy them. For example, suppose that a corrupt p assport oﬃcial in
a developing country insists on a bribe for every transaction, so th at he can aﬀord
to pay for his children to go to school. A machine observing this will not learn to
take bribes itself: it has no need of money and understands (and wis hes to avoid)
the toll imposed on others by the taking of bribes. The machine will ins tead ﬁnd
other, socially beneﬁcial ways to help send the children to school. Sim ilarly, a machine
observinghumanskillingeachotherinwarwill notlearnthat killingisgood :obviously,
those on the receiving end very much prefer not to be dead.
The diﬃcult issue that remains is this: what should machines learn from humans
who enjoy the suﬀering of others? In such cases, any simple aggre gation scheme for
preferences (such as adding utilities) would lead to some reduction in the utilities
of others in order to satisfy, at least partially, these perverse pr eferences. It seems
reasonable to require that machines simply ignore positive weights in t he preferences
of some for the suﬀering of others (Harsanyi, 1977).
1.7 Looking Further Ahead
If we assume, for the sake of argument, that all of these obstac les can be overcome,
as well as all of the obstacles to the development of truly capable AI systems, are
we home free? Would provably beneﬁcial, superintelligent AI usher in a golden age
for humanity? Not necessarily. There remains the issue of adoption : how can we ob-
tain broad agreement on suitable design principles, and how can we en sure that only
suitably designed AI systems are deployed?
On the question of obtaining agreement at the policy level, it is necess ary ﬁrst to
generate consensus within the research community on the basic ide as of–—and design
templates for–—provably beneﬁcial AI, so that policy makers have some concrete
guidance on what sorts of regulations might make sense. The econo mic incentives
noted earlier are of the kind that would tend to support the installat ion of rigorous
standards at the early stages of AI development, because failure s would be damaging
to entire industries, not just to the perpetrator and victim. We alr eady see this in
miniature with the imposition of machine-checkable software standa rds for cell-phone
applications.
OnthequestionofenforcementofpoliciesforAIsoftwaredesign, Iamlesssanguine.
If Dr. Evil wants to take over the world, he or she might remove the safety catch, so
to speak, and deploy an AI system that ends up destroying the wor ld instead. This
problem is a hugely magniﬁed version of the problem we currently face with malware.
Our track record in solving the latter problem does not provide grou nds for optimism
concerning the former. In Samuel Butler’s Erewhon and in Frank Herbert’s Dune,
the solution is to ban all intelligent machines, as a matter of both law an d cultural
Conclusion 19
imperative. Perhaps if we ﬁnd institutional solutions to the malware p roblem, we will
be able to devise some less drastic approach for AI.
The problem of misuse is not limited to evil masterminds. One possible fu ture for
humanity in the ageof superintelligent AI is that ofa raceof lotus eat ers, progressively
enfeebled as machines take over the management of our entire civiliz ation. This is the
future imagined in E. M. Forster’s story The Machine Stops , written in 1909. We
may say, now, that such a future is undesirable; the machines may a gree with us and
volunteer to stand back, requiring humanity to exert itself and main tain its vigor. But
exertion is tiring, and we may, in our usual myopic way, design AI syst ems that are
notquiteso concerned about the long-term vigor of humanity and just a littlemore
helpful than they would otherwise wish to be. Unfortunately, this p rocess continues in
a direction that is hard to resist.
1.8 Conclusion
Finding a solution to the AI control problem is an important task; it ma y be, in
Bostrom’s words, “the essential task of our age.” It involves buildin g systems that
are far more powerful than ourselves while still guaranteeing that those systems will
remain powerless, forever.
Up to now, AI researchhas focused on systems that are better a t making decisions,
but this is not the same as making better decisions. No matter how ex cellently an
algorithm maximizes, and no matter how accurate its model of the wo rld, a machine’s
decisions may be ineﬀably stupid, in the eyes of an ordinary human, if it fails to
understand human preferences.
This problem requires a change in the deﬁnition of AI itself—–from a ﬁe ld con-
cerned with a unary notion of intelligence as the optimization of a given objective,
to a ﬁeld concerned with a binary notion of machines that are provab ly beneﬁcial for
humans. Takingthe problem seriouslyseems likelyto yield new waysoft hinking about
AI, its purpose, and our relationship to it.
References
Armstrong, Stuart and Mindermann, S¨ oren (2019). Occam’s raz or is insuﬃcient
to infer the preferences of irrational agents. In Advances in Neural Information
Processing Systems 31 .
Bostrom, Nick (2014). Superintelligence: Paths, Dangers, Strategies . Oxford Univer-
sity Press.
Brooks, Rodney (2017). The seven deadly sins of AI predictions. M IT Technology
Review, October 6.
Chalmers, David John (2010). The singularity: A philosophical analys is.Journal of
Consciousness Studies ,17, 7–65.
Chan, Lawrence, Hadﬁeld-Menell, Dylan, Srinivasa, Siddhartha, an d Dragan, Anca
(2019). The assistive multi-armed bandit. In Proceedings of the Fourteenth
ACM/IEEE International Conference on Human–Robot Interac tion.
Cherniak, C. (1986). Minimal Rationality . MIT Press.
Gates, W. (2015). Ask me anything. Reddit, January 28.
Hadﬁeld-Menell, Dylan, Dragan, Anca D., Abbeel, Pieter, and Russell, Stuart J.
(2017a). Cooperative inverse reinforcement learning. In Advances in Neural Infor-
mation Processing Systems 29 .
Hadﬁeld-Menell, Dylan, Dragan, Anca D., Abbeel, Pieter, and Russell, Stuart J.
(2017b). The oﬀ-switch game. In Proceedings of the Twenty-Sixth International
Joint Conference on Artiﬁcial Intelligence .
Harsanyi, John (1977). Morality and the theory of rational behav ior.Social Re-
search,44, 623–656.
Keeney, Ralph L. and Raiﬀa, Howard (1976). Decisions with Multiple Objectives:
Preferences and Value Tradeoﬀs . Wiley.
Kelly, Kevin (2017). The myth of a superhuman AI. Wired, April 25.
Kumparak, G. (2014). Elon Musk compares building artiﬁcial intelligen ce to ‘sum-
moning the demon’. TechCrunch, October 26.
Malik, Dhruv, Palaniappan, Malayandi, Fisac, Jaime F., Hadﬁeld-Menell, Dylan,
Russell, Stuart J., and Dragan, Anca D. (2018). An eﬃcient, gener alized Bellman
update for cooperative inverse reinforcement learning. In Proceedings of the Thirty-
Fifth International Conference on Machine Learning .
Minsky, Marvin L. (1986). The Society of Mind . Simon and Schuster.
Ng, Andrew Y. and Russell, Stuart J. (2000). Algorithms for invers e reinforcement
learning. In Proceedings of the Seventeenth International Conference o n Machine
Learning .
Omohundro, S. (2008). The basic AI drives. In AGI-08 Workshop on the Sociocul-
tural, Ethical and Futurological Implications of Artiﬁcia l Intelligence .
Osborne,H.(2017). StephenHawkingAIwarning:Artiﬁcialintellige ncecoulddestroy
References 21
civilization. Newsweek, November 7.
Pettigrew, Richard (2020). Choosing for Changing Selves . Oxford University Press.
Russell, Stuart J. (1998). Learning agents for uncertain environ ments. In Proceedings
of the Eleventh ACM Conference on Computational Learning Th eory.
Shah, Rohin, Krasheninnikov, Dmitrii, Alexander, Jordan, Abbeel, P ieter, and Dra-
gan, Anca (2019). The implicit preference information in an initial sta te. InPro-
ceedings of the Seventh International Conference on Learni ng Representations .
Stone, Peter, Brooks, Rodney A., Brynjolfsson, Erik, Calo, Ryan , Etzioni, Oren,
Hager, Greg, Hirschberg, Julia, Kalyanakrishnan, Shivaram, Kama r, Ece, Kraus,
Sarit et al. (2016). Artiﬁcial intelligence and life in 2030. Technical re port, Stanford
University One Hundred Year Study on Artiﬁcial Intelligence: Repor t of the 2015-
2016 Study Panel.
Turing, A. (1951). Can digital machines think? Lecture broadcast on BBC Third
Programme. Typescript available at www.turingarchive.org.
von Wright, Georg (1972). The logic of preference reconsidered. Theory and Deci-
sion,3, 140–167.
Wellman,M.P.andDoyle,Jon(1991).Preferentialsemanticsforg oals.InProceedings
of the Ninth National Conference on Artiﬁcial Intelligence .
Wiener, Norbert (1960). Some moral and technical consequence s of automation.
Science,131(3410), 1355–1358.


Is the rise of killer machines closer than we think?.

The New Fisker Ocean
Dynamic. All-electric. Up to 350-mile range. Feel the
Ocean.
FISKER Jetzt einkaufen
Is the rise of killer machines closer than
we think?
British computer scientist and world-leading AI expert
Stuart Russell says we urgently need to understand the
potential of superintelligent machines – before they turn on
us. By Damian WhitworthINTERVIEW
PREVIOUS ARTICLE  NEXT ARTICLEMENU
 saturday january 29 2022Magazine
Stuart Russell, 59
RICHARD ANSETT/BBC
Damian Whitworth
Saturday January 29 2022,  12.01am GMT, The Times
Share Save
Acouple of years ago Stuart Russell, a British computer
scientist who is one of the world’s leading experts on
artiﬁcial intelligence, was approached by a ﬁlm
director who wanted him to be a consultant on a
movie. The director complained that there was too much doom
and gloom about the future of superintelligent machines. He
wanted Russell to explain how the human heroes in the ﬁlm
could save our species by outwitting AI. “Sorry,” Russell told the
director. “They can’t.”
Russell is a professor of computer science at the University of
California, Berkeley, and a fellow of Wadham College, Oxford,
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
who has worked for the UN to create a new system for
monitoring the nuclear test-ban treaty, advised Downing Street
and the White House, and co-written the standard university
textbook on artiﬁcial intelligence. Success in creating
superintelligent AI, he has predicted, “would be the biggest
event in human history… and perhaps the last event in human
history”.
AI could lead us into a golden age, where we can enjoy lives that
are no longer burdened by drudgery. Or it could destroy us as a
species. Even if we learn to live with superintelligent machines,
they may take all our jobs or create mayhem on battleﬁelds.
Vladimir Putin has said whoever takes the lead in AI “will
become the ruler of the world”, prompting the billionaire
entrepreneur Elon Musk to predict that nations competing for
AI superiority will be the most likely cause of a third world war.
When Russell gave the Reith lectures last year, the headlines
were mostly about the havoc that lethal autonomous weapons
systems could wreak. But Russell has a wider vision, which is by
turns thrilling and more terrifying than coronaviruses and
global warming.
While the human brain has evolved over millions of years, the
development of computers and robots to simulate the human
mind’s ability to solve problems, make decisions and learn has
taken a few decades. From the very beginning of AI, says
Russell, machines have been deﬁned as intelligent “to the
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
extent that their actions can be expected to achieve their
objectives”. We set them  tasks and they get on with them.
ADVERTISEMENT
He believes we should make a very signiﬁcant tweak to that
deﬁnition so that machines are seen as “beneﬁcial” to the
extent that their actions can be expected to achieve “our”
objectives. If we don’t design them  with our wellbeing
speciﬁcally in mind, we could be creating an existential
problem for ourselves.
In the past decade AI has started to fulﬁl some of its promise.
Machines can thrash us at chess. When Russell was taking a
sabbatical in Paris, he used machine translation to complete his
tax return. In a recent breakthrough that could transform
medicine, AI can now predict the structure of most proteins.
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
Today Russell is on a visit to the UK and we are sitting outside a
café in London, our conversation recorded by an app on my
phone that has learnt to recognise my voice and provides a
reasonable simultaneous transcription of our conversation
(although its claim, for example, that Russell is talking about
“kick-ass machines made of cheese” does underline that AI
armageddon is still some way o).
These AIs are limited to harnessing considerable computational
power to complete well-deﬁned tasks. Google’s search engine
“remembers” everything, but can’t plan its way out of a paper
bag, as Russell puts it. The goal of AI research is creating a
general-purpose AI that can learn how to perform the whole
range of human tasks from, say, teaching to running a country.
Such a machine “could quickly learn to do anything that human
beings can do”, says Russell. And given that computers can
already add billion-digit numbers in a fraction of a second,
“Almost certainly it would be able to do things that humans
can’t do.”
The creation of a superintelligent AI, which Russell has likened
to the arrival of a superior alien civilisation (but more likely), is
an enormous challenge and a long way o. But many experts
believe it could happen in the next few decades, and Russell is
an evangelist for the need  to prepare for such an eventuality.
SPONSORED
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
He likes to talk about Alan Turing, the father of theoretical
computer science and AI, who in 1951 gave a lecture in which he
chillingly predicted the arrival of superintelligent machines. “It
seems probable that once the machine thinking met hod had
started, it would not take long to outstrip our feeble powers,”
said Turing. “At some stage therefore we should have to expect
the machines to take control.”
The danger, Russell suggests, is that our relationship with
machines becomes analogous to the relationship gorillas have
with us today. We had a common ancestor but “once humans
came along, and they’re this much more intelligent than gorillas
and chimpanzees, then game over. I think that’s sort of how
Turing saw it. Intelligence is power. Power is control. That will
be the end of it.”
Russell doesn’t believe that is necessarily the end of it, if we go
about things the right way. But he wants us to be clear about the
threat. Science ﬁction has sometimes suggested that machines
will supersede us when they develop human consciousness;
that when they are aware of them selves and their surroundings
and motivations, they will seek to take over the world. Russell
Three bosses discuss the
This
country
claims
to offer
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
believes this is a red herring. The threat will come less from
machines deciding they hate us and want to kill us than from
their advanced competency. A highly sophisticated machine
with a ﬁxed objective could stop at nothing to achieve that
objective and fail to take into account other human priorities.
He calls this the “King Midas problem” after the mythical ﬁgure
who asked for everything he touched to be turned to gold,
realising too late that this would include food, drink and his
family.
ADVERTISEMENT
Already we give machines objectives that are not perfectly
aligned with our own. Social-media algorithms are designed to
maximise click-through in order to keep people on the site and
so make as much money as possible from advertising. They
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
have unfortunate side eects. Users with more extreme
preferences appear to be more predictable, says Russell, so the
algorithm works out what keeps them  online and the diet of
content they are fed is contributing to growing extremism
around the world. “When a person is interacting with a system
for six or eight hours a day, the algorithm is making choices that
aect your behaviour, nudging you hundreds of times a day.
And that’s happening to billions of people.” He would love to
see the internal data from big tech companies “to really
understand what’s going on”, but adds, “In America, you’ve got
60 million people who are living in a fantasy world.”
Imagine a more sophisticated AI that is capable of going into a
coee shop to get you a latte. It will be unhelpful to café society
if it tears the place apart because it is ﬁxed on achieving the
task whatever the cost.
Here we are entering the territory of 2001: A Space Odyssey, in
which Hal, the spaceship computer, kills four of the ﬁve
astronauts on board because he deems them  a threat to the
mission.
AI’s potential to help in medicine is already being realised, but
Russell raises the spectre of a superintelligent AI system being
charged with ﬁnding a cure for cancer. It could quickly digest all
the literature and make hypotheses, but all that will be wildly
counterproductive if it then concludes that the quickest way to
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
ﬁnd a cure is to induce tumours in all of us in order to carry out
trials.
We might recruit an AI to ﬁght the acidiﬁcation of the oceans,
only to ﬁnd that its solution is to use a quarter of the oxygen in
the atmosphere to achieve this and we all asphyxiate.
Solving the King Midas problem also solves the gorilla problem,
by ensuring that AI is not in conﬂict with humans and we don’t
end up existing at the whim of the machines.
So we need  to create AI systems carefully. They must be built so
they are altruistic towards humans and uncertain about what all
our preferences are. Then the AI system would ask what our
preferences are regarding oxygen before going ahead and
deacidifying the oceans.
ADVERTISEMENT
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
“We have to build machines a dierent way [so that] they are
trying to achieve whatever our objective is but they know that
there may be other things we care about. So if we say, ‘I’d like a
cup of tea,’ that doesn’t mean you can mow down all the other
people at Starbucks to get to the front of the line.”
And the machine must be devised so it will always allow us to
turn it o. Otherwise, its logical conclusion would be to
deactivate its “o” switch in order to eliminate an obvious
threat to completing the task.
Given the starkness of some of his misgivings about the future, I
was expecting Russell to be an intense prophet of cyber-doom
in real life, but he is reasonable, softly spoken with a mid-
Atlantic accent, and often funny, displaying an understated wit
that is familiar from some of his writings.
He is in London for a holiday with his wife, Loy Sheﬂ ott,
founder and CEO of Consumer Financial, a marketing ﬁrm for
ﬁnancial services companies. They have four children who
range in age from 15 to 23.
Russell, 59, was born in Portsmouth and moved around the
country because of his father’s job running Crown Paints and
Wallcoverings. They also lived in Toronto for a few years. His
mother was a fashion designer and teacher. Russell boarded at
St Paul’s School in southwest London where even in an
academic hothouse environment he clearly stood out. The
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
school didn’t teach computer studies back then, so he went on
Wednesday afternoons to a local technology college where he
could study the subject for A-level.
He left school at 16 having taken his A-levels early, spent a gap
year at IBM and then,  at 17, went to Oxford, where he was
awarded a ﬁrst in physics. He moved to the US to do a PhD in
computer science at Stanford University and then joined the
University of California at Berkeley, where he is professor of
electrical engineer ing and computer sciences and director of the
Centre for Human-Compatible Artiﬁcial Intelligence. With Peter
Norvig, Google’s former research director, he wrote the standard
university textbook on AI and in his most recent book, Human
Compatible: AI and the Problem of Control, he outlined some of
his concerns about the future of artiﬁcial intelligence.
Even if machines don’t take over the planet and eradicate us
and we ﬁnd a way to stay in control, living with them may
present enormous challenges. What happens when they can do
all – or, at least, the vast majority – of the roles that ﬁll our
working days? While he says they are currently useless at
interviewing, it seems a reasonable bet that there are future
interviewers being born today who will be made redundant by
AI, along with house painters, drivers and radiographers.
ADVERTISEMENT
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
For many millennia, Russell points out, most humans have been
in “robot” jobs; if they are released from agricultural, industrial
and clerical roles by real robots it could transform human
existence. “If all goes well, it will herald a golden age for
humanity. Our civilisation is the result of our intelligence, and
having access to much greater intelligence could enable a much
better civilisation,” he said in one of his Reith lectures.
Robots could build bridges, improve crop yields, cook for 100
people, run elections, while we get on with… what? We would
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
need  to reconﬁgure our economy and ﬁnd new purpose while
ensuring we don’t become enfeebled by relying on machines.
A lot of us, suggests Russell, will be engaged in interpersonal
services, supplying our humanity to others, whether as
therapists, tutors or companions. We would have all the time in
the world to strive to perfect the art of living, through art,
gardening or playing games.
“The need  will not be to eat or be able to aord a place to live,
but the need  for purpose,” says Russell. We are used to adapting
to new  jobs, but less so to having no job at all.
Is there not a danger that we end up with millions of therapists
and slightly crap artists? “I don’t feel that’s the route to
fulﬁlment,” he says, smiling.
The most immediate problem facing us comes in the form of
lethal autonomous weapons. They are already with us. The
threat is not that AI weapons are going to turn upon us because
our objectives and theirs collide, but that they can be used by
nefarious states or groups to target their enemies.
Israel’s Harop has a 10ft wingspan and the ability to loiter and
search for targets and, when it recognises them , make a
kamikaze attack. The UN has reported that a smaller drone may
have autonomously targeted militia ﬁghters in Libya.
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
Miniature drones could be mass-produced cheaply, says Russell,
and you could pack a million of them into a shipping container
and then track people through technology that recognises a face
or “anything you want: yarmulkes or turbans or whatever”.
He can envisage a mass attack by a swarm. “I think it could
happen that we would get attacks with a million weapons.”
We’ve legislated internationally against biological and chemi cal
weapons and to stop nuclear proliferation. The systems are not
perfect, but do mean the world community can go after those
who don’t comply and make it hard for them  to get the
ingredients to create these weapons. Russell is frustrated by the
reluctance of governments, including the UK and US, to ban
lethal autonomous weapons outright. Ocials at the Obama
White House listened very carefully when he was part of a
delegation there. “Their response on weapons of mass
destruction was, ‘But we would never make weapons like that.’
In that case, why won’t you ban them? And they didn’t have an
answer.”
I joke that by now computers must all know who he is and are
probably listening in on this conversation and swapping notes.
“I’m just trying to prevent the machines from making a terrible
mistake,” he says.
A small part of me is paranoid that someo ne – or some artiﬁcial
someone – might spy on me through the camera in my
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
computer. Was it just a coincidence that I started getting all
those grotesque adverts for ear-cleaning devices after using a
cotton bud in what I thought was the privacy of my own home
oce? I can’t believe I’m telling Russell this, but I keep a sticky
note over the lens when I’m not on a video call. Rather to my
surprise, he says, “I think that’s a good idea.” People who know
more about computer security than he does say the same
apparently.
I wonder what he thinks of Elon Musk’s hopes to build a brain-
machine interface or “neural lace”, inspired by Iain M Banks’s
Culture novels. “His solution to the existential risk is that we
actually merge with the machines,” he says. “If we all have to
have brain surgery just to survive, perhaps we made a mistake
somewhere along the line.”
How worried is he that his children or any future grandchildren
will face a dystopian future with AI? “It doesn’t feel like a
visceral fear. It feels like climate change.” But in the worst-case
scenario AI would be terminal for our species, whereas with
climate change we could probably cling on in the last temperate
corners of the world. So AI could be worse than global warming?
“In the worst case, yes. We have to follow our reasoning where it
leads us. And if the machines really are more intelligent than us
and we’ve made a mistake and set them  up to pursue objectives
that end up having these disastrous side eects, we would have
no more power than chess players have when they are playing
against the best chess programmes.”PREVIOUS ARTICLE  NEXT ARTICLEMagazine
The great thing about the chess app on my phone is that I can
take a move back when I make a mistake. “Oh, you play like
that?” he says, raising an eyebrow. On the way over on the plane
he was playing a rather more formidable chess programme. “It
doesn’t let you take any moves back.”
AI: the next 10 years
By Monique Rivalland
Health
The race is on to transform healthcare with AI and the market is
estimated to be worth £120 billion by 2028. So what can we
expect? Artiﬁcially intelligent equipment will detect and
Sony’s Aibo, a robotic puppy
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
diagnose disease earlier and more accurately. New drug
discovery will be sped up. An AI developed by Google Health
can already identify signs of diabetic retinopathy from eye
scans with 90 per cent accuracy. At hospitals and care homes
basic nursing tasks could be carried out by AI assistants. The
ﬁeld of neuroprosthetics, which develops brain implants,
robotic limbs and cyborg devices, will help us overcome
cognitive and physical limitations. This month BioNTech,
maker of the Pﬁzer Covid-19 vaccine, launched an “early
warning system” with London-based AI ﬁrm InstaDeep to
detect new  variants of the coronavirus before they spread.
Pets
Japan is leading the way in AI pets. Sony’s Aibo, which costs
£2,127, is a robotic puppy. Aibo will respond to comm ands as
well as read human emotions and distinguish between family
members. When tired, Aibo returns to his charging station.
Towards the end of 2020, almost a year into the pandemic, local
government in New York started oering AI-powered furry
tabby cats from robotics company Joy For All to care homes and
older people in social isolation. China’s Unitree wants to make
its four-legged robots, currently £1,980, as aordable as phones.
It won’t be long before AI companions need not resemble
traditional pets for humans to warm to them.  Spot The Dog is
not exactly a pet but a robotic canine that is so agile it is used to
explore remote environments too dangerous or extreme for
humans. Made by Boston Dynamics and sold for £55,312, it
could assist with mining, police searches and space exploration. PREVIOUS ARTICLE  NEXT ARTICLEMagazine
Weapons
Robots and drones could carry out perilous tasks such as bomb
disposal, but the biggest change to warfare will come in the
shape of artiﬁcially intelligent killing machines. In November
2020, Israel assassinated Iran’s top nuclear scientist using a
high-tech, computer-powered sharpshooter with multiple
camera eyes, capable of ﬁring 600 rounds a minute.
Transport
There are more than ten unicorn start-ups – that’s companies
valued at $1 billion – vying for leadership in the autonomous
vehicle industry. They’re in China, America, Britain and Canada
and include personal transport as well as trucks and haulage.
This month the MK Dons (Milton Keynes) football team have
been trialling driverless cars called Fetch to take them to and
from training. Self-driving cars are supposed to be safer and
more ecient than human drivers and are expected on British
roads later this year. The government has announced that cars
ﬁtted with automatic lane-keeping systems will be permitted to
drive at up to 37mph in a single lane without the driver
interacting with it.
Education
The main beneﬁ t here is that AI will better tailor education to
students’ need s. Virtual tutors will assist human teachers in the
classroom, oering support to students by giving instant
answers to commonly asked questions. Facial-recognition techPREVIOUS ARTICLE  NEXT ARTICLEMagazine
could analyse the emotions of children to determine who’s
struggling or bored and better personalise their experience.
Communication
Microsoft and Skype already have a voice translator that can
translate between 11 languages, including Chinese, English,
French, Japanese, Russian and Spanish. This is likely to advance
quickly to real-time translation of hundreds of languages,
taking us a step closer to universal conversation. Google is
working on an AI assistant that can complete simple phone-
based tasks such as calling your doctor to make an
appointment. No more waiting on hold.
Media
Journalists, beware. Simple or factual news will increasingly be
written by algorithms. It has started: The Washington Post’s “AI
Writer” wrote more than 850 stories during the Rio Olympics in
2016; Bloomberg uses AI tech to relay complex data, and
Associated Press uses natural language AI to produce 3,700
earnings reports a year.
Sources Forbes, McKinsey, Statists, IHS Markit, Pega
Related articles
We should be wary of the tasks we offload to artificial
intelligenceCOMMENT | CRISTIN LEACH
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
PAID PROMOTIONAL LINKS
Motley Fool
Issues Rare
“All In” Buy
Alert
The Motley
Fool
Your team of
ﬁnancial
advisors now
includes
robo-
advisors.
NerdWallet
Billionaires
Buy 1 Mil
shares Of
Micro-Cap
Behind the
Markets
Trying to
ﬁnd the next
tech
unicorn?
OurCrowd
6 Credit
Cards You
Should Not
Ignore If You
Have
Excellent
Credit
NerdWalletPromoted by  DianomiDecember 11 2021,  12.01am GMT
Cristín LeachI once attended a writing workshop with the Irish author Lia Mills. Several
lessons were learnt about writing, but one in...
AI software helps mathematicians pinpoint patterns
December 02 2021,  12.01am GMT
Tom Whipple, Science Editor“A mathematician,” the Hungarian number theorist Alfred Renyi said, “is a
machine for turning coee into theorems”. The...
How to save the world — by a man who might actually
know
September 29 2021,  12.01am BST
Hugo RifkindINTERVIEW
Mo Gawdat glimpsed the apocalypse in a robot arm. Or rather, in a bunch of
robot arms, all being developed together. An arm...
C t(15)PREVIOUS ARTICLE  NEXT ARTICLEMagazine
Comments are subject to our community guidelines, which can be viewed here.Comments(15)
Stuart Russell
SAdd to the conversation...
Sort by Recommended
H Helios·9 HOURS AGO
'AI' has become a buzzword used far too liberally. Most so called
'AI' has no 'intelligence' but is merely a sophisticate dumb
algorithm.
ReplyRecommend (6)
S Seldon·9 HOURS AGO
Yup. Linear regression about 90% of the time when used in
corporate  speak.
ReplyRecommend (5)
1 replyL LarryC·2 HOURS AGO
Machine Learning is a much more common term these days
than AI: perhaps a concession that the processes aren't that
smart, but they are eective. It's quite sobering to consider
that the approach of deﬁning an objective (but not necessarily
the means of obtaining it) is resulting in near-human le...See
more
ReplyRecommend
PREVIOUS ARTICLE  NEXT ARTICLEMagazine
1reply
M Mike S·9 HOURS AGO
Fascinating. Thank you for this article
ReplyRecommend (5)
G Glasgow Kiss·9 HOURS AGO
The Singularity has been reached, we just don’t know it yet.
ReplyRecommend (1)
M Mospe·7 HOURS AGO
Amongst human beings it is not the most intelligent who end up
being in charge, witness are politicians. We need to concentrate on
producing biddable AI Like Musk is producing a rather weedy robot
ReplyRecommend
M MH18·1 HOUR AGO
AI possibilities are constantly referred to as capabilities.  You only
have to try to  get a chatbot to understand and answer a relatively
simple question to realise how absolutely bloody useless they are.
Referring you to a human is not an example AI solving a problem.
ReplyRecommend
H Howard Jones·2 HOURS AGO
Ever since I was a child (now rather a long time ago) we have been
told that technology would liberate us from drudgery and we'd all PREVIOUS ARTICLE  NEXT ARTICLEMagazine
live lives of leisure.  It hasn't quite worked out that way, so far at
least, but now it begins to look as if AI might actually achieve this.
What none of these mode...See more
ReplyRecommend (1)
2 2022RAM·7 HOURS AGO
Has God been using AI to control the universe for eons? God
knows.
ReplyRecommend
J Johnny H·4 HOURS AGO
I'd love to know what he thinks about Isaac Asimov's 3 Laws of
Robotics.
1. A robot shall not harm a human, or by inaction allow a human to
come to harm.
2. A robot shall obey any instruction given to it by a human.
3. A robot shall avoid actions or situations that could cause it to
come to harm its...See more(Edited)
ReplyRecommend
B Bat-Ori·8 HOURS AGO
'Alexa, wash all my blue shirts except the short-sleeved ones and
the one I wore yesterday'.  'Sorry, I don't know that song'.  Are
battleﬁeld robots more competent than that?
ReplyRecommend
M Mike O'Neill·1 HOUR AGO PREVIOUS ARTICLE  NEXT ARTICLEMagazine
We won't end up with millions of therapists. An AI therapist will be
much better at the job. As for artists…
ReplyRecommend
FeedbackView more comments
PREVIOUS ARTICLE  NEXT ARTICLE
About us Contact us
Help The Times Editorial Complaint
The Sunday Times Editorial Complaints Place an announcement
Classiﬁed advertising Display advertising
The Times corrections The Sunday Times corrections
CareersGET IN TOUCH
The Times e-paper
Times Currency ServicesMORE FROM THE TIMES AND THE SUNDAY TIMES BACK TO TOPMagazine
PREVIOUS ARTICLE  NEXT ARTICLETimes Print Gallery
Times Crossword Club
Times+
Times Expert Traveller
Schools Guide
Best Places to Live
Sportswomen of the Year Awards
Podcasts
Privacy & cookie policy Licensing
Cookie settings Site map
Topics Commissioning terms
Terms and conditions Do not sell my personal information© Times Newspapers Limited 2022.
Registered in England No. 894646.
Registered oﬃce: 1 London Bridge Street, SE1 9GF.
Magazine


Understanding Recommenders..

[![Luke Thorburn](https://miro.medium.com/v2/resize:fill:48:48/1*pgY6plckE3fmSouBjppMHg.jpeg)](/@lukethorburn?source=collection_home---------0----------------------------)[Luke Thorburn](/@lukethorburn?source=collection_home---------0----------------------------)[·May 7](/understanding-recommenders/making-amplification-measurable-2be548e5986c?source=collection_home---------0----------------------------)[Making Amplification Measurable
-------------------------------

— Luke Thorburn, Jonathan Stray, Priyanjana Bengani The term “amplification” is ubiquitous in discussions of recommender systems and even legislation, but is not well-defined. We have previously argued that the term is too ambiguous to be used in law, and that the three most common interpretations — comparison with a…](/understanding-recommenders/making-amplification-measurable-2be548e5986c?source=collection_home---------0----------------------------)[Recommender Systems](/tag/recommender-systems?source=collection_home--------------------recommender_systems-----------------)[9 min read](/understanding-recommenders/making-amplification-measurable-2be548e5986c?source=collection_home---------0----------------------------)[![Making Amplification Measurable](https://miro.medium.com/v2/resize:fill:224:224/1*zxJA84Jokc5tX3JwAy_snQ.png)![Making Amplification Measurable](https://miro.medium.com/v2/resize:fill:160:112/1*zxJA84Jokc5tX3JwAy_snQ.png)](/understanding-recommenders/making-amplification-measurable-2be548e5986c?source=collection_home---------0----------------------------)[Recommender Systems](/tag/recommender-systems?source=collection_home--------------------recommender_systems-----------------)[9 min read](/understanding-recommenders/making-amplification-measurable-2be548e5986c?source=collection_home---------0----------------------------)