Suffering-focused AI safety: Why “fail-safe” measures might be our top intervention

Report FRI-16-1
Suﬀering-focused AI safety: Why “fail-safe”
measures might be our top intervention
Lukas Gloor
Foundational Research Institute
lukas.gloor@foundational-research.org
June 2016
Abstract
AI-safety eﬀorts focused on suﬀering reduction should place particular emphasis on
avoiding risks of astronomical disvalue. Among the cases where uncontrolled AI de-
stroys humanity, outcomes might still diﬀer enormously in the amounts of suﬀering
produced. Rather than concentrating all our eﬀorts on a speciﬁc future we would like
to bring about, we should identify futures we least want to bring about and work
on ways to steer AI trajectories around these. In particular, a “fail-safe”1approach
to AI safety is especially promising because avoiding very bad outcomes might be
much easier than making sure we get everything right. This is also a neglected cause
despite there being a broad consensus among diﬀerent moral views that avoiding the
creation of vast amounts of suﬀering in our future is an ethical priority.
Contents
1 Introduction 2
2 How AI outcomes might contain suﬀering 2
2.1 Type I: Controlled AI gone bad . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Type II: “Near misses” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.3 Type III: Uncontrolled AI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3 Suﬀering-focused AI safety: Some proposals 5
3.1 Inﬂuencing outcomes with controlled AI . . . . . . . . . . . . . . . . . . . . . . 5
3.2 Diﬀerential progress in AI safety . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.3 AI safety where it is most needed . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.4 Graceful fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4 Concluding thoughts 7
Acknowledgements 8
References 8
1“Fail-safe” in the sense that if control fails, at least the AI causes less suﬀering than would have been the
case without fail-safe measures.
Suﬀering-focused AI safety −Foundational Research Institute
1 Introduction
Classical AI safety looks very diﬃcult. In or-
der to ensure a truly “utopian” outcome, the
following things need to happen:
1. The AI that gains a decisive strategic ad-
vantage over the competition needs to be
built by the right group of people.
2. These people would need to have ﬁgured
out how to program favorable values into
an AI (or program the AI to learn these
values).
3. They would also need to have come up
with a satisfying description of what
these values are, either directly, or in-
directly with the help of a suitable ex-
trapolation procedure (e.g. Muehlhauser
& Helm, 2012).
4. Finally, the creators of the ﬁrst superin-
telligence might need to get decision the-
ory right (see Soares & Fallenstein, 2015)
and safeguard the AI from a lot of hard-
to-anticipate failure modes.
Succeeding with all of this together is the only
safe way of bringing about a ﬂourishing utopia
mostly free of suﬀering. (There might be other
paths to utopia, provided enough luck.) But
what about outcomes where something goes
wrong? It is important to consider that things
can go wrong to very diﬀerent degrees. For
value systems that place primary importance
on the prevention of suﬀering, this aspect is
crucial: the best way to avoid bad-case sce-
narios speciﬁcally may not be to try and get
everything right. Instead, it makes sense to fo-
cus on the worst outcomes (in terms of the
suﬀering they would contain) and on tractable
methods to avert them. As others are trying to
shoot for a best-case outcome (and hopefully
they will succeed!), it is important that some
people also take care of addressing the biggest
risks. This perspective to AI safety is espe-
cially promising both because it is currently
neglected and because it is easier to avoid a
subset of outcomes rather than to shoot foronehighlyspeciﬁcoutcome.Finally,itissome-
thing that people with many diﬀerent value
systems could get behind.
2 How AI outcomes might contain
suﬀering
Fromasuﬀering-focusedperspective,themain
reason to be concerned about the risks from
artiﬁcial intelligence is not the possibility of
human extinction or the corresponding failure
tobuildaﬂourishing,intergalacticcivilization.
Rather, it is the thought of misaligned or “ill-
aligned” AI as a powerful but morally indif-
ferent optimization process which, in the pur-
suit of its goals, may transform galactic re-
sources into (among other things) suﬀering.
Either like the suﬀering we see on Earth to-
day, or by bringing about optimized struc-
tures that possibly also contain novel forms
of suﬀering. The things a superintelligent AI
would build to pursue its goals might include
a ﬂeet of “worker bots”, factories, supercom-
puters, space colonization machinery, etc. It is
possible that all of these end up being built
in a way that does not permit suﬀering. How-
ever, given that evolution has produced plenty
of sentient minds, this weakly suggests that
some of the easiest ways to implement mind
architectures do come with the capacity for
suﬀering. In the absence of an explicit anti-
suﬀering preference, even the slightest bene-
ﬁt to the AI’s objectives would lead to the
instantiation of suﬀering minds. What makes
this especially worrying is that the stakes in-
volved will be huge: Space colonization is an
attractive subgoal for almost any powerful op-
timization process, as it leads to control over
the largest amount of resources (Omohundro,
2008). Even if only a small portion of these
resources were used for purposes that include
suﬀering,theresultingdisvaluewouldsadlybe
astronomical.
To ﬁnd the best interventions to prevent suf-
fering, it is crucial to study how AI outcomes
2
Suﬀering-focused AI safety −Foundational Research Institute
diﬀerintheirexpectedamountofsuﬀeringand
whether there are ways to favorably aﬀect the
likely outcomes. This paper aims to provide
an overview on “bad-case scenarios” in the AI
context and some general suggestions for how
they could be avoided. Future work will in-
clude checking fundamental assumptions and
looking at the most promising proposals in
more detail.
2.1 Type I: Controlled AI gone bad
In these scenarios, the team that builds the
ﬁrst superintelligence knew what it was doing
and managed to successfully shape the result-
ing AI’s goals exactly the way they wanted it
togo.Unfortunately,thegoalsarenotwhatwe
would wish them to be and thus lead to bad
or very bad consequences. Things that could
go wrong include:
a) Anthropocentrism: perhaps the result-
ing AI would not care about the suﬀer-
ing of “weird” and/or voiceless nonhu-
man minds such as suﬀering subroutines
or animal minds in ancestor simulations.
b) Retributivism: perhaps the AI gets built
by people who want to punish members
of an outgroup (e.g. religious fundamen-
talists punishing sinners).
c) Uncooperative: perhaps the AI’s goal
is something like classical utilitarianism
(or any other “monotone” maximizing
function) with no additional regards for
cooperation with other value systems.
Even though such an AI would all else
equal prefer to not create suﬀering, it
seems possible that the anti-suﬀering
concern would in practice be overridden
by opportunity costs: if happiness sim-
ulations contain much more utility for
the AI than the disutility produced bythe accidental suﬀering created in the
buildup, then such an AI would behave
similarly “recklessly” as e.g. a paperclip-
maximizing AI.2Another concern lies in
how strongly the AI would value robust-
ness to deterioration.
d) Libertarianism regarding computations:
perhaps the creators of the ﬁrst super-
intelligence instruct the AI to give every
human alive at the time of the singular-
ity control of a planet or galaxy, with
no additional rules to govern what goes
onwithinthoseterritories.Someofthese
human rulers are curious and amused by
seeing others hurt, or worse, might be
psychopaths.
2.2 Type II: “Near misses”
In this scenario, the ﬁrst superintelligence is
built by people who were aware of the risks,
and who tried to get things right. Unfortu-
nately, mistakes happened and the resulting
outcome with “nearly-controlled AI” is bad or
very bad.
a) Miserable creatures: perhaps the AI’s
goal function includes terms that at-
tempt to specify sentient or human-like
beings and conditions that are meant to
be good for these beings. However, be-
cause of programming mistakes, unan-
ticipated loopholes or side-eﬀects, the
conditions speciﬁed actually turn out to
be bad for these beings. Worse still, the
AI has a maximizing function and wants
to ﬁll as many regions of the universe as
possible with these poor creatures.
b) Black swans: perhaps the AI cares about
sentientorhuman-likemindsin“proper”
ways, but has bad priors, ontology, de-
cision theory, or other fundamental con-
2An AI that values happiness simulations but takes cooperation with suﬀering reducers into account would in
expectation still create some amount of suﬀering, but depending on just how costly it is to use more “suﬀering-
proof” algorithms in the colonization process, or to do fewer or less ﬁne-grained ancestor simulations, it might
be possible to cut down on most of the instrumentally produced suﬀering at a cost that isn’t very high (e.g. not
higher than 15% of total happiness simulations produced).
3
Suﬀering-focused AI safety −Foundational Research Institute
stituents that would make it act in un-
fortunate and unpredictable ways.
2.3 Type III: Uncontrolled AI
Uncontrolled AI is stipulated to have goals
that were never intended by its creators, or
at least were not intended to take over control
in the form of a singleton (Bostrom, 2006).
Unless the AI in question has a neuromor-
phic design, it is unlikely that its goals would
(directly) have something to do with sentient
beings. However, even non-neuromorphic un-
controlled AI might instrumentally instanti-
ate suﬀering minds in the process of achieving
its goal, as a side-eﬀect of the computations
it performs. The following presents a list of
ways how uncontrolled AI might create vast
amounts of suﬀering:
•Suﬀering subroutines: For tasks like in-
venting and developing advanced tech-
nologies, as well as for coordinating its
expansion into space for resource accu-
mulation, the AI would have to rely on
a ﬂeet of “robot workers” or “robot scien-
tists” of various kinds (which, of course,
might work and function very unlike hu-
man workers or scientists). It is possible
that these processes would be capable of
suﬀering (Tomasik, 2014), although it is
unclear whether would perform at their
best if they (occasionally) suﬀer. What is
clear is that if they do suﬀer under use-
ful circumstances, an AI that is not ex-
plicitly aligned with compassionate goals
would not have any qualms about instan-
tiating astronomical numbers of them.
•Ancestor simulations (Bostrom, 2003;
Bostrom, 2014, Ch.8): in order to gather
information relevant to its goals, e.g. for
improving its understanding of human
psychology or sociology (Bostrom, 2014,pp. 125-26) or for studying the density of
aliensintheuniverseandwhattheirlikely
values are, a superintelligence might sim-
ulatemanyrunsofDarwinianevolutionin
planet-sized supercomputers (Sandberg,
1999). It is possible that these simulations
would be ﬁne-grained enough to contain
sentient minds.
•Warfare: it may be that the density of life
in the universe is high enough for coloniz-
ing AIs to eventually encounter one an-
other. If so, uncontrolled AI might end
up clashing with other superintelligences,
including AIs that build expanding civ-
ilizations of happy sentient beings. Per-
haps they could agree to a compromise,
but such an encounter could also lead to
ﬁghting over resources and warfare at the
points of contact.
It is interesting to note that none of the afore-
mentioned scenarios involve large quantities of
suﬀering in the types of structures that the AI
directlyoptimizesfor.Itseemslikelythatwhat
makes uncontrolled AI bad is what it builds
for instrumental reasons, including coloniza-
tion machinery, science simulations, and other
strategic computations. This suggests that op-
portunity costs are relevant: If an uncontrolled
AI’s goal function does not have diminishing
returnsonresources(liketheinfamous“paper-
clipmaximizer”),thentheroomforinstrumen-
tally important computations might be small.
By contrast, an AI with goals that are easy to
fulﬁll3– e.g. a “paperclip protector” that only
cares about protecting a single paperclip, or
an AI that only cares about its own reward
signal – would have much greater room pur-
suing instrumentally valuable computations.
This line of thought suggests that outcomes
from uncontrolled AI should be steered away
from “paperclip-protector types.”4
3Thanks to Carl Shulman for bringing this idea to our attention.
4However, it may be that paperclip protectors are easier to compromise with, and thus more likely to also
pursue maximization of sorts in exchange for beneﬁts from “paperclip-maximizer types.”
4
Suﬀering-focused AI safety −Foundational Research Institute
3 Suﬀering-focused AI safety: Some
proposals
Bad-case scenarios can be avoided in several
ways. Using the typology from above, we can
distinguish three classes of interventions:
•Inﬂuencing outcomes with con-
trolled AI: inﬂuence the values imple-
mented in outcomes with controlled AI
either through value spreading or by sup-
portingtheAIsafetyprojectwiththebest
values
The following can be labelled “fail-safe” mea-
sures:
•Targeted AI safety:
a) Diﬀerential progress: push classical
AI safety diﬀerentially in ways that
best insulates it from the worst
“near misses”
b) AI safety where it is most needed:
identify the paradigm in general AI
that would create the worst out-
comes in the event of a control
failure, and work towards identify-
ing (general) control or shut-down
mechanisms in those areas
•Graceful fails : research ways to make AI
fail in a “benign” way conditional on it
failing(“It might be the end of the world,
but it could have been (much) worse”)
The following sections present some prelimi-
nary ideas for the type of interventions that
could be done in those areas.
3.1 Inﬂuencing outcomes with
controlled AI
This class of interventions is primarily tar-
geted at “controlled AI gone bad” outcomes. It
is something FRI and others in the EA com-
munity have already thought about a great
deal.
•Improve values: raise awareness of anti-
speciesism, anti-substratism, concern forweird minds or suﬀering-focused ethics in
general.
•Cooperation: promote the idea that the
gains from cooperation make it impor-
tant for diﬀerent value-systems to work
together.
•Cooperative AI-safety eﬀorts: by funding
the AI-safety project with the best val-
ues or the best approach to cooperation,
we make it more likely that controlled AI
won’t create vast amounts of suﬀering.
3.2 Diﬀerential progress in AI safety
This intervention is targeted at the worst out-
comes in the “near misses” category. The goal
is to ﬁgure out solutions to the worst prob-
lems ﬁrst, such that any additional progress in
AI safety is unlikely to lead to “near-misses,”
which may be worse than outcomes with un-
controlled AI in terms of suﬀering produced.
This intervention seems comparatively more
neglected than interventions in the category
above.
•Secure value implementation: Look into
the pros and cons of various approaches to
specifying an AI’s values and work on the
one least likely to lead to really bad out-
comes. Questions to study may include:
What’s more likely to go wrong –
something like indirect normativity,
or an explicit speciﬁcation of a goal
function?
What are other promising approaches
to AI safety, and how bad would the
outcome be if they fail according to
their weak points?
Would corrigibility reduce risks of as-
tronomical suﬀering?
•Foundational research: Which unantici-
pated failure modes could there be for
controlled AI? Once they are identiﬁed,
how can we prevent them?
5
Suﬀering-focused AI safety −Foundational Research Institute
3.3 AI safety where it is most needed
This intervention is targeted at preventing the
worstoutcomesinvolvinguncontrolledAI.Not
all uncontrolled AIs are expected to behave
equally; there are diﬀerent paradigms in AI re-
search, i.e. diﬀerent basic architectures for at-
tempting to build smarter-than-human intelli-
gence. Examples include reinforcement learn-
ing (Sutton & Barto, 1998), proof-based seed
AI, and neuromorphic AIs (Hasler & Marr,
2013). Classical AI safety is necessarily con-
cerned with the tradeoﬀ between success prob-
ability and “controllability;” if a particular ap-
proach to building AI is likely to lead to su-
perintelligence, but in a way where it is hard
to control the values of the resulting agent,
then this poses a likely dead end. By contrast,
a suﬀering-focused “fail-safe” approach does
not have to worry about controllability to the
same extent, as long as there’s enoughcontrol-
labilitytopredictablyavoidoutcomeswiththe
most suﬀering. This suggests that, conditional
on uncontrolled AI being worse than (nearly-
)controlled AI in expectation, suﬀering reduc-
ers can be less constrained by issues with con-
trollability, and can thus focus more on ap-
plying AI safety to the paradigm/architecture
with the highest probability of bringing about
superintelligence.
In addition, we could try to identify diﬀer-
ences among the typesof uncontrolled AI each
AI-paradigm would produce, in order to then
focusonAIsafetyinthedomainwherefailures
of control result in the worst outcomes. Sup-
pose for instance that one way of building AI
appears to show promising results and rapid
progress,yetcontrolseemshardandlikelyfail-
ure modes are particularly worrying. In this
case, we should consider putting eﬀorts into
AI safety applied to that particular domain.
Speciﬁc examples of this more general idea in-
clude:
•AI safety for machine learners: WithDeepMind’s recent success, it seems not
unlikely that the ﬁrst superintelligent AI
may be all reinforcement learning as the
top control structure (as opposed to GO-
FAI, theorem proving or logic-based AI).
IncontrasttoAIarchitectureswheregoals
refer to the state of the world (Hibbard,
2011) rather than to internal or observa-
tional signals, pure reinforcement learners
maybeatahigherriskofwireheading,i.e.
of hacking their input signals to continu-
ously attain the maximum reward. If the
wireheading AI is smart enough to plan
into the future, it might patiently avoid
doing so and develop plans for takeover
ﬁrst: As Bostrom notes in Superintel-
ligence(Bostrom, 2014, 122–23), smart
wireheaders would still want to colonize
other galaxies for option value, protec-
tion, research, etc. However, unlike paper-
clip maximizers, wireheaders would more
generally function like “paperclip protec-
tors” in that they would have almost no
opportunity costs (Ring & Orseau, 2011).
If the observation that lower opportunity
costs make outcomes with uncontrolled
AI worse is accurate (see the discussion
under “Type III: Uncontrolled AI), this
wouldmakeAIsafetyappliedtoreinforce-
ment learning architectures particularly
promising. Speciﬁc interventions could in-
clude work on Interruptibility (Armstrong
& Orseau, 2016), but also methods of
value-loading (examples here and here).
•AI safety for “em-ﬁrst” scenarios: Condi-
tional on whole brain emulation (Sand-
berg & Bostrom, 2008; Hanson, 2016) be-
coming technologically feasible before the
advent of de-novo AI, de-novo AI as even-
tually developed will more likely exhibit
a neuromorphic design.5This would make
it more likely that the values are human-
like, which also comes with the possi-
5If only because neuroscience research would be very advanced in such a world.
6
Suﬀering-focused AI safety −Foundational Research Institute
bility of “bad values” or “near misses.”
AI safety applied to “em-ﬁrst” scenarios
couldthusbeapromisingintervention(al-
though with early arrival timelines being
longer,itmaynotbefavoredbyhastecon-
siderations).
Caveat: Pursuing any of the interventions
above would require that we make sure to not
speed up progress in AI development – both
in general and in “unsafe” or “hard-to-control”
approaches speciﬁc to those interventions.
3.4 Graceful fails
This class of interventions is targeted at pre-
venting the worst outcomes involving “nearly-
controlled” or uncontrolled AI. Instead of try-
ing to make fails less likely, the idea is to come
up with safety nets or mechanisms such that,
if control fails, the outcome will be as good
as it gets under the circumstances. Graceful
fails are easiest to bring about in the domain
of near misses, as the researchers working on
AI safety are aware of the risks and thus open
to the idea of safety nets.
•Backup goal functions: Research ways to
implement multi-layered goal functions,
with a “backup goal” that kicks in if the
implementation of the top layer does not
fulﬁll certain safety criteria. The backup
would be a simpler, less ambitious goal
that is less likely to result in bad out-
comes. Diﬃculties would lie in selecting
thesafetycriteriainwaysthatpeoplewith
diﬀerent values could all agree on, and
in making sure that the backup goal gets
triggeredunderthecorrectcircumstances.
InthecontextofAIresearchthatisnotguided
by safety precautions, graceful fails propos-
als seem harder to implement. The challenge
is to get an AI project that might result in
uncontrolled AI to care about whatever pro-
posal there is to implement. Perhaps there are
proposals to consider in rather early stages of
AI development where researchers think hardtakeoﬀs are very unlikely. Should the AI de-
sign in question nevertheless undergo an intel-
ligenceexplosion,theremightbewaystomake
the result counterfactually less bad.
•Dummy goals: We could come up with a
goal function with the following proper-
ties:
Easy to program
Interesting (in the sense that it’s easy
to create problems for an AI with this
goal function to solve, such that re-
searchers can track their progress on
general intelligence)
Leads to decent outcome in the un-
likely case of hard takeoﬀ
If AI research and testing at early stages is
always conducted with goals of this type, an
unanticipated hard takeoﬀ would at least be
comparatively benign.
4 Concluding thoughts
Suﬀering-focused AI safety, and “fail-safe”
measures in particular, make up a large and
promising area of interventions for FRI (and
perhaps other organizations) to investigate
further. Advantages are that they are ne-
glected and often “less ambitious” than clas-
sical AI safety, which means they might end
up being more tractable. Moreover, “fail-safe”
measures are a promising project to focus on
because the interventions discussed are pos-
itive or at worst “unobjectionable” from the
perspectives of virtually all other value sys-
tems.
The main reason we had not explicitly
zoomed in on this category of interventions
earlier was that FRI was initially too focused
on answering the more general, complicated
question whether AI safety on the whole is
net positive for suﬀering reducers. Upon re-
ﬂection, this question may be less important.
AI safety is a very broad category where we
shouldexpectalotofroomfortargetedeﬀorts.
The general lesson to draw is that partitioning
7
Suﬀering-focused AI safety −Foundational Research Institute
broad categories can be an important step to-
wards making progress. More research into the
feasibilityofsuﬀering-focusedAIsafetycarries
high information value for this very reason: As
we look into more proposals and their prac-
tical feasibility, we might uncover more rele-
vant distinctions among AI outcomes, value-
transferproposals,orgeneralapproachestoAI
– a process which, in turn, will make it easier
to identify eﬀective interventions.
Acknowledgements
Brian Tomasik provided the central idea for
this paper and gave important feedback on
earlier versions. Thanks also go to Caspar
Oesterheld and Max Daniel for contributing
to the ideas and scenarios in the paper, and
to Simon Knutsson, Adrian Hutter, David Al-
thaus, Adrian Rohrheim, Andrei Pöhlmann,
Jan Leike and Kaj Sotala for valuable feed-
back and edits.
References
Armstrong, M. S., and L. Orseau. “Safely
Interruptible Agents.” Machine
Intelligence Research Institute 2016.
Bostrom, Nick. “Are We Living in a
Computer Simulation?.” The Philosophical
Quarterly 53.211 (2003): 243-255.
Bostrom, Nick. “What Is a Singleton.”
Linguistic and Philosophical Investigations
5.2 2006: 48-54.
Bostrom, Nick. Superintelligence: Paths,
Dangers, Strategies . Oxford: Oxford UP,
2014. Print.
Hanson, Robin. The Age of Em: Work, Love
and Life When Robots Rule the Earth.
Oxford: Oxford University Press, 2016.
Print.
Hasler, Jennifer, and Harry Bo Marr.
“Finding a Roadmap to Achieve Large
Neuromorphic Hardware Systems.”
Frontiers in Neuroscience 7 2013: 118.Hibbard, Bill. “Model-based Utility
Functions.” Journal of Artiﬁcial General
Intelligence 3.1 2011: 1-24.
Muehlhauser, Luke, and Louie Helm. 2012.
“Intelligence Explosion and Machine
Ethics.”In Singularity Hypotheses: A
Scientiﬁc and Philosophical Assessment .
Ed. Amnon Eden, Johnny Søraker, James
H. Moor, and Eric Steinhart. Berlin:
Springer, 2012. Print.
Omohundro, Stephen M. “The Basic AI
Drives.”AGI. Vol. 171. 2008.
Ring, Mark, and Laurent Orseau. “Delusion,
Survival, and Intelligent Agents.”
International Conference on Artiﬁcial
General Intelligence. Springer Berlin
Heidelberg, 2011.
Sandberg, Anders. “The Physics of
Information Processing Superobjects:
Daily Life Among the Jupiter Brains.”
Journal of Evolution and Technology 5.1
1999: 1-34.
Sandberg, Anders, and Nick Bostrom. Whole
Brain Emulation: A Roadmap . Future of
Humanity Institute, 2008. Print.
Soares, Nate, and Benja Fallenstein. “Toward
Idealized Decision Theory.” arXiv preprint
arXiv:1507.01986 2015.
Sutton, Richard S., and Andrew G. Barto.
Reinforcement Learning: An Introduction.
Vol. 1. No. 1. Cambridge: MIT press,
1998.
Tomasik, Brian. “Do Artiﬁcial
Reinforcement-Learning Agents Matter
Morally?.” arXiv preprint arXiv:1410.8233
2014.
8


Technical and social approaches to AI safety

I often divide solutions to the AI control problem into two parts: technical and social. I think any solution requires progress on both fronts, but strength on one front can compensate for a weakness on the other.

This view suggests that most of the value comes from having a “good” technical solution (quantification below), rather than a “perfect” solution or a “not terrible” solution. It also suggests that there is value from progress on both technical and social solutions, rather than justifying a focus on one or the other.

Technical solutions
-------------------

If we can build AI systems that are both efficient and safe, then we have addressed the most serious concerns about AI risk, regardless of how well or poorly society coordinates. If efficient AI is unsafe, then coordination may be needed to prevent people from using it.

1. **Efficient**: An efficient system is as cheap and effective, for the tasks to which it is applied, as any other system we can build using available technology. As AI becomes cheaper relative to human labor, the efficiency penalty for human involvement becomes larger. In the long run, efficient systems must require negligible human labor.
2. **Safe**:A safe system applies its efforts in the way that its users would want. In particular, if it makes money or acquires other resources or influence, the user retains effective control over most of those resources.

(Note that a system may involve both humans and machines.)

We can crudely quantify efficiency by the ratio between a system’s cost and the cost of the most efficient system that can do the same task. We can crudely quantify safety by the expected fraction of the system’s outputs which the user captures (with the remainder being applied towards some other ends).

A simple model is that a system’s output depends linearly on its efficiency, and that the output is split between the user and whatever other goals the system is implicitly or explicitly pursuing. For example, we could imagine AIs which pursue task-specific goals which are instrumentally valuable to but imperfectly aligned with its users’ goals. Then this system may accomplish the particular tasks to which it is applied, but may also divert some resources to pursuing those task-specific goals in ways that the user would not endorse.

Many futurists expect systems to have safety very close to 0 or 1, and that fits into this model just fine (you could still have intermediate degrees of safety, in light of uncertainty).

There may be many unsafe AI systems that don’t fit this simple model at all; I’ll have this model in mind throughout this post for concreteness, but I don’t think that we should too attached to conclusions that are very sensitive to the model (and I’ll try to mostly be agnostic).

Note that we can apply a very similar model to the more familiar principal-agent problem, in which a principal tries to induce an agent to act in the principal’s interests, and does so imperfectly. The principal can achieve greater efficiency by passing a larger share of their gains on to the agent, or by compromising and adopting values closer to the agents’. This example can help build some intuitions about the case of unsafe AI, but probably shouldn’t be taken too seriously.

Social solutions
----------------

If we can’t always build systems which are efficient and safe, then users must make a tradeoff between safety and efficiency. Users may opt for unsafe systems for a variety of reasons:

* They are maximizing their own returns, which may involve some compromise of safety.
* They share the values pursued by some efficient but unsafe system, or they value the expansion of intelligence or complexity for its own sake.
* They are making a trade with someone or something that shares the values of some efficient but unsafe system, and is willing to subsidize its use.

If the efficiency gains for unsafe systems are large, then resources invested unsafely will compound faster than resources invested safely. The result is that the (proportional) influence of the original human population will be gradually diminished. The quality of our technical solutions determines how large the gap is.

If we coordinate to ensure that no resources are invested unsafely, then we can address the AI control problem regardless of how big this gap is. The difficulty of this coordination problem, the amount of time available to solve it, the strength of the incentives to defect, the necessary level of success, and the consequences of failure, all depend on how good a technical solution to the control problem is available. As the efficiency gap becomes large, all of these parameters become extremely unfavorable.

If the productivity gap is relatively large, say an order of magnitude, then any investment in unsafe systems would very rapidly outpace society at large, and tight controls would be needed to prevent trouble. To put this in quantitative perspective, a 10x productivity boost in the current world would correspond to 30x–1,000x increase in proportional influence per decade (depending on whether the is 10x is compared to average growth or compared to rates of return on other capital investments). So even a very small fraction of unsafe investment could quickly become a major global influence, unless there was a strong social response to thwart the basic economic dynamic.

Examples of social solutions
----------------------------

This is a very broad category. For example, it includes:

* Coordination amongst AI researchers to preferentially develop and distribute safe AI.
* International political coordination to restrict deployment of unsafe AI, or to expropriate resources controlled by unsafe AI and sympathetic humans.
* AI is mostly developed by a single, especially safety-conscious, project. This project maintains a large enough lead over its rivals that it can afford to use inefficient but safe AI, and it manages to preserve this lead either indefinitely or until safe-and-efficient AI can be developed.

Of course there are no clear lines between these categories, or between them and other possible social solutions. I should emphasize that I am skeptical of most of these approaches, especially the kinds of draconian approaches that could deal with very large efficiency gaps.

I suspect that there will be at least some informal coordination amongst AI researchers, if and when the social benefits of coordination becomes clear. For example, I expect researchers and engineers to generally be happier to work for projects which they see as contributing to human welfare, and this will make life marginally harder for unsafe projects. As a result, the state of the art will be somewhat better for safe systems, it will be somewhat easier for people to apply safe systems to their problems, and so on. I expect this kind of informal coordination to cover small gaps in efficiency; for large efficiency gaps, I could see things going either way.

It would be surprising to me if international political coordination were strong enough to block the adoption of unsafe AI’s if they were many times more efficient than their safe counterparts (compare to nuclear disarmament or other international coordination problems we face today). If they were only modestly more efficient, then it would not necessarily require international coordination (just action by the most efficient jurisdictions), and regulation would only need to make very slightly harder for unsafe projects.

I am somewhat skeptical about the feasibility or desirability of world-takeover by an early AI project, though I think that there are exotic situations where it becomes plausible.

Upshot
======

I suspect that researchers working on the AI control problem should aim high. I think that there is a good chance that society can handle a 10% efficiency gap between safe and unsafe systems, but that it is pretty unlikely that it can handle a 10x gap.

So we capture more value by moving from 10% efficiency to 90% efficiency for safe systems, than by moving from 1% to 10% or from 0.001% to 1%. The relative benefits are mostly determined by the probability that social solutions will be good enough to handle gaps of each size.

On the flip side, I don’t think that we need to focus on reaching 100% efficiency either. 90% efficiency seems high enough that there is a good chance that social solutions can handle the gap.

I also think that there are meaningful benefits from progress on both technical and social solutions. I don’t think that social solutions are so robust that we don’t have to worry about having a very good technical solution (I’m surprised by how often I encounter this view!), nor do I think that technical solutions are so clearly bimodal that we don’t care about social solutions since we will be either doomed or have no problem at all.

Postscript: “Foom”
==================

I think that the breakdown above applies whether you expect AI development to proceed briskly or slowly.

If we expect AI development to go really fast (think days or weeks to go from “who cares” to “radically superhuman”), while the rest of the world continues to operate on its comparatively glacial pace, then it becomes more plausible that the first developers of AI will take over the world before anyone else catches up. This provides an easy social solution, and generally lowers the bar for technical solutions: maybe it’s OK if their AI is 10x less efficient than it would otherwise be, so that it takes it 10 weeks to take over the world rather than 1 week. This is only a problem if the competition is right on their tails.

I don’t think this situation is very likely. But if you do, then you should become more interested in going from 0.001% to 10% efficiency, since that may be all that you really need. You are also probably working on a somewhat different set of safety problems (and I suspect are overall more pessimistic).


Three impacts of machine intelligence

I think that the development of human level AI in my lifetime is quite plausible; I would give it more than a 1-in-4 chance. In this post I want to briefly discuss what I see as the most important impacts of AI. I think these impacts are the heavy hitters by a solid margin; each of them seems like a big deal, and I think there is a big gap to #4.

* **Growth will accelerate, probably very significantly.** Growth rates will likely rise by at least an order of magnitude, and probably further, until we run into severe resource constraints. Just as the last 200 years have experienced more change than 10,000 BCE to 0 BCE, we are likely to see periods of 4 years in the future that experience more change than the last 200.
* **Human wages will fall, probably very far.** When humans work, they will probably be improving other humans’ lives (for example, in domains where we intrinsically value service by humans) rather than by contributing to overall economic productivity. The great majority of humans will probably not work. Hopefully humans will remain relatively rich in absolute terms.
* **Human values won’t be the only thing shaping the future.** Today humans trying to influence the future are the only goal-oriented process shaping the trajectory of society. Automating decision-making provides the most serious opportunity yet for that to change. It may be the case that machines make decisions in service of human interests, that machines share human values, or that machines have other worthwhile values. But it may also be that machines use their influence to push society in directions we find uninteresting or less valuable.

My guess is that the first two impacts are relatively likely, that there is unlikely to be a strong enough regulatory response to prevent them, and that their net effects on human welfare will be significant and positive. The third impact is more speculative, probably negative, more likely to be prevented by coordination (whether political regulation, coordination by researchers, or something else), and also I think more important on a long-run humanitarian perspective.

None of these changes are likely to occur in discrete jumps. Growth has been accelerating for a long time. Human wages have stayed high for most of history, but I expect them to begin to fall (probably unequally) long before everyone becomes unemployable. Today we can already see the potential for firms to control resources and make goal-oriented decisions in a way that no individual human would, and I expect this potential to increase continuously with increasing automation.

Most of this discussion not particularly new. The first two ideas feature prominently in Robin Hanson’s speculation about an economy of human emulations (alongside many other claims); many of the points below I picked up from Carl Shulman; most of them are much older. I’m writing this post here because I want to collect these thoughts in one place, and I want to facilitate discussions that separate these impacts from each other and analyze them in a more meaningful way.

Growth will accelerate
======================

There are a number of reasons to suspect that automation will eventually lead to much faster growth. By “much faster growth” I mean growth, and especially intellectual progress, which is at least an order of magnitude faster than in the world of today.

I think that avoiding fast growth would involve solving an unprecedented coordination problem, and would involve large welfare losses for living people. I think this is very unlikely (compare to environmental issues today, which seem to have a lower bar for coordination, smaller welfare costs to avert, and clearer harms).

Automating tech progress leads to fast growth.
==============================================

The stereotyped story goes: “If algorithms+hardware to accomplish X get 50% cheaper with each year of human effort, then they’ll also (eventually) get 50% cheaper with each year of AI effort. But then it will only take 6 months to get another 50% cheaper, 3 months to get another 50% cheaper, and by the end of the year the rate of progress will be infinite.”

In reality things are very unlikely to be so simple, but the basic conclusion seems quite plausible. It also lines up with the predictions of naive economic models, on which constant returns to scale (with fixed tech) + endogenously driven technology — > infinite returns in finite time.

Of course the story breaks down as you run into diminishing returns to intellectual effort, and once “cheap” and “fast” diverge. But based on what we know now it looks like this breakdown should only occur very far past human level (this could be the subject for a post of its own, but it looks like a pretty solid prediction). So my money would be on a period of fast progress which ends only once society looks unrecognizably different.

One complaint with this picture is that technology already facilitates more tech progress, so we should be seeing this process underway already. But we do see accelerating growth (see the section on the historical record, below), except for a historically brief period of 50–75 years. So this seems like a weak objection.

Substituting capital for labor leads to fast growth.
====================================================

Even if we hold fixed the level of technology, automating human labor would lead to a decoupling of economic growth from human reproduction. Society could instead grow at the rate at which robots can be used to produce more robots, which seems to be much higher than the rate at which the human population grows, until we run into resource constraints (which would be substantially reduced by a reduced dependence on the biosphere).

Extrapolating the historical record suggests fast growth.
=========================================================

Over the course of history the proportional rate of growth has increased substantially, from more than 1,500 years per doubling around 10k years ago, to around 150 years per doubling in the 17th century, to around 15 years per doubling in the 20th century. A reasonable extrapolation of the pre-1950 data appears to suggest an asymptote with infinite population sometime in the 21st century. The last 50 years represent a notable departure from this trend, although history has seen similar periods of relative stagnation.

(See the graph to the left, produced from Bradford Delong’s data [here](http://www.j-bradford-delong.net/TCEH/1998_Draft/World_GDP/Estimating_World_GDP.html); data and full size [here](https://docs.google.com/spreadsheets/d/1Mn9q6_6tcYPknoe09gd0PMtQmyV2W7Z-yMGXZIJ86ow/edit?usp=sharing). The graph starts at a doubling time of 8000 years, and plateaus around 25 years, about 300 times faster. The average time required for growth rates to double is about 1.5 doublings, or 40 years at the current rate. Each subsequent doubling of the growth rate takes half as long.)

![]()I don’t think that extrapolating this trend forward results in particularly robust or even meaningful predictions, but I do think it says one thing: we shouldn’t be surprised by a future with much faster growth. The knee jerk response of “that seems weird” is inconsistent with the actual history (though it is a very reasonable intuitive reaction for someone who lived through the 2nd half of the 20th century). The more recent trend of slow growth may well continue, but I don’t think we should be surprised if this stagnation was a temporary departure from trend, comparable to previous departures in severity and scale.

Wages will fall
===============

I think this is the most robust of the three predictions. It seems very likely that eventually machines will be able to do all of the things that a human can do, as well as a human can do it. At that point, it would require significant coordination to prevent the broad adoption of machines as human replacements. Moreover, continuing to use human labor in this scenario seems socially undesirable; if we don’t have to work it seems crazy to make work for ourselves, and failing to make use of machine labor would involve even more significant sacrifices in welfare.

(Humans may still demand other humans’ labor in distinctively human roles. And there may be other reasons for money to move around between humans. But overall most new valuable stuff and valuable ideas in the world will be produced by machines. In the optimistic scenario, the main reason that this value will be flowing *to* humans, rather than merely *amongst* humans, will be because they own some of the machines.)

Historically humans have not been displaced by automation. I think this provides some evidence that in the near term automation will not displace humans, but in the long run it looks inevitable, since eventually machines really will be better at *everything.* Simple theories do suggest a regime where humans and automation are complementary, followed by a regime in which they are substitutes (as horses were once complementary with carriages, but were eventually completely replaced by automation). Robin Hanson illustrates in slides 30–32 [here](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&cd=2&cad=rja&uact=8&ved=0CCwQFjAB&url=http%3A%2F%2Fhanson.gmu.edu%2Fppt%2FEcon%2520of%2520AI%2520n%2520Nanotech.ppt&ei=2er3U77YC6K5igL1_ICQCQ&usg=AFQjCNGPwLHFp4yP3hIeKL7_nlZ81nCEhQ&bvm=bv.73612305%2Cd.cGE). So at some point I think we should expect a more substantive transition. The most likely path today seems to be a fall in wages for many classes of workers while driving up wages for those who are still complementary with automation (a group that will shrink until it is eventually empty). “[Humans need not apply](https://www.youtube.com/watch?v=7Pq-S557XQU)” has been making the rounds recently; and despite many quibbles I think it does a good job of making the point.

I don’t have too much to say on this point. I should emphasize that this isn’t a prediction about what will happen soon, just about what will have happened by the time that AI can actually do everything humans can do. I’m not aware of many serious objections to this (less interesting) claim.

Human values won’t be the only things shaping the future
========================================================

I’d like to explicitly flag this section as more speculative. I think AI opens up a new possibility, and that this is of particular interest for those concerned with the long-term trajectory of society. But unlike the other sections, this one rests on pretty speculative abstractions.

Many processes influence what the world will look like in 50 years. Most of those processes are not goal-directed, and push things in a somewhat random direction; an asteroid might hit earth and kill us all, the tectonic plates will shift, we’ll have more bottles in landfills because we’ll keep throwing bottles away, we’ll send some more radiation into space. One force stands out amongst all of these by systematically pushing in a particular direction: humans have desires for what the future looks like (amongst other desires), and they (sometimes) take actions to achieve desired outcomes. People want themselves and their children to be prosperous, and so they do whatever they think will achieve that. If people want a particular building to keep standing, they do whatever they think will keep it standing. As human capacities increase, these goal-oriented actions have tended to become more important compared to other processes, and I expect this trend to continue.

There are very few goal-directed forces shaping the future aside from human preferences. I think this is a great cause for optimism: if humans survive for the long run, I expect the world to look basically how we want it to look. As a human, I’m happy about that. I don’t mean “human preferences” in a narrow way; I think humans have a preference for a rich and diverse future, that we care about other life that exists or could have existed, and so on. But it’s easy to imagine processes pushing the world in directions that we don’t like, like the self-replicator that just wants to fill the universe with copies of itself.

We can see the feeble beginnings of competing forces in various human organizations. We can imagine an organization which pursues its goals for the future even if there are no humans who share those goals. We can imagine such an organization eventually shaping what other organizations and people exist, campaigning to change the law, developing new technologies, *etc.*, to make a world better suited to achieving its goals. At the moment this process is relatively well contained. It would be surprising (though not unthinkable) to find ourselves in a future where 30% of resources were controlled by PepsiCo, fulfilling a corporate mission which was completely uninteresting to humans. Instead PepsiCo remains at the mercy of human interests, by design and by necessity of its structure. After all, PepsiCo is just a bunch of humans working together, legally bound to behave in the interest of some other humans.

As automation improves the situation may change. Enterprises might be autonomously managed, pursuing values which are instrumentally useful to society but which we find intrinsically worthless (e.g. PepsiCo can create value for society even if its goal is merely maximizing its own profits). Perhaps society would ensure that PepsiCo’s values are to maximize profit only insofar as such profit-maximization is in the interest of humans. But that sounds complicated, and I wouldn’t make a confident prediction one way or the other. (I’m talking about firms because its an example we can already see around us, but I don’t mean to be down on firms or to suggest that automation will be most important in the context of firms.)

At the same time, it becomes increasingly difficult for humans to directly control what happens in a world where nearly all productive work, including management, investment, and the design of new machines, is being done by machines. We can imagine a scenario in which humans continue to make all goal-oriented decisions about the management of PepsiCo but are assisted by an increasingly elaborate network of prosthetics and assistants. But I think human management becomes increasingly implausible as the size of the world grows (imagine a minority of 7 billion humans trying to manage the equivalent of 7 trillion knowledge workers; then imagine 70 trillion), and as machines’ abilities to plan and decide outstrip humans’ by a widening margin. In this world, the AI’s that are left to do their own thing outnumber and outperform those which remain under close management of humans.

Moreover, I think most people don’t much care about whether resources are held by agents who share their long-term values, or machines with relatively alien values, and won’t do very much to prevent the emergence of autonomous interests with alien values. On top of that, I think that machine intelligences can make a plausible case that they deserve equal moral standing, that machines will be able to argue persuasively along these lines, and that an increasingly cosmopolitan society will be hesitant about taking drastic anti-machine measures (whether to prevent machines from having “anti-social” values, or reclaiming resources or denying rights to machines with such values).

Again, it would be possible to imagine a regulatory response to avoid this outcome. In this case the welfare losses of regulation would be much smaller than in either of the last two, and on a certain moral view the costs of no regulation might be much larger. So in addition to resting on more speculative abstractions, I think that this consequence is the one most likely to be subverted by coordination. It’s also the one that I feel most strongly about. I look forward to a world where no humans have to work, and I’m excited to see a radical speedup in technological progress. But I would be sad to see a future where our descendants were maximizing some uninteresting values we happened to give them because they were easily specified and instrumentally useful at the time.

*Originally published at* [*rationalaltruist.com*](http://rationalaltruist.com/2014/08/23/three-impacts-of-machine-intelligence/) *on August 23, 2014.*


Benefits and Risks of Artificial Intelligence

Discussions about Artificial Intelligence (AI) have jumped into the public eye over the past year, with several luminaries speaking publicly about the threat of AI to the future of humanity.

Over the last several decades, AI — computing methods for automated perception, learning, understanding, and reasoning — have become commonplace in our lives. We plan trips using GPS systems that rely on AI to cut through the complexity of millions of routes to find the best one to take. Our smartphones understand our speech, and Siri, Cortana, and Google Now are getting better at understanding our intentions. AI algorithms detect faces as we take pictures with our phones and recognize the faces of individual people when we post those pictures to Facebook. Internet search engines, such as Google and Bing, rely on a fabric of AI subsystems. On any day, AI provides hundreds of millions of people with search results, traffic predictions, and recommendations about books and movies. AI translates among languages in real time and speeds up the operation of our laptops by guessing what we’ll do next. Several companies, such as Google, BMW, and Tesla, are working on cars that can drive themselves — either with partial human oversight or entirely autonomously.
Beyond the influences in our daily lives, AI techniques are playing a major role in science and medicine. AI is at work in hospitals helping physicians understand which patients are at highest risk for complications, and AI algorithms are helping to find important needles in massive data haystacks. For example, AI methods have been employed recently to discover subtle interactions between medications that put patients at risk for serious side effects.

The growth of the effectiveness and ubiquity of AI methods has also stimulated thinking about the potential risks associated with advances of AI. Some comments raise the possibility of dystopian futures where AI systems become “superintelligent” and threaten the survival of humanity. It’s natural that new technologies may trigger exciting new capabilities and applications — and also generate new anxieties.

The mission of the[**Association for the Advancement of Artificial Intelligence**](http://aaai.org) is two-fold: to advance the science and technology of artificial intelligence and to promote its responsible use. The AAAI considers the potential risks of AI technology to be an important arena for investment, reflection, and activity.

One set of risks stems from programming errors in AI software. We are all familiar with errors in ordinary software. For example, apps on our smartphones sometimes crash. Major software projects, such as HealthCare.Gov, are sometimes riddled with bugs. Moving beyond nuisances and delays, some software errors have been linked to extremely costly outcomes and deaths. The study of the “verification” of the behavior of software systems is challenging and critical, and much progress has been made. However, the growing complexity of AI systems and their enlistment in high-stakes roles, such as controlling automobiles, surgical robots, and weapons systems, means that we must redouble our efforts in software quality.

There is reason for optimism. Many non-AI software systems have been developed and validated to achieve high degrees of quality assurance. For example, the software in autopilot systems and spacecraft systems is carefully tested and validated. Similar practices must be developed and applied to AI systems. One technical challenge is to guarantee that systems built automatically via statistical “machine learning” methods behave properly. Another challenge is to ensure good behavior when an AI system encounters unforeseen situations. Our automated vehicles, home robots, and intelligent cloud services must perform well even when they receive surprising or confusing inputs.

A second set of risks is cyberattacks: criminals and adversaries are continually attacking our computers with viruses and other forms of malware. AI algorithms are no different from other software in terms of their vulnerability to cyberattack. But because AI algorithms are being asked to make high-stakes decisions, such as driving cars and controlling robots, the impact of successful cyberattacks on AI systems could be much more devastating than attacks in the past. US Government funding agencies and corporations are supporting a wide range of cybersecurity research projects, and artificial intelligence techniques in themselves will provide novel methods for detecting and defending against cyberattacks. Before we put AI algorithms in control of high-stakes decisions, we must be much more confident that these systems can survive large scale cyberattacks.

A third set of risks echo the tale of the Sorcerer’s Apprentice. Suppose we tell a self-driving car to “get us to the airport as quickly as possible!” Would the autonomous driving system put the pedal to the metal and drive at 300 mph while running over pedestrians? Troubling scenarios of this form have appeared recently in the press. Other fears center on the prospect of out-of-control superintelligences that threaten the survival of humanity. All of these examples refer to cases where humans have failed to correctly instruct the AI algorithm in how it should behave.

This is not a new problem. An important aspect of any AI system that interacts with people is that it must reason about what people intend rather than carrying out commands in a literal manner. An AI system should not only act on a set of rules that it is instructed to obey — it must also analyze and understand whether the behavior that a human is requesting is likely to be judged as “normal” or “reasonable” by most people. It should also be continuously monitoring itself to detect abnormal internal behaviors, which might signal bugs, cyberattacks, or failures in its understanding of its actions. In addition to relying on internal mechanisms to ensure proper behavior, AI systems need to have the capability — and responsibility — of working with people to obtain feedback and guidance. They must know when to stop and “ask for directions” — and always be open for feedback.
Some of the most exciting opportunities ahead for AI bring together the complementary talents of people and computing systems. AI-enabled devices are allowing the blind to see, the deaf to hear, and the disabled and elderly to walk, run, and even dance. People working together with the Foldit online game were able to discover the structure of the virus that causes AIDS in only three weeks, a feat that neither people nor computers working alone could come close to matching. Other studies have shown how the massive space of galaxies can be explored hand-in-hand by people and machines, where the tireless AI astronomer understands when it needs to occasionally reach out and tap the expertise of human astronomers.
In reality, creating real-time control systems where control needs to shift rapidly and fluidly between people and AI algorithms is difficult. Some airline accidents occurred when pilots took over from the autopilots. The problem is that unless the human operator has been paying very close attention, he or she will lack a detailed understanding of the current situation.

AI doomsday scenarios belong more in the realm of science fiction than science fact. However, we still have a great deal of work to do to address the concerns and risks afoot with our growing reliance on AI systems. Each of the three important risks outlined above (programming errors, cyberattacks, “Sorcerer’s Apprentice”) is being addressed by current research, but greater efforts are needed.

We urge our colleagues in industry and academia to join us in identifying and studying these risks and in finding solutions to addressing them, and we call on government funding agencies and philanthropic initiatives to support this research. We urge the technology industry to devote even more attention to software quality and cybersecurity as we increasingly rely on AI in safety-critical functions. And we must not put AI algorithms in control of potentially-dangerous systems until we can provide a high degree of assurance that they will behave safely and properly.

[Tom Dietterich](http://www.eecs.oregonstate.edu/~tgd)
President, [AAAI](http://aaai.org)

[Eric Horvitz](http://research.microsoft.com/~horvitz)
Former President, AAAI and AAAI Strategic Planning Committee