AI governance research agenda

AI Governance:
A Research Agenda

Allan Dafoe
Centre for the Governance of AI
Future of Humanity Institute
University of Oxford

First draft July 2017
v1.0 August 27 2018

Visit fhi.ox.ac.uk/govaiagenda  to check for the most recent version of this paper.

AI Governance: A Research Agenda
Abstract 1
Artificialintelligence(AI)isapotentgeneralpurposetechnology.Futureprogresscouldbe
rapid,andexpertsexpectthatsuperhumancapabilitiesinstrategicdomainswillbeachieved
inthecomingdecades.Theopportunitiesaretremendous,includingadvancesinmedicine
andhealth,transportation,energy,education,science,economicgrowth,andenvironmental
sustainability.Therisks,however,arealsosubstantialandplausiblyposeextreme
governancechallenges.Theseincludelabordisplacement,inequality,anoligopolisticglobal
marketstructure,reinforcedtotalitarianism,shiftsandvolatilityinnationalpower,strategic
instability,andanAIracethatsacrificessafetyandothervalues.Theconsequencesare
plausiblyofamagnitudeandonatimescaletodwarfotherglobalconcerns.Leadersof
governmentsandfirmsareaskingforpolicyguidance,andyetscholarlyattentiontotheAI
revolutionremainsnegligible.ResearchisthusurgentlyneededontheAIgovernance
problem:theproblemofdevisingglobalnorms,policies,andinstitutionstobestensurethe
beneficial development and use of advanced AI.

Thisreportoutlinesanagendaforthisresearch,dividingthefieldintothreeresearchclusters.
Thefirstcluster,thetechnicallandscape,seekstounderstandthetechnicalinputs,
possibilities,andconstraintsforAI.Thesecondcluster,AIpolitics,focusesonthepolitical
dynamicsbetweenfirms,governments,publics,researchers,andotheractors.Thefinal
researchclusterofAIidealgovernanceenvisionswhatstructuresanddynamicswewould
ideally create to govern the transition to advanced artificial intelligence.

 Visit www.fhi.ox.ac.uk/govaiagenda  to check for the most recent version of this paper.
1Thisdocumentreceivedinputfrommanycontributors.ThetextwasprimarilywrittenbyAllanDafoe.Severalportions
receivedsubstantialinputfromotherindividuals,mostaffiliatedwiththeGovernanceofAIProgram,notedwhereforeach
portion.Thisworkdrawsfromthebodyofthinkingandinsightinthecommunityofscholarsandscientiststhinkingaboutthese
issues.Inparticular,forcomments,conversations,andrelatedworkthisdocumentbenefitsfromMilesBrundage,Owen
Cotton-Barratt,RichardDanzig,DanielDewey,OwainEvans,PauldeFont-Reaulx,GenevieveFried,BenGarfinkel,KatjaGrace,
RoxHeston,GeoffreyIrving,CharlotteJander,JadeLeung,ChrisOlah,CatherineOlsson,CullenO’Keefe,AndrewSnyder-Beattie,
KenSchultz,ClaudiaShi,DuncanSnidal,NateSoares,JaanTallinn,RobertTrager,HelenToner,BrianTse,EliezerYudkowsky,
BaobaoZhang,andespeciallyDarioAmodei,NickBostrom,OwainEvans,CarrickFlynn,RoseHadshar,HoldenKarnofsky,Jan
Leike, Matthijs Maas, Luke Muehlhauser, Toby Ord, Michael Page, Carl Shulman, and Remco Zwetsloot.

1

AI Governance: A Research Agenda
Preface
Thisdocumentismeanttohelpintroduceandorientresearcherstothespaceofplausibly
importantproblemsinAIgovernance.Itoffersaframingoftheoverallproblem,anattempt
tobecomprehensiveinposingquestionsthatcouldbepivotal,andreferencestopublished
articles relevant to these questions. Some disclaimers are in order.

(1) Focusonextremerisks :ThisdocumentfocusesonextremerisksfromadvancedAI.This
isnotnecessarilymeanttoimplythatadvancedAIposesahighprobabilityofextreme
dangers,orthatthemostimportantrisksaretheextremeones.Thisdocumentfocuseson
risksmorethanopportunitiesforseveralreasons,includingthattheyareoftenmore
time-sensitivetoanticipate,theyareoftenlesslikelytobeaddressedbythemarket,and
manypeoplearelossaverseleadingtosubstantialwelfaregainsfrompreventingrisks.This
documentfocusesonextremerisksmorethanmoderateandsmallrisksbecauseitis
addressedtoacommunityofscholars,policymakers,andphilanthropistswhoprioritize
addressingextremestakes,suchasmanyintheEffectiveAltruismcommunityandatour
collaborating institutions.

(2) Notsufficient :Thisdocumentisanintroduction,nottheintroduction.Tocalibrate
expectations,considerthatittakesyearsofreading,thinking,conversations,andtrialand
errorforaspiringresearchersinmaturefieldstofindawaytomakeacontribution.Foranew
field like AI Governance, “getting up to speed” can still require several months.

(3) Neitherparsimoniousnorcomprehensive :Thereisatradeoffbetweenparsimonyand
comprehensiveness.Thisdocumentmaybeunsatisfyingonbothfronts.Itisnot
parsimonious,andcanfeellikea“firehose”ofquestionsandideas.Norisitcloseto
comprehensiveordetailed:somesentencessummarizealargebodyofthought,andthereare
manyconnectionsthatarenotmadeexplicit.Ifyouwantmoreparsimony,lookatthetableof
contentsandfocusonboldedtermsandtopicsentences.Ifyouwantmore
comprehensiveness, follow the references in a given section.

2

AI Governance: A Research Agenda
(4) Longanddense :Inaimingfor(superficial)comprehensiveness,thisdocumentislong
and dense. You should not expect to digest it in one sitting. You may want to skim most of it.

(5) Nosinglefocus :Youshouldnotexpectthisdocumenttogiveyoustrongadviceabout
whatworktoprioritize.Thisisadeliberatechoice.ForanygivenindividualIcangive
recommendationsofwhatworkshouldbeprioritized.Forthecommunityofresearchers,I
believethereisavastrangeofworkthatshouldbedoneandthatindividualsshould
specialize according to their comparative advantage, interest, and insight. More on this below.

(6) Notauthoritative :Thisdocumentaimsfor(superficial)comprehensiveness,andso
coverstopicsbeyondmyexpertise.Thesetopicsmaybediscussedinlessdetailthantheir
importanceandtractabilitywarrants.Iencouragerelevantexpertstowriteshortreviewsfor
topics that are currently neglected.

Thisspaceisrapidlyevolving.Thisdocumentwillbeupdatedtoreferencenewwork,andto
reflectthechangingresearchlandscape.Comments,suggestions,references,andquestions
areverywelcome,astheywillhelpimprovelaterversionsofthisdocument.Pleaseemail
them (to info@governance.ai ) with the subject line ‘Research agenda feedback’.

3

AI Governance: A Research Agenda
Contents
Preface 2
Contents 4
Introduction 5
Transformative AI 8
Overview 11
Other Overviews 14
Technical Landscape 15
1. Mapping Technical Possibilities 15
1.1 Rapid and Broad Progress? 15
1.2 Kinds, Capabilities, and Properties of AI 18
1.3 Other Strategic Technology 21
2. Assessing AI Progress 21
2.1 Measuring Inputs, Capabilities, and Performance 22
2.2 Modeling AI Progress 22
2.3 Forecasting AI Progress 24
3. AI Safety 25
3.1 The Problem of AI Safety 25
3.2 AI Safety as a Field of Inquiry 27
3.3 The Implications of AI Safety for AI Governance 31
AI Politics 34
4. Domestic and Mass Politics 34
4.1 Forms of Government 35
4.1.1 Inequality 35
4.1.2 Surveillance 36
4.1.3 Repression and Persuasion 36
4.1.4 Advisors and Collective Action 37
4.2 Inequality, Job Displacement, Redistribution 37
4.3 Public Opinion and Regulation 39
5. International Political Economy 39
6. International Security 41
6.1 Near-term Security Challenges 42
6.2 Control, Closing, and Securitization 42
6.3 Race Dynamics 43

4

AI Governance: A Research Agenda
 6.4 Avoiding or Ending the Race 45
Third-Party Standards, Verification, Enforcement, and Control 46
AI Ideal Governance 48
7. Values and Principles 49
8. Institutions and Mechanisms 50
9. Positive Visions 51
Appendix A: Forecasting Desiderata 52
Introduction
Artificialintelligenceislikelytobecomesuperhumanatmostimportanttaskswithinthis              2
century.Thiswouldposetremendousopportunitiesandrisksforhumanity.Further,AI
expertsforeseeanon-trivialprobabilitythatthenextdecade(~10%)ortwo(~25%)could               3
seeAIcapabilitiesemergethatcouldradicallytransformwelfare,wealth,orpower,toan
extentcomparabletothenuclearrevolutionoreventheindustrialrevolution.These
possibilitiesarestrikinglyneglected,inpartbecausetheyinvolvemassiveglobaland
intergenerationalexternalities.Thereisthusahighleverageopportunitytoaddresswhat
maybethemostimportantglobalissueofthe21stcentury.Seekingtodothis,thefieldof AI
governancestudieshowhumanitycanbestnavigatethetransitiontoadvancedAIsystems ,              4
focusingonthepolitical,economic,military,governance,andethicaldimensions.This
document provides an overview of this research landscape.

2 Defined simply as the development of machines capable of sophisticated (intelligent) information processing. Compare also the
definition by Nils Nilsson: “Artificial intelligence is that activity devoted to making machines intelligent, and intelligence is that quality
that enables an entity to function appropriately and with foresight in its environment.” Nilsson, Nils J. The Quest for Artificial
Intelligence: A History of Ideas and Achievements . Cambridge; New York: Cambridge University Press, 2010. For a survey of
definitions of ‘intelligence’, see Legg, Shane, and Marcus Hutter. “A Collection of Definitions of Intelligence.” ArXiv:0706.3639 [Cs] ,
June 25, 2007.  http://arxiv.org/abs/0706.3639 ; for different definitions of ‘AI’ used within the field, see Russell, S.J., and Peter Norvig.
Artificial Intelligence: A Modern Approach . Upper Saddle River, NJ: Prentice Hall, 2009 (3rd edition).
http://www.cin.ufpe.br/~tfl2/artificial-intelligence-modern-approach.9780131038059.25368.pdf , p. 5.
3 These probabilities refer to the median respondent’s beliefs about when we will see AI that is better than all humans at all tasks. For
more on AI experts beliefs, see: Grace, Katja, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. “When Will AI Exceed
Human Performance? Evidence from AI Experts.” ArXiv:1705.08807 [Cs] , May 24, 2017.  http://arxiv.org/abs/1705.08807 ; see also
Grace, Katja. “2016 Expert Survey on Progress in AI.” AI Impacts , December 14, 2016.
https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/ . Note that these forecasts should not be regarded as especially reliable, given
that the respondents are not known to be calibrated in their statements of probabilities, the respondents are not experts at forecasting nor
are experts at macro-developments in AI; nevertheless, it provides a perspective on timelines. Interestingly, informally I would say the
median of other more careful approaches yields a similar timeline.
4 “Advanced AI” gestures towards systems substantially more capable (and dangerous) than existing (2018) systems, without necessarily
invoking specific generality capabilities or otherwise as implied by concepts such as “Artificial General Intelligence” (“AGI”).

5

AI Governance: A Research Agenda
AIgovernanceisoftenpairedwithAIsafety.Bothhavethegoalofhelpinghumanitydevelop                 5
beneficialAI.AIsafetyfocusesonthetechnicalquestionsofhowAIisbuilt;AIgovernance
focusesontheinstitutionsandcontextsinwhichAIisbuiltandused.Specifically,AI
governanceseekstomaximizetheoddsthatpeoplebuildingandusingadvancedAIhavethe
goals,incentives,worldview,time,training,resources,support,andorganizationalhome
necessary to do so for the benefit of humanity.

Tomotivatethisworkitcanbehelpfultoconsideranurgent,thoughnotimplausible,
hypotheticalscenario.Supposethatinoneyear’stimealeadingAIlabperceivesthat
profoundprogressmaybeonthehorizon.Itconcludesthatgivenabigpush,in6to24
monthsthelabislikelytodeveloptechniquesthatwouldachievenovelsuperhuman
capabilitiesinstrategicdomains.Thesedomainsmightincludeliedetection,social-network
mappingandmanipulation,cyber-operations,signalsandimageryintelligence,strategy,
bargainingorpersuasion,engineering,science,andpotentiallyAIresearchitself.Despiteour
knowledge(inthisscenario)thatthesetechnicalbreakthroughsarelikely,wewouldhave
uncertaintyaboutthedetails.Whichtransformativecapabilitieswillcomefirstandhowthey
willwork?How couldsuccessive(small)capabilitiesinteracttobecomejointly
transformative?HowcanonebuildadvancedAIinasafeway,andhowdifficultwillitbeto
doso?Whatdeploymentplansandgovernanceregimeswillbemostlikelytoleadtoglobally
beneficial outcomes?

TheAIgovernanceproblemistheproblemofpreparingforthisscenario,alongwithallother
high-stakesimplicationsofadvancedAI.Thetaskissubstantial.Whatdoweneedtoknow
anddoinordertomaximizethechancesoftheworldsafelynavigatingthistransition?What
advicecanwegivetoAIlabs,governments,NGOs,andpublics,nowandatkeymomentsin
thefuture?Whatinternationalarrangementswillweneed--whatvision,plans,technologies,
protocols,organizations--toavoidfirmsandcountriesdangerouslyracingforshort-sighted
5 Amodei, Dario, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. “Concrete Problems in AI Safety.”
ArXiv:1606.06565 [Cs] , June 21, 2016.  http://arxiv.org/abs/1606.06565 ; Russell, Stuart, Daniel Dewey, and Max Tegmark. “Research
Priorities for Robust and Beneficial Artificial Intelligence.” Future of Life Institute - AI Magazine , 2015.
https://futureoflife.org/data/documents/research_priorities.pdf?x90991 ; Everitt, Tom, Gary Lea, and Marcus Hutter. “AGI Safety
Literature Review.” ArXiv:1805.01109 [Cs] , May 3, 2018.  http://arxiv.org/abs/1805.01109 ; Metz, Cade. “Teaching A.I. Systems to
Behave Themselves.” The New York Times , August 13, 2017, sec. Technology.
https://www.nytimes.com/2017/08/13/technology/artificial-intelligence-safety-training.html .

6

AI Governance: A Research Agenda
advantage?Whatwillweneedtoknowandarrangeinordertoelicitandintegratepeople’s
values, to deliberate with wisdom, and to reassure groups so that they do not act out of fear?

ThepotentialupsidesofAIaretremendous.Thereislittlethatadvancedintelligencecouldn’t
helpuswith.AdvancedAIcouldplayacrucialrolesolvingexistingglobalproblems,from
climatechangetointernationalconflict.AdvancedAIcouldhelpusdramaticallyimprove
health, happiness, wealth, sustainability, science, and self-understanding.  6

Thepotentialdownsides,however,arealsoextreme.Letusconsiderfoursourcesof
catastrophic risk stemming from advanced AI.
❖(1) Robusttotalitarianism couldbeenabledbyadvancedliedetection,social
manipulation,autonomousweapons,andubiquitousphysicalsensorsanddigital
footprints.Powerandcontrolcouldradicallyshiftawayfrompublics,towardselites
andespeciallyleaders,makingdemocraticregimesvulnerabletototalitarian
backsliding, capture, and consolidation.
❖(2)Preventive,inadvertent,orunmanageablegreat-power(nuclear)war .Advanced
AIcouldgiverisetoextremefirst-strikeadvantages,powershifts,ornovel
destructivecapabilities,eachofwhichcouldtemptagreatpowertoinitiatea
preventivewar.AdvancedAIcouldmakecrisisdynamicsmorecomplexand
unpredictable,andenablefasterescalationthanhumanscouldmanage,increasingthe
risk of inadvertent war.
❖(3)BroadlysuperhumanAIsystemscouldbebuiltthatare notfullyalignedwith
humanvalues ,leadingtohumanextinctionorotherpermanentlossinvalue.This              7
riskislikelymuchgreateriflabsandcountriesareracingtodevelopanddeploy
advancedAI,asresearchingandimplementingAIsafetymeasuresisplausiblytime             8
and resource intensive.
6 Bostrom, Nick, Allan Dafoe, and Carrick Flynn. “Public Policy and Superintelligent AI: A Vector Field Approach.” Future of
Humanity Institute, 2018.  http://www.nickbostrom.com/papers/aipolicy.pdf .
7 Bostrom, Nick. Superintelligence: Paths, Dangers, Strategies . Oxford: Oxford University Press, 2014. Yudkowsky, Eliezer. “Artificial
Intelligence as a Positive and Negative Factor in Global Risk.” In Global Catastrophic Risks , edited by Nick Bostrom and Milan M.
Cirkovic. New York: Oxford University Press, 2008, pp. 308–45. See also the reading syllabus by Bruce Schneier. “Resources on
Existential Risk - for: Catastrophic Risk: Technologies and Policies.” Berkman Center for Internet and Society, Harvard University,
2015.  https://futureoflife.org/data/documents/Existential%20Risk%20Resources%20(2015-08-24).pdf?x70892 .
8 Armstrong, Stuart, Nick Bostrom, and Carl Shulman. “Racing to the Precipice: A Model of Artificial Intelligence Development.”
Technical Report. Future of Humanity Institute, 2013.
https://www.fhi.ox.ac.uk/wp-content/uploads/Racing-to-the-precipice-a-model-of-artificial-intelligence-development.pdf .

7

AI Governance: A Research Agenda
❖(4)Finally,evenifweescapethepreviousthreeacuterisks,wecouldexperience
systematicvalueerosionfromcompetition ,inwhicheachactorrepeatedly
confrontsasteeptrade-offbetweenpursuingtheirfinalvaluesorpursuingthe
instrumentalgoalofadaptingtothecompetitionsoastohavemorepowerand
wealth. 9

Theseriskscanbeunderstoodasnegativeexternalities:harmsfromsocio-technical
developmentsthatimpactindividualsotherthanthoseresponsibleforthedevelopments.
Theseexternalitiesareespeciallychallengingtomanageastheymaybeextremein
magnitude,complexandhardtopredict,andtheywillspillacrossbordersandgenerations.
Buildingtherightinstitutions(includingnormsandpoliticalarrangements)isplausiblyclose
toanecessaryandsufficientconditiontoadequatelyaddresstheserisks.Withtheright
institutionstheseriskscanberadicallyreducedandplausiblyeliminated.Withoutthem,it
maybethatnothingshortofatechnicalmiraclewillbesufficienttosafelynavigatethe
transition to advanced AI systems.
Transformative AI
Steppingbackfromthisscenario,researchonAIgovernanceconsidersAI’smost
transformativepotentialcapabilities,dynamics,andimpacts.Thestakescouldbeextreme:
absentaninterruptionindevelopment,AIthiscenturyislikelytobesufficiently
transformativeto“precipitateatransitioncomparableto(ormoresignificantthan)the
agriculturalorindustrialrevolution.” Givenourcurrentuncertainties aboutwhich           10 11
capabilitieswillhavethegreatestimpacts,however,itcanbeusefultoattendtoabroad
rangeofpotentiallytransformativecapabilitiesanddynamics.Accordingly,wefocuson
9 Thanks to Daniel Dewey for suggesting this clear statement of the risks.
10 Karnofsky, Holden. “Potential Risks from Advanced Artificial Intelligence: The Philanthropic Opportunity.” Open Philanthropy
Project, 2016.  http://www.openphilanthropy.org/blog/potential-risks-advanced-artificial-intelligence-philanthropic-opportunity ;
Muehlhauser, Luke. “How Big a Deal Was the Industrial Revolution?” 2017.  http://lukemuehlhauser.com/industrial-revolution/ .
11 As an analogy for the difficulty we may have in perceiving a transformative capability, it took about 10 years from Fleming’s
discovery of penicillin to the production of a compelling proof-of-concept and recognition by a major funder (Rockefeller) that this was
worth seriously investing in. Bud, Robert. Penicillin: Triumph and Tragedy . Oxford: Oxford University Press, 2008, pp. 23–34.
Likewise with airplanes: in 1901, two years before building the first heavier-than-air plane, Wilbur Wright said to his brother “that men
would not fly for fifty years.” McCullough, David. The Wright Brothers . Simon & Schuster. p. 208; also quoted in Yudkowsky, Eliezer.
“There’s No Fire Alarm for Artificial General Intelligence.” Machine Intelligence Research Institute, October 13, 2017.
https://intelligence.org/2017/10/13/fire-alarm/ .

8

AI Governance: A Research Agenda
transformativeAI(TAI),understoodasadvancedAIthatcouldleadtoradicalchangesin
welfare, wealth, or power. 12

WhataresomewaysthatAIcouldbetransformative?AIcouldvastlyincreasewealth,health,
andwell-being.Itcouldtransformworkbyradicallyalteringemploymentprospectsorjob              13
security.Itcouldincreaseeconomicinequality,domesticallyandglobally.Itcouldprovide
newtoolsofstaterepressionandcontrol,empoweringauthoritariangovernments;itcould
alsoenablenewformsofeffectivedemocraticdecision-makingandaccountability,
empoweringdemocracy.Itcouldtransforminternationalpoliticaleconomy(IPE);for
example,AIisincreasinglyperceivedasastrategicindustry,activatingmassiveindustrial
policy to support national AI champions and assets.

AIcouldtransforminternationalsecuritybyalteringkeystrategicparameters,suchasthe
securityofnuclearretaliation,theoffense-defensebalance,thestabilityofcrisis            14 15
escalation,theefficiencyofnegotiations,theviabilityofmutualprivacypreserving
surveillance,andthevolatilityandpredictabilityofthefuturebalanceofpower.Itcould
enablenewoperationalandstrategiccapabilities,forinstanceinmass-persuasion,
cyber-operations,commandandcontrol,intelligence,aircombat,subseacombat,materials
12Itisworthbeingclearaboutthemagnitudesofimpactthatoneiscontemplating.“Transformativeness”anditscognateshave
beenusedinabroadsetofways.Atthehighend,ithasbeenusedtorefertothemostextremeimpacts,suchasAIthat
“precipitatesatransitioncomparableto(ormoresignificantthan)theagriculturalorindustrialrevolution.”Callimpactsatthe
scaleoftheagriculturalrevolutionorindustrialrevolution revolutionaryimpacts .Atthelowend,thetermisusedtorefertothe
mundane reorganization of industries. I use the term here to refer to impacts of intermediate magnitude.

Therearetradeoffsinanydefinitionofaconcept.Expandingtheconceptwillallowustocontributetoandbenefitfromthe
broaderconversationontheimpactsofAI,mostofwhicharenotfocusedonrevolutionaryimpacts.However,doingsomaytake
our eyes off the most important impacts, diluting our efforts.

Irecommendtheaboveintermediatedefinition,delimitedatinnovationsthatcouldinduceradicalchanges.Inadditionto
helpingusaddressthebroadersetoflargeimplications,doingsowillhelpusremainattentivetothemanyseeminglysmall
achievements,events,ordynamicsthatcangeneratemassiveimpacts.Tomost,thefirstdomesticatedplantsandthefirststeam
engines would not have looked revolutionary.
13 Aghion, Philippe, Benjamin Jones, and Charles Jones. “Artificial Intelligence and Economic Growth.” Cambridge, MA: Stanford
Institute for Economic Policy Research (SIEPR), October 2017.
https://pdfs.semanticscholar.org/b0f0/989edd61ffa192c2a54e8edded9b84781719.pdf ; Korinek, Anton, and Joseph E. Stiglitz. “Artificial
intelligence and its implications for income distribution and unemployment.” No. w24174, National Bureau of Economic Research,
2017. https://www8.gsb.columbia.edu/faculty/jstiglitz/sites/jstiglitz/files/w24174.pdf .
14 Geist, Edward, and Andrew J Lohn. “How Might Artificial Intelligence Affect the Risk of Nuclear War?” RAND, 2018.
https://www.rand.org/pubs/perspectives/PE296.html ; Lieber, Keir A., and Daryl G. Press. “The New Era of Counterforce: Technological
Change and the Future of Nuclear Deterrence.” International Security 41, no. 4 (April 2017): 9–49.
https://doi.org/10.1162/ISEC_a_00273 ; for work on the general interface of nuclear weapons with cybersecurity, see Futter, Andrew.
Hacking the Bomb: Cyber Threats and Nuclear Weapons.  Washington, DC: Georgetown University Press, 2018.
15 Dafoe, Allan and Ben Garfinkel. “How Does the Offense-Defense Balance Scale?” Future of Humanity Institute, 2018.
https://drive.google.com/file/d/1AR9DEjPheYrJxUGpOdORxk-qYnyYHj0h/view.

9

AI Governance: A Research Agenda
science,engineering,andscience.Theseadvantagesmaycomeinsufficientstrengthor
combinationstoradicallytransformpower.Eventhemereperceptionbygovernmentsand
publicsofsuchmilitary(oreconomic)potentialcouldleadtoaradicalbreakfromthecurrent
technologyandworldorder:shiftingAIleadershiptogovernments,givingrisetoamassively
fundedAIraceandpotentiallythesecuritizationofAIdevelopmentandcapabilities.This              16
couldunderminetheliberalworldeconomicorder.Theintensityfromaracedynamiccould               17
leadtocatastrophiccorner-cuttinginthehurrieddevelopmentanddeploymentof(unsafe)
advancedAIsystems.Thisdangerposesextremeurgency,andopportunity,forglobal             18
cooperation.

Thesetransformativeinnovationsandimpactsmayarisegraduallyoverthecomingdecades,
facilitatingouranticipationandgovernanceoftheirarrival.Buttheymayalsoemergemore
suddenlyandunexpectedly,perhapsduetorecursiveself-improvement,advancesin
especiallypotentgeneralpurposeapplicationsorevengeneralintelligence,oroverhangof
computationalpowerorothercrucialinputs.Thespeed,suddenness,andpredictabilityofthe
arrival of new capabilities will shape the character of the challenges we will face.

From amorelong-term andabstractperspective,theemergenceofmachine
superintelligence (AIthatisvastlybetterthanhumansatallimportanttasks)wouldenable
revolutionarychanges,moreprofoundthantheagriculturalorindustrialrevolutions.
Superintelligenceofferstremendousopportunities,suchastheradicalreductionofdisease,
poverty,interpersonalconflict,andcatastrophicriskssuchasclimatechange.However,
superintelligence,andadvancedAImoregenerally,mayalsogeneratecatastrophic
vulnerabilities,includingextremeinequality,globaltyranny,instabilitiesthatsparkglobal
(nuclear)conflict,catastrophicallydangeroustechnologies,or,moregenerally,insufficiently
controlledoralignedAI.Evenifwesuccessfullyavoidsuchtechnologicalandpoliticalpitfalls,
tremendousgovernancequestionsconfrontusregarding whatwewant ,and whatweoughtto
16 On the deleterious effects of framing AI development as a ‘race’, see Cave, Stephen, and Seán S. Ó hÉigeartaigh. “An AI Race for
Strategic Advantage: Rhetoric and Risks.” In AAAI / ACM Conference on Artificial Intelligence, Ethics and Society , 2018.
http://www.aies-conference.com/wp-content/papers/main/AIES_2018_paper_163.pdf .
17 Danzig, Richard, ed. “An Irresistible Force Meets a Moveable Object: The Technology Tsunami and the Liberal World Order.”
Lawfare Research Paper Series  5, no. 1 (August 28, 2017).  https://assets.documentcloud.org/documents/3982439/Danzig-LRPS1.pdf .
18 Armstrong, Stuart, Nick Bostrom, and Carl Shulman. “Racing to the Precipice: A Model of Artificial Intelligence Development.”
Technical Report. Future of Humanity Institute, 2013.

10

AI Governance: A Research Agenda
want ,theanswerstowhichwillrequireustoknowourselvesandourvaluesmuchbetter
than we do today.
Overview
AIgovernancecanbeorganizedinseveralways.Thisagendadividesthefieldintothree
complementaryresearchclusters:the technicallandscape , AI politics ,and AIideal
governance .Eachoftheseclusterscharacterizesasetofproblemsandapproacheswithin
whichthedensityofconversationislikelytobegreater.However,mostworkinthisspace
willneedtoengagewiththeotherclusters,reflectingandcontributinghigh-levelinsights.
Thisframeworkcanperhapsbeclarifiedusingananalogytotheproblemofbuildinganew
city.The technicallandscapeexaminesthetechnicalinputsandconstraintstotheproblem,
suchastrendsinthepriceandstrengthofsteel. Politicsconsidersthecontending
motivationsofvariousactors(suchasdevelopers,residents,andbusinesses),themutually
harmfuldynamicsthatcouldpotentiallyarisebetweenthem,andstrategiesforcooperating
toovercomesuchdynamics. Idealgovernanceinvolvesunderstandingthewaysthat
infrastructure,laws,andnormscanbeusedtobuildthebestcity,andproposingidealmaster
plans of these to facilitate convergence on a common good vision.

Thefirstcluster,the technicallandscape ,seekstounderstandthetechnicalinputs,
possibilities,andconstraintsforAI,andservesasafoundationfortheotherclustersofAI
governance.Thisincludes mappingwhatcouldbethecapabilitiesandpropertiesof
advancedandtransformativeAIsystems,whenparticularcapabilitiesarelikelytoemerge,
andwhethertheyarelikelytoemergegraduallyinsequenceorrapidlyacross-the-board.To
theextentpossible,thisclusteralsoinvolves modelingandforecastingAIprogress :the
productionfunctionofAIprogress,giveninputssuchascompute,talent,data,andtime, and
theprojectionofthisprogressintothefuture .Wealsoneedtoassesstheviability,
constraints,costs,andpropertiesofscalably safeAIsystems .Towhatextentwillweneedto
investresourcesandtimelateinthedevelopmentprocess?Whatinstitutionalarrangements
bestpromoteAIsafety?TowhatextentwillthecharacteristicsofsafeAIbeapparenttoand
observablebyoutsiders,aswouldbenecessaryfor(non-intrusive)externaloversightand
verification agreements?

11

AI Governance: A Research Agenda

Thesecondclusterconcerns AIpolitics ,whichfocusesonthepoliticaldynamicsbetween
firms,governments,publics,researchers,andotheractors,andhowthesewillbeshapedby
andshapethetechnicallandscape.HowcouldAItransform domesticandmasspolitics ?
WillAI-enabledsurveillance,persuasion,androboticsmaketotalitariansystemsmore
capableandresilient?Howwillcountriesrespondtopotentiallymassiveincreasesin
inequalityandunemployment,andhowwilltheseresponsessupportorhinderotherglobal
governanceefforts?Whenandhowwillvariousactorsbecomeconcernedandinfluential
(whatcouldbetheir“AISputnik”moments)?HowcouldAItransformthe international
politicaleconomy ?WillAIcometobeseenasthecommandingheightsofthemodern
economy,warrantingmassivestatesupportandintervention?Ifso,whatpolicieswillthis
entail,whichcountriesarebestpositionedtoseizeAIeconomicdominance,andhowwillthis
AI nationalism interact with global free trade institutions and commitments?

Potentiallymostimportantly,howwillAIinteractwith internationalsecurity ?Whatarethe
near-termsecuritychallenges(andopportunities)posedbyAI?CouldAIradicallyshiftkey
strategicparameters,suchasbyenablingpowerfulnewcapabilities(incyber,lethal
autonomousweapons[LAWs],militaryintelligence,strategy,science),byshiftingthe
offense-defensebalance,orbymakingcrisisdynamicsunstable,unpredictable,ormore
rapid?CouldtrendsinAIfacilitatenewformsofinternationalcooperation,suchasby
enablingstrategicadvisors,mediators,orsurveillancearchitectures,orbymassively
increasingthegainsfromcooperationandcostsofnon-cooperation?IfgeneralAIcomestobe
seenasacriticalmilitary(oreconomic)asset, underwhatcircumstancesis thestatelikelyto
control,close,orsecuritizeAIR&D?Whataretheconditionsthatcouldsparkandfuelan
international AIrace ?Howgreatarethedangersfromsucharace,howcanthosedangersbe
communicatedandunderstood,andwhatfactorscouldreduceorexacerbatethem?What
routesexistforavoidingorescapingtherace,suchasnorms,agreements,orinstitutions
regardingstandards,verification,enforcement,orinternationalcontrol?Howmuchdoesit
mattertotheworldwhethertheleaderhasalargelead-margin,is(basedin)aparticular
country(e.g.theUSorChina),orisgovernedinaparticularway(e.g.transparently,by
scientists)?

12

AI Governance: A Research Agenda

Insteeringawayfromdangerousrivalrousdynamicsitwillbehelpfultohaveaclearsenseof
whatwearesteeringtowards,whichbringustothefinalresearchcluster:whatarethe ideal
governancesystemsforglobalAIdynamics? Whatwouldwecooperatetobuildifwecould?
Whatpotentialglobalgovernancesystems--includingnorms,policies,laws,processes,and
institutions -- canbestensurethebeneficialdevelopmentanduseofadvancedAIsystems?To
answerthisweneedtoknowwhat values humanitywouldwantourgovernancesystemsto
pursue,orwouldwantifweunderstoodourselvesandtheworldbetter.Morepragmatically,
whatarethespecificinterestsofpowerfulstakeholders,andwhat institutional
mechanismsexisttoassurethemofthedesirabilityofacandidategovernanceregime?
Insightsforlong-termglobalgovernancearerelevanttocontemporaryandmedium-termAI
governance,aswewouldliketoembedtheprinciplesandinstitutionalmechanismsthatwill
becrucialforthelong-termtoday,whilethestakesarerelativelylow.Itwillalsofacilitate
cooperationtodayifwecanassurepowerfulactorsofalong-termplanthatiscompatible
with their interests.

Inworkinginthisspaceacrossthethreeresearchclusters,researchersshouldprioritizethe
questionswhichseemmost important ,tractable , and neglected , andforwhichtheyhavea
comparativeadvantageand interest . Questionsaremorelikelytobeimportantiftheyare
likelytoidentifyoraddresscrucialconsiderations,oriftheydirectlyinformurgentpolicy
decisions.Questionsaremorelikelytobetractableiftheresearchercanarticulatea
promisingwell-definedresearchstrategy,thequestionscanbetackledinisolation,orthey
reducetoresolvablequestionsoffact.Questionsaremorelikelytobeneglectediftheydonot
directlyandexclusivelycontributetoanactor’sprofitorpower,likemanylong-termor
globalissues,andiftheyfalloutsideofthefocusoftraditionalresearchcommunities.
Researchersshouldupweightquestionsforwhichtheyhavecomparativelyrelevantexpertise
orcapabilities,andinwhichtheyareespeciallyinterested.Ultimately,perhapsthesimplest
ruleofthumbistojustbeginwiththosequestionsorideasthatmostgrabyou,andstart
furiously working.

13

AI Governance: A Research Agenda
Other Overviews
ArichoverviewtoissuesinthefieldisgiveninBostrom,Nick.Superintelligence:Paths,
Dangers,Strategies.Oxford:OxfordUniversityPress,2014.Seeespeciallychapters4(‘The
KineticsofanIntelligenceExplosion’),5(‘DecisiveStrategicAdvantage’),11(‘Multipolar
scenarios’), and 14 (‘The strategic picture’).

TheFutureofLifeInstituteoffersasetofresourcesonGlobalAIPolicyhere:
https://futureoflife.org/ai-policy/ .

14

AI Governance: A Research Agenda

 Technical Landscape
Workonthetechnicallandscapeseekstounderstandthetechnicalinputs,possibilities,and
constraintsforAI,providinganessentialfoundationforourlaterstudyofAIpolitics,ideal
governance,andpolicy.Thisincludesmappingwhatthecapabilitiesandpropertiesof
transformativeAIsystemscouldbe,whentheyarelikelytoemerge,andwhethertheyare
likelytoemergeinparticularsequencesormany-at-once.Thisresearchclusterbenefitsfrom
expertiseinAI,economicmodeling,statisticalanalysis,technologyforecastingandthehistory
oftechnology,expertelicitationandaggregation,scenarioplanning,andneuroscienceand
evolution.
1. Mapping Technical Possibilities
Thisclusterinvestigatesthemoreabstract,imaginativeproblemareaofmappingtechnical
possibilities,andespeciallypotentiallytransformativecapabilities.Arewelikelytoseea
rapidbroad(andlocal?)achievementofmanytransformativecapabilities?Whatkindsof
transformativecapabilitiescouldplausiblyemerge,andinwhatorder?Whataretheir
strategicproperties,suchasbeingoffense-ordefense-biased,ordemocracyorautocracy
valenced?
1.1 Rapid and Broad Progress?
AfirstissueconcernshowrapidandgeneraladvanceswillbeinAI.Somebelievethat
progresswill,atsomepoint,allowforsuddenimprovementsinAIsystems’capabilities
acrossabroadrangeoftasks.Ifso,muchofthefollowingproposedworkonsequencesand
kindsofAIwouldbeunproductive,sincemosttransformativecapabilitieswouldcomeonline
atthesameexplosivemoment.Forthisreason,thisagendadrawsinitialattentiontothis
question.

Rapidgeneralprogresscouldcomeaboutfromseveralmechanisms,enumeratedwithsome
redundancy:
❖(1a)Manyimportanttasksmayrequireacommoncapability,theachievementof
whichwouldenablemasteryacrossallofthem.Forexample,deeplearningunlocked

15

AI Governance: A Research Agenda
seeminglydisparatecapabilities,spanningimagerecognition,languagetranslation,
speechrecognition,gameplaying,andothers.Perhapsasubstantialadvancein
“efficient meta-learning” or transfer learning could catalyze advances in many areas.
❖(1b)Clustersofnovelpowerfultechnologicalcapabilitiesthatarelikelytobe
unlockedincloseproximitytoeachother,perhapsbecausetheyfacilitateeachother
or depend on solving some common problem.
❖(2a) Complements :Scientificandtechnologicaladvancesoftendependonhaving
severalcrucialinputs,eachofwhichactsasastrongcomplementtotheothers.For
example,thedevelopmentofpoweredflightseemstohaverequiredsufficient
advancesintheinternal-combustionengine.Complementaritiescouldleadtojumps           19
in capabilities in several ways.
➢(i) Unjammedbottlenecks :Therecouldberapidalleviationofacrucial
bottleneck.Forexample,wehaveseensuddenjumpsincapabilitiesfromthe
provisionofasinglelargetrainingdatasetforaparticulartask.Similarly,the
generationofacrucialtrainingsetforagenerallyapplicabletaskcouldleadto
a broad front of progress.
➢(ii) Overhang :Therecouldbealatentreservoirofacrucialcomplement,that
becomessuddenlyunlockedoraccessible.Forexample,itcouldcomefrom:            20
hardwareoverhanginwhichthereisalargereservoirofcomputeavailable
toberepurposedfollowinganalgorithmicbreakthrough;fromabundant
insecurecomputethatcanbeseizedbyanexpansionistentity;from insight
overhang ,iftherearegeneralpowerfulalgorithmicimprovementswaitingto
beuncovered;orfrom dataoverhang ,suchasthecorpusofdigitizedscience
textbooks waiting to be read, and the internet more generally.
➢(iii) Complementaryclustersofcapabilities :AdvancesinonedomainofAI
couldstronglycomplementprogressinotherdomains,leadingtoaperiodof
rapidprogressineachofthesedomains.Forexample,naturallanguage
19Crouch,Tom D,Walter James Boyne etal. “History offlight.” Encyclopædia Britannica ,2016.
https://www.britannica.com/technology/history-of-flight/The-generation-and-application-of-power-the-problem-of-propulsio
n .
20Unjammedbottlenecksandoverhangsarecloselyrelatedperspectivesoncomplements,focusingeitheronthelastnecessary
inputoranalreadyachievednecessaryinput.Forexample,considertheprogressfunctionf(X,Y,Z)=MIN(X,Y,Z).Ifatbaseline
X=0,Y=1,Z=1thenprogressinXfrom0to1wouldrepresentanunjammedbottleneck.IfatbaselineX=0,Y=0,Z=1,thenZcould
be regarded as a form of overhang.

16

AI Governance: A Research Agenda
understandingcouldmakeitcost-effectivetoefficientlycreatemassive
datasetsforavarietyofpurposesfromtheinternet,creationofthesedatasets
couldimprovemachineunderstandingofhowmanytaskdomainsinthe
worldrelatetoeachother,whichcouldimprovetransferlearningbetween
those domains, which could further improve natural language understanding.
❖(2b)Rapidprogressinacrucialbottleneck/complementofAIresearch.Forexample,
wehaveseensuddenjumpsincapabilitiesfromtheprovisionofasinglelarge
trainingdatasetforaparticulartask.Similarly,thegenerationofacrucialtrainingset
for a generally applicable task could lead to a broad front of progress.
❖(3)SubstantialAIadvancesontaskscrucialforfutureAIR&D,permittinghighly              21
recursiveself-improvement.Thismightleadtoanendogenousgrowthpositive
feedbackprocess,sometimescalledan“intelligenceexplosion”,whereeach          22
generation of AI accelerates the development of the subsequent generation.
❖(4) Radical increases in investment in AI R&D.
❖(5)AlargeratioofR&Dcoststoexecutioncosts,sothatonceaparticularcapabilityis
achieveditcouldbemassivelydeployed.Forexample,thelearningprocesscouldbe
highlycomputeintensive(suchaswithgeneticalgorithms),butoncetrainedthat
same compute could be used to run thousands of instances of the new algorithm.

SomeargumentsforrapidgeneralprogresshavebeenarticulatedbyEliezerYudkowsky,             23
NickBostrom,andMIRI.Someargumentsagainst(spatiallylocal)rapidgeneralprogress             24
hasbeenexpressedbyRobinHanson,PaulChristiano,AIImpactsandKatjaGrace,and               25 26 27
BenGoertzel,andisimplicittomostmainstreamperspectives.Theskepticalposition             28 29
21 Autoregressive parameter persistently above 1.
22Good,I.J.“SpeculationsConcerningtheFirstUltraintelligentMachine.”In AdvancesinComputers ,editedbyFranzL.Altand
Moris Rubinoff, 6:31–88. New York: Academic Press, 1964.
23Yudkowsky,Eliezer.“IntelligenceExplosionMicroeconomics.”MachineIntelligenceResearchInstitute,2013.
https://intelligence.org/files/IEM.pdf.
24 Bostrom, Nick. Superintelligence: Paths, Dangers, Strategies.  Oxford: Oxford University Press, 2014. Chapter 4.
25Hanson, Robin. “I Still Don’t Get Foom.” Overcoming Bias (blog), July 24, 2014.
http://www.overcomingbias.com/2014/07/30855.html .
26Christiano, Paul. “Takeoff Speeds.” The Sideways View (blog), February 24, 2018.
https://sideways-view.com/2018/02/24/takeoff-speeds/ .
27Grace,Katja.“LikelihoodofDiscontinuousProgressaroundtheDevelopmentofAGI.” AIImpacts ,February23,2018.
https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/ .
28Goertzel,Ben.“Superintelligence:Fears,PromisesandPotentials:ReflectionsonBostrom’sSuperintelligence,Yudkowsky’s
FromAItoZombies,andWeaverandVeitas’s‘Open-EndedIntelligence.’” JournalofEvolution&Technology24,no.2(November
2015): 55–87. http://www.kurzweilai.net/superintelligence-fears-promises-and-potentials .

17

AI Governance: A Research Agenda
 drawssupportfromthefactthatAIprogressandtechnologicalprogresstendstobegradual,
piecemeal,uneven,andspatiallydiffuse.IfthisremainstrueforAIthenweshouldexpect
some transformative capabilities to come online far before others.
1.2 Kinds, Capabilities, and Properties of AI
AIcouldbetransformativeinmanyways.Weshouldsystematicallythinkthroughthekinds
ofAIthatcouldbedeveloped,andwhattheircapabilitiesandpropertiesmightbe.For                30
scenarioswhereprogressisnotrapidandbroad,itwillalsobeusefultoarticulateprobable
sequencesinAIcapabilities,ornecessaryachievements,priortoparticularkindsof
transformative AI.

SomeexamplesofpotentiallytransformativecapabilitiesincludeAIthatissuperhumanin,or
otherwisetransformativeof,particularareassuchascybersecurity,autonomousweapons,
surveillance,profiling,lie-detection,persuasionandmanipulation,finance,strategy,
engineering,manufacturing,andotherareasofscienceandtechnology.SuchAI,ifarriving
unbundledfromothertransformativecapabilities,isoftencalled“narrowAI”.Inadditionto
producingnewcapabilities,AIcouldbetransformativethroughincrementaleffects,suchas
incrementalchangesinthecostsorperformanceofexistingcapabilities,tothepointthatit
transforms industries and world order.  31

29Seealsothereadinglist,compiledbyMagnusVinding,onargumentsagainstthehardtake-off’hypothesis:
https://magnusvinding.com/2017/12/16/a-contra-ai-foom-reading-list/ .
30Thereissomeworkon“kindsofintelligence”thatmayspeaktothis.Foraninformalintroduction,seeShanahan,Murray.
“Beyond Humans,What Other Kinds ofMinds Might Be out There?” Aeon ,October 19,2016.
https://aeon.co/essays/beyond-humans-what-other-kinds-of-minds-might-be-out-there ;Shanahan,Murray.“TheSpaceof
PossibleMinds”EDGE,May18,2018. https://www.edge.org/conversation/murray_shanahan-the-space-of-possible-minds .See
alsotheCFIprojecton‘KindsofIntelligence’,at http://lcfi.ac.uk/projects/kinds-of-intelligence/ ,andspecificallyJosé
Hernández-Orallo,“TheMeasureofAllMinds”,2017,CambridgeUniversityPress,http://allminds.org/.AlsoseeNIPS2017
symposium: http://www.kindsofintelligence.org/ .
31AdiscussionofthepositiveaspectsofthisisinHarford,Tim.“WhatWeGetWrongaboutTechnology.” FTMagazine ,July17,
2017. https://www.ft.com/content/32c31874-610b-11e7-8814-0ac7eb84e5f1 .Negativepossibilitiesalsoexist.Forexample,
evenwithoutanyspecificcapabilitiesthatareespeciallytransformativeornovel,AIandassociatedtrajectoriescoulddisplace
sufficientworkerstogenerateapoliticaleconomiccrisisonthescaleoftheGreatDepression.“Isuspectthatifcurrenttrends
continue,wemayhaveathirdofmenbetweentheagesof25and54notworkingbytheendofthishalfcentury,becausethisisa
trendthatshowsnosignofdecelerating.Andthat'sbeforewehave...seenasingledriverreplaced[byself-drivingvehicles]...,
notatrucker,notataxicabdriver,notadeliveryperson....Andyetthatissurelysomethingthatisenroute.”Quotedin
Matthews,Christopher.“Summers:Automation isthemiddleclass'worstenemy.” Axios ,June4,2017.
https://www.axios.com/summers-automation-is-the-middle-class-worst-enemy-1513302420-754facf2-aaca-4788-9a41-38f87f
b0dd99.html .

18

AI Governance: A Research Agenda
TodateAIsystemsremain narrow ,inthesensethatatrainedsystemisabletosolvea
particularproblemwell,butlackstheabilitytogeneralizeasbroadlyasahumancan.
Further,advancesinAIcapabilitiesarehighlyuneven,relativetothedistributionofhuman
capabilities,andthistrendseemslikelytopersist:gameplayingalgorithmsarevastly
superhumanatsomegames,andvastlysubhumanatothers.AIsystemstodayare              32
sometimesanalogizedas“idiotsavants”:theyvastlyoutperformhumansatsometasks,but
areincompetentatother“simple”adjacenttasks.AIsystemsareapproachingorarenow
superhumanattranslatingbetweenlanguages,categorizingimages,recognizingfaces,and           33
drivingcars,buttheystillcan’tanswerwhatseemlikesimplecommon-sensequestionssuch
as Winograd Schemas.

ItmaybethecasethatmanykindsofTAIwillarrivefarbeforeAIhasachieved“common
sense”orachild’sabilitytogeneralizelessonstoanewtaskdomain.Manythinkers,however,
thinktheoppositeisplausible.Theyreasonthatthereisplausiblysomefacultyofgeneral
intelligence,somecorecognitivemodule,somecommonfactortomostkindsof“few-shot”
learning(learningfromonlyafewexamples).Thisgeneralintelligence,onceachievedateven
merelythelevelofafour-year-oldhuman,wouldenableAIsystemstobebuiltthatquickly
learninnewdomains,benefitingfromanddirectingtheirsuperhumanmemory,processing
speed,sensorarrays,accesstoinformationandwealthofstoredinformation,andlibraryof
specializedsystems.Thisartificialgeneralintelligence(AGI)--AIthatcanreasonbroadly
acrossdomains--couldthenrapidlycatalyzeprogressacrossthetaskspace;thisissometimes
called“seedAGI”.TheconceptofAGIismorestrategicallyrelevanttotheextentthat(1)the                  34
conceptmapsontoaclusterofcapabilitiesthatcomeasabundle(likelyasaconsequenceof
32Mnih,Volodymyr,KorayKavukcuoglu,DavidSilver,AndreiA.Rusu,JoelVeness,MarcG.Bellemare,AlexGraves,etal.
“Human-LevelControlthroughDeepReinforcementLearning.” Nature518,no.7540(February2015):529–33.
https://doi.org/10.1038/nature14236 . See Figure 3, p. 531.
33 For many definitions of the task, but not all.
34Yudkowskyprovidesahelpfuloverviewoftheconceptofgeneralintelligencehere:Yudkowsky,Eliezer.“GeneralIntelligence.”
Arbital ,n.d. https://arbital.com/p/general_intelligence/ .SeealsoGoertzel,Ben.“Artificialgeneralintelligence:concept,stateof
theart,andfutureprospects.”JournalofArtificialGeneralIntelligence5.1(2014):1-48.NotethatAGI,asdefinedinthis
documentandbyYudkowskyandGoertzel,isconceptuallydistinctfrombroadhumanlevelcapabilities.Onecouldinprinciple
haveanAGIwithsub-(adult)-humanreasoning,ornon-AGIsystemswithmanysuperhumancapabilities.However,itdoesseem
plausiblethatinpracticeAGIwillbeanecessaryandsufficientconditionforhuman-levelcapabilitiesinnearlyalldomains,given
(1)theabilityofthegeneralintelligencetocallonalltheotherexistingsuperhumanassetsofmachineintelligence,and(2)the
vastarrayofproblemsseemingtorequiregeneralintelligence--thereisscarcedataandextremeinterdependencieswithother
domains--thatareotherwiseunlikelytobesolvedbynarrowAI.Notethat“human-levelAIinX”,“highhuman-levelAIinX”,and
“superhuman AI in X” can be used to characterize narrow AI systems.

19

AI Governance: A Research Agenda
generalreasoning),(2)AGIhastransformativeimplications,suchasignitingrapidgeneral
progress, (3) AGI arrives early in the sequence of transformative capabilities.  35

Wewouldliketoknowmoreabouttheprobablestrategicpropertiesofnovelcapabilitiesand
kindsofAI.Forexample,couldthey enhancecooperationbygivingadvice,bymediatingor
arbitratingdisputes,byidentifyinggains-from-cooperationamongststrangers?CouldAIand
cheapsurveillanceenablerobustmonitoringofcompliancetoagreements,andcryptographic
systemsthatprotectparticipantsfromexposingtheirsensitiveinformation?CouldAIenable
overcomingcommitmentproblems throughbindingcostlycommitmentsthatare          36
hard-codedintoAI-adjudicatedcontracts?TowhatextentwillAIenabledcapabilitiesbe
defense-biased (vsoffense-biased),definedhereascostingrelativelymoretoattackthanto
defend,foragivengoal?Broadlyspeaking,defense-biasedtechnologymakesamultipolar
worldmorestable.Towhatextentwillnewtechnologiesbe destruction-biased ,defined             37
hereasmakingitrelativelyeasytodestroyvalue(butpotentiallyhardtocapturevalue)?Do
newAIcapabilitiesprovide first-moveradvantages ,sothatactorshave(economicor
military)incentivestodevelopanddeploythemquickly?Anextremeformofpower              38
advantage,whichmaybemorelikelyfromfirst-moveradvantagesandoffensebias,is
decisivestrategicadvantage :anadvantagesufficientto“achievecompleteworld
domination”.Thestrategiccharacter,andtheperceivedstrategiccharacter,offuture            39
technologywillshapetheinternationallandscape,determininghowsecureorvulnerableare
greatpowersunderthestatusquo,andhowabletheyaretocooperatetoovercome
commitment problems and the security dilemma.

35Forexample,contra(1)itcouldbethatgeneralreasoningcomesindifferentflavors,andAIbecomesvastlysuperhumanat
someformswhilestillremainingsubhumanatothers.Contra(3),AGIcouldplausiblybemuchhardertoachievethan
super-surveillance AI, super-hacking AI, even super-inventing AI (e.g. with circuit design). (2) seems plausible.
36 Particularly between great powers, who otherwise lack a powerful legal structure within which to make commitments.
37Forexample,YannLeCunn(NYUEthicsofAIConference)statedthathebelievesnarrowdefensiveAIsystemswilldominate
generalAIsystems,becauseofspecialization;hislogicimpliesthatnarrowoffensiveAIsystemsshouldalsodominategeneralAI
systems,suggestingaworldwheregeneralAIcannotflourishwithoutmassivenarrowAIdefenses.EricSchmidt(2:52:59during
talkat FutureofWarConference )conjecturedthatcyber-AIsystemswilldominateonthedefense.(Fordiscussionofnear-term
defensevsoffensebias,seeBrundage,Miles,ShaharAvin,etal.“TheMaliciousUseofArtificialIntelligence:Forecasting,
Prevention, and Mitigation.” ArXiv:1802.07228 [Cs] , February 20, 2018.  http://arxiv.org/abs/1802.07228 .)
38Thereareseveralkindsoffirstmoveradvantage,suchasfromthefirsttoattack,orthefirsttodevelopacapability.Bothcan
beunderstoodasaformofoffensebias,thoughtherearesubtletiesindefinitionrelatedtowhatkindsofsymmetryare
presumed to be present.
39 Bostrom. Superintelligence ; p 96.

20

AI Governance: A Research Agenda

 ThesequestionsofthepotentialstrategicpropertiesofAIcanalsobeframedinamore
generalway.Towhatextentwill(particularkindsof)AIbe,orhavetheoptionofbeingmade
tobe, transparent , stabilizing/destabilizing , centralizing/decentralizingofpower,
politicallyvalenced(towardsauthoritarianismordemocracy),or wisdom-enhancing
(advisorAIs)?Howlikelyisitthatwewillgetsomeofthese(e.g.wisdomAI)beforeothers
(e.g. decisive cyber first strike AI)?

InresearchingthepossiblestrategiccapabilitiesofAI,wemustalsoaskhowfarour
estimatesandmodelscanbereliedupon.Willdevelopmentsbe predictable and
foreseeableintheircharacter?TowhatextentwillAIbe dualuse ,makingithardto
distinguish between the development, training, and deployment of
dangerous/destabilizing/militarysystemsandsafe/stabilizing/economicsystems?Towhat
extentwilldevelopmentsbepredictableintheirtimelinesandsequencing ,andwhatare
ourbestforecasts(see section2.3 )?Ifwecanestimateroughlywhenstrategicallyrelevantor
transformativethresholdsarelikelytobereached,orinwhatorder,thenwecanformulatea
better map of the coming transformations.
1.3 Other Strategic Technology
Manyothernoveltechnologiescouldplayastrategicortransformativerole.Itisworth
studyingthemtotheextentthattheyposetransformativepossibilitiesbeforeTAI,thatthey
shapekeyparametersofAIstrategyontheroadtoTAI,orthattheyrepresenttechnological
opportunitiesthatcouldbeunlockedbyasuperR&DAI.Thesetechnologiesinclude:
atomicallyprecisemanufacturing,cheapandagilerobotics,syntheticbiology,geneticand
cognitiveenhancement,cyber-innovationsanddependencies,quantum computing,
ubiquitousandpotentsurveillance,liedetection,andmilitarycapabilitiessuchasanti-missile
defense, hypersonic missiles, energy weapons, ubiquitous subsea sensor networks, etc.
2. Assessing AI Progress
Theprevioussection,MappingTechnicalPossibilities,triedtocreativelyenvisionlonger-run
transformativepossibilitiesandthecharacterofAI.Thissectionseekstobemorepreciseand
quantitativeaboutassessingexistingandfutureprogressinAI.Canweimproveour

21

AI Governance: A Research Agenda

 measuresofAIinputs,investments,andperformance?CanwemodelAIprogress:the
relationshipbetweenmeasurableinputsandindicators,andfutureAIinnovation?
Supplementingmodel-basedforecastswithexpertassessment,towhatextentcanwe
forecast AI progress?
2.1 Measuring Inputs, Capabilities, and Performance
Whatarethekeycategoriesof inputtoAIR&D,andcanwemeasuretheirexisting
distributionandratesofchange?PlausiblythekeyinputstoAIprogressarecomputing
power(compute),talent,data,insight,andmoney.Canwesensiblyoperationalizethese,or              40
findusefulproxiesforthem?WhatarethemostimportantAI capabilitiesthatweshouldbe
tracking?Canweconstructsensible,tractable,strategicallyrelevantmeasuresof
performance , thateithertrackorpredicttransformativecapabilities?Priorandexisting
metrics of performance are summarized and tracked by the Electronic Frontier Foundation. 41

Thismeasurementexerciseshouldbedisaggregatedatthelevelofthestrategicallyrelevant
actor.Whoarethemainorganizationsandcountriesinvolved,andwhatisthedistributionof
andratesofchangeintheirinputs,capabilities,andperformance?Laterinthisdocumentwe
willaskaboutthestrategicpropertiesoftheseorganizationsandcountries,suchastheir
institutionalconfiguration(e.g.legalstructure,leadershipselectionprocess),goals(political,
economic,other),andaccesstootherstrategicassets.Asarelativelypoorlyunderstoodand
potentiallypivotalactor,currentresearchisespeciallyseekingtobetterunderstandChina’s
inputs, capabilities, and performance.  42
2.2 Modeling AI Progress
Aswellasmappingtechnicalpossibilities,wewanttobeabletomodelprogressinAI
development towards these possibilities.
40Hwang,T.“ComputationalPower and the SocialImpact ofArtificialIntelligence.” 2018,1–44.
http://dx.doi.org/10.2139/ssrn.3147971 .Hilbert,M.,andP.López.“Theworld'stechnologicalcapacitytostore,communicate,
and compute information.” Science  332, issue 6025 (April 1, 2011). http://doi.org/10.1126/science.1200970 .
41 At https://www.eff.org/ai/metrics .
42E.g.onerelevantquestioniswhetherChinacanbecomeadecisiveworldleaderinAIwithoutbecomingmorescientifically
open.Wagner,CarolineS.,andKoenJonkers.“OpenCountriesHaveStrongScience.” NatureNews550,no.7674(October5,
2017). https://doi.org/10.1038/550032a ,p.32.ForanoverviewofChina’sAIlandscape,seeDing,Jeffrey.“DecipheringChina’s
AIDream:Thecontext,components,capabilities,andconsequencesofChina’sstrategytoleadtheworldinAI.”Futureof
Humanity Institute, March 2018. https://www.fhi.ox.ac.uk/wp-content/uploads/Deciphering_Chinas_AI-Dream.pdf .

22

AI Governance: A Research Agenda

Amodelingstrategyistolookforrobusttrendsinaparticularinput,suchascompute/$.                 43
ThecanonicalhardwaretrendisMoore’sLaw.KurzweilandNordhausobservean             44 45 46
impressivelyconsistentexponentialtrendincomputingperformancegivencost,beginning
beforeMoore’sLaw.Fromthissomearguethatthetrendwillcontinue. AIImpactsfindsthe                 47
recentdoublingtimeinFLOPS/$tobeabout3-5years,slowerthanthe25yeartrendof1.2
years.AmodeiandHernandeznotethatoverthepastfiveyearsthereappearstobean                 48
exponentialincreaseinthetotalcomputeusedtotrainleadingAIapplications,witha3.5
month doubling time.  49

Morecomplexapproachescouldtrytobuildcausaland/orpredictivemodelsofAIprogress
(onparticulardomains)asafunctionofinputsofcompute,talent,data,investment,time,and
indicatorssuchaspriorprogressandachievements(modelingthe“AIproductionfunction”).
Towhatextentdoesperformancescalewithtrainingtime,data,compute,orotherfungible
assets?Whatisthedistributionofbreakthroughsgiveninputs,andwhatistheexistingand                50
likelyfuturedistributionofinputs?Howquicklycantheseassetsbebought?Howeasyisitto
enter or leapfrog?

Modelingtheseinputsmayyieldinsightsonratesofprogressandthekeyfactorswhichslow
orexpeditethis.Whatdothesemodelsimplyforlikelybottlenecksinprogress?Doesitseem
43Suchanapproachworksbestwhenwecan(1)crediblyextrapolatethetrendintheinput(whichmaynotbetrueifthereisa
change in underlying dynamics), and (2) can map the input to outcomes that matter (which may not be true for lots of reasons).
44Fortheoriginalpaper,seeMoore,G.E.“CrammingMoreComponentsOntoIntegratedCircuits.” Electronics38,no.8(April19,
1965):82–85. https://doi.org/10.1109/JPROC.1998.658762 .Cf.alsoSchaller,R.R.“Moore’sLaw:Past,PresentandFuture.”
IEEESpectrum34,no.6(June1997):52–59. https://doi.org/10.1109/6.591665 ;“Trendsinthecostofcomputing.” AIImpacts ,
March 10, 2015. https://aiimpacts.org/trends-in-the-cost-of-computing/ .
45Kurzweil, Ray. “The Law of Accelerating Returns.” Kurzweilai (blog), March 7, 2001.
http://www.kurzweilai.net/the-law-of-accelerating-returns .
46Nordhaus,WilliamD.“Areweapproachinganeconomicsingularity?Informationtechnologyandthefutureofeconomic
growth.” Cowles Foundation Discussion Paper No. 2021, September 2015. Figure 1,
https://cowles.yale.edu/sites/default/files/files/pub/d20/d2021.pdf .
47 Kurzweil’s data shows the trend beginning in 1900, Nordhaus’s data shows the trend beginning in 1940.
48Grace, Katja. “Recent Trend in the Cost of Computing.” AI Impacts,November 11,2017.
https://aiimpacts.org/recent-trend-in-the-cost-of-computing/ .Doublingtimeisequaltotimetoa10xincreasedividedby3.3
(because log 2 (10)=3.3).
49Amodei,Dario,and Danny Hernandez.“AI and Compute.” OpenAI (blog),May 16,2018.
https://blog.openai.com/ai-and-compute/ .
50Silver,D.,A.Huang,C.J.Maddison,A.Guez,andL.Sifre.“MasteringthegameofGowithdeepneuralnetworksandtreesearch.”
Nature 529 (January 28, 2016). http://doi.org/10.1038/nature16961 .

23

AI Governance: A Research Agenda
 likelythatwewillexperiencehardwareorinsightoverhang?Putdifferently,howprobable              51
isitthatacrucialinputwillsuddenlyincrease,suchaswithalgorithmicbreakthroughs,
implyingagreaterprobabilityofrapidprogress?Moregenerally,fromtheperspectiveof
developers, how smooth or sudden will progress be?

ArticulatingtheoreticallyinformedpredictionsaboutAIprogresswillhelpustoupdateour
modelsofAIprogressasevidencearrives.Thestatusquoinvolvesexpertsoccasionally
makingad-hocpredictions,beingcorrectormistakenbyunquantifiedamounts,andthen
possiblyupdatinginformally.Amorescientificapproachwouldbeonewhereexplicit
theories,oratleastschoolsofthought,mademanytestableandcomparablepredictions,
whichcouldthenbeevaluatedovertime.Forexample,canwebuildamodelthatpredicts
timeuntilsuper-humanperformanceatatask,givenpriorperformanceandtrendsininputs?
Givensuchamodel,wecouldrefineittoassessthekindsoftasksandcontextswhereitis
likelytomakeespeciallygoodorbadpredictions.Fromthiscanwelearnsomethingabout
thesizeofthehumanrangeinintelligencespace,fordifferentkindsoftasks?Ifourmodels                  52
areaccuratethenwewouldhaveausefulforecastingtool;iftheyarenotthenwewillhave
hardevidenceofourignorance.Weshouldbuildmodelspredictingotherstrategically
relevantparameters,suchastheratiooftrainingcomputecoststoinference/execution
computecosts,ormoregenerallytheratioofR&Dcoststoexecutioncosts(costsofrunning
the system).  53
2.3 Forecasting AI Progress
Usingtheabovemeasurementsandmodels,andwithexpertjudgment,towhatextentcanwe
forecastthedevelopmentofAI(inputsandperformance)?Thereareseveraldesideratafor
goodforecastingtargets.Givensuchforecastingeffortswecouldask,howwellcalibratedand
accuratearedifferentgroupsofexpertsandmodelsfordifferentkindsofforecasting
problems?Howbestcanweelicit,adjust,andaggregateexpertjudgment?Howdifferentare
51Forexample,arelativelysimplealgorithmictweakgeneratedmassiveimprovementsinAtarigameplaying.Itisplausiblethat
therearemanyothersucheasilyimplementablealgorithmictweaksthatanAIcoulduncoverandimplement.Bellemare,MarcG.,
WillDabney,andRémiMunos.“Adistributionalperspectiveonreinforcementlearning.” ArXiv:1707.06887[Cs] ,July21,2017.
https://arxiv.org/abs/1707.06887 .
52Alexander,Scott.“WhereTheFallingEinsteinMeetsTheRisingMouse.” SlateStarCodex (blog),August3,2017.
http://slatestarcodex.com/2017/08/02/where-the-falling-einstein-meets-the-rising-mouse/ .
53Theseparametersarerelevanttothescaleofdeploymentofanewsystem,topredictingthekindsofactorsandinitiatives
likely to be innovating in various domains, and to other aspects of AI governance.

24

AI Governance: A Research Agenda

 theproblemsofnear-termandlong-termforecasting,andtowhatextentcanweuselessons
from or performance in near-term forecasting to improve long-term forecasts?

Near-termforecastingworkiscurrentlybeingdonebyMetaculus.Manysurveyshaveasked              54
untrainedanduncalibratedexpertsaboutnear-andlong-termforecasts.Itcanalsobe
productivetoevaluatepreviousforecastingefforts,toseehowwellcalibratedtheyare,andif
there are conditions that make them more or less accurate. 55
3. AI Safety  56
3.1 The Problem of AI Safety
AISafetyfocusesonthetechnicalchallengeofbuildingadvancedAIsystemsthataresafeand
beneficial.Justastodayalotofengineeringeffortgoesintoensuringthesafetyofdeployed
systems--makingsurebridgesdon’tfalldown,carbrakesdon’tfail,hospitalprocedures
administerthecorrectmedicationstopatients,nuclearpowerplantsdon’tmelt,andnuclear
bombsdon’tunintentionallyexplode--soitisplausiblethatsubstantialeffortwillbe             57
required to ensure the safety of advanced and powerful AI systems.

RelativelysimpleAIsystemsareatriskofaccidentsandunanticipatedbehavior.Classifiers
areknowntobefragileandvulnerabletosubtleadversarialattacks.Onemilitaryreport
characterizesAI(specificallydeeplearning)as“weakonthe‘ilities’”,whichinclude             58
reliability,maintainability,accountability,verifiability,debug-ability,fragility,attackability.
AIcanbehaveinamannerthatisnotforeseenorintended,asillustratedbyMicrosoft’s
failuretoanticipatetherisksofits“Tay”chatbotlearningfromTwitteruserstomake
54 At https://www.metaculus.com/questions/ .
55Muehlhauser, Luke. “Retrospective Analysis of Long-Term Forecasts”.
https://osf.io/ms5qw/register/564d31db8c5e4a7c9694b2be .
56Weuse AISafety torefertothedistinct,specializedfieldfocusingontechnicalaspectsofbuildingbeneficialandsafeAI.AI
SafetyandAIGovernancecanbeusedasexclusive(andexhaustive)categoriesfortheworkneededtobuildbeneficialAI.This
agenda summarizes the aspects of AI Safety especially relevant to AI Governance.
57Thougheventhisoftenrequiresconsiderableorganizationalefforts,andinvolvesmanyclosecalls;cf.Sagan,ScottD. The
LimitsofSafety:Organizations,Accidents,andNuclearWeapons .Princeton:PrincetonUniversityPress,1993;Schlosser,Eric.
CommandandControl:NuclearWeapons,theDamascusAccident,andtheIllusionofSafety .Reprintedition.NewYork:Penguin
Books, 2014.
58Potember,Richard.“PerspectivesonResearchinArtificialIntelligenceandArtificialGeneralIntelligenceRelevanttoDoD.”
JASON - The MITRE Corporation, 2017.  https://fas.org/irp/agency/dod/jason/ai-dod.pdf , p. 2.

25

AI Governance: A Research Agenda
offensivestatements.Complexsystems,especiallywhenfast-movingandtightlycoupled,can
leadtoemergentbehaviorand‘normalaccidents’.The2010“flashcrash”isillustrative,in               59
whichautomatedtradingalgorithmsproduced20,000“erroneoustrades”andasudden
trilliondollardeclineinUSfinancialmarketvalue;thisundesiredbehaviorwasstoppednot
by real-time human intervention but by automated safety mechanisms.  60

ThepreviouskindsofaccidentsarisebecausetheAIis“toodumb”.MoreadvancedAI
systemswillovercomesomeoftheserisks,butgainanewkindofaccidentriskfrombeing
“tooclever”.Inthesecasesapowerfuloptimizationprocessfinds“solutions”thatthe
researchersdidnotintend,andthatmaybeharmful.Anecdotesaboundaboutthe              61
surprisingroutesbywhichartificiallife“findsaway”,fromaboat-racingAIthat
reward-hackedbydrivingincircles,toageneticalgorithmintendedtoevolveanoscillating               62
circuitthathackeditshardwaretofunctionasareceiverforradiowavesfromnearby
computers,toatic-tac-toealgorithmthatlearnedtodefeatitsrivalsusingamemorybomb.               63
  64
59Perrow,Charles. Normalaccidents:Livingwithhighrisktechnologies .Princeton,NJ:PrincetonUniversityPress,2011;for
discussionsofnormalaccidentsinAIsystems,see:Maas,Matthijs.“Regulatingfor‘NormalAIAccidents’—OperationalLessons
fortheResponsibleGovernanceofAIDeployment.”NewOrleans:AssociationfortheAdvancementofArtificialIntelligence,
2018. http://www.aies-conference.com/wp-content/papers/main/AIES_2018_paper_118.pdf ;andfornormalaccidentsinthe
specificcontextofmilitaryweaponsystems,seeDanzig,Richard.“TechnologyRoulette:ManagingLossofControlasMany
Militaries Pursue Technological Superiority.” Center for a New American Security, June 2018.
https://s3.amazonaws.com/files.cnas.org/documents/CNASReport-Technology-Roulette-DoSproof2v2.pdf?mtime=2018062807
210 ;Scharre,Paul. ArmyofNone:AutonomousWeaponsandtheFutureofWar .NewYork:W.W.Norton&Company,2018,pp.
150-155; Scharre,Paul.“AutonomousWeaponsandOperationalRisk.”EthicalAutonomyProject.20YYFutureofWarfare
Initiative. Center for a New American Security, 2016.
https://s3.amazonaws.com/files.cnas.org/documents/CNAS_Autonomous-weapons-operational-risk.pdf ;Borrie,J.“Safety,
UnintentionalRiskandAccidentsintheWeaponizationofIncreasinglyAutonomousTechnologies.”UNIDIRResources.UNIDIR,
2016.  http://www.unidir.org/files/publications/pdfs/safety-unintentional-risk-and-accidents-en-668.pdf .
60U.S.CommodityFuturesTradingCommission,andU.S.Securities&ExchangeCommission.“FindingsRegardingtheMarket
EventsofMay6,2010:ReportoftheStaffsoftheCFTCandSECtotheJointAdvisoryCommitteeonEmergingRegulatoryIssues.”
2010. https://www.sec.gov/news/studies/2010/marketevents-report.pdf ,p.104.Linton,Oliver,andSoheilMahmoodzadeh.
“Implications ofhigh-frequency trading for security markets.” AnnualReview ofEconomics10 (2018).
https://www.annualreviews.org/doi/pdf/10.1146/annurev-economics-063016-104407 .
61StuartRussellwrites:“Asystemthatisoptimizingafunctionofnvariables,wheretheobjectivedependsonasubsetofsize
k<n,willoftensettheremainingunconstrainedvariablestoextremevalues;ifoneofthoseunconstrainedvariablesisactually
somethingwecareabout,thesolutionfoundmaybehighlyundesirable.Thisisessentiallytheoldstoryofthegenieinthelamp,
orthesorcerer’sapprentice,orKingMidas:yougetexactlywhatyouaskfor,notwhatyouwant.Ahighlycapabledecisionmaker
–especiallyoneconnectedthroughtheInternettoalltheworld’sinformationandbillionsofscreensandmostofour
infrastructure–canhaveanirreversibleimpactonhumanity.”Russell,Stuart.“OfMythsAndMoonshine.” Edge ,2014.
https://www.edge.org/conversation/the-myth-of-ai#26015 .
62Amodei,Dario,and Jack Clark.“Faulty Reward Functions in the Wild.” OpenAI(blog),2016.
https://openai.com/blog/faulty-reward-functions/ .
63Bird,JonandPaulLayzell.“TheEvolvedRadioanditsImplicationsforModellingtheEvolutionofNovelSensors.” Proceedings
of the 2002 Congress on Evolutionary Computation , CEC'02 (Cat. No.02TH8600), 2002. See
https://people.duke.edu/~ng46/topics/evolved-radio.pdf .
64ManyotherexamplescanbefoundinLehman,Joeletal.“TheSurprisingCreativityofDigitalEvolution.” ArXiv:1803.03453[Cs] ,
March 29, 2018. https://arxiv.org/abs/1803.03453 .

26

AI Governance: A Research Agenda

FutureadvancesinAIwillposeadditionalrisks(aswellasofferadditionalopportunitiesfor
safety).AIsystemswillbedeployedinevermorecomplexandconsequentialdomains.They
willbemoreintelligentatparticulartasks,whichcouldunderminethevalueofhuman
oversight.Theywillbegintoacquiremodelsoftheirenvironment,andofthehumansand
institutionstheyinteractwith;theywillgainunderstandingofhumanmotivation,beableto
observeandinferhumanaffect,andbecomemorecapableofpersuasionanddeception.As
systemsscaletoandbeyondhuman-level(inparticulardimensions),theymayincreasingly
be able to intelligently out-maneuver human built control systems.

Thisproblemisanalogoustotheproblemofalignmentincapitalism(of“avoidingmarket
failures”):howtobuildalegalandregulatoryenvironmentsothattheprofitmotiveleads
firmstoproducesocialvalue.Historyisrepletewithexamplesoflargenegativeexternalities
causedbyfirmsperverselyoptimizingforprofit,fromfraudulentprofitsproducedthrough
‘creative’accounting(e.g.Enron),topoliciesthatriskdisasterorgeneratepollution(e.g.
DeepwaterHorizonoilspill),tofirmsthatactivelydeceivetheirinvestors(e.g.Theranos)or
regulators(e.g.Volkswagenemissionsscandal).Thesescandalsoccurdespitetheexistenceof
informedhumanswith“commonsense”withinthecorporations,andthegovernance
institutionsbeingcapableofcomparableintelligence.PowerfulAIsystemsmaylackeventhat
commonsense,andcouldconceivablybemuchmoreintelligentthantheirgoverning
institutions.
3.2 AI Safety as a Field of Inquiry
MuchoftheinitialthinkingaboutAIsafetywasfocusedonthechallengeofmaking
hypothesizedhuman-levelorsuperhumanAIsafe.Thislineofinquiryledtoanumberof
important insights. 65
1.Orthogonality thesis: Intelligent systems could be used to pursue any value system. 66
65Bostrom. Superintelligence ;Yudkowsky,Eliezer.“ArtificialIntelligenceasaPositiveandNegativeFactorinGlobalRisk.”In
Global Catastrophic Risks , edited by Nick Bostrom and Milan M. Cirkovic, pp. 308–45. New York: Oxford University Press, 2008.
66 Bostrom. Superintelligence , pp. 105-108.

27

AI Governance: A Research Agenda
2.Instrumentalconvergence:systemswillhaveinstrumentalreasonsforacquiring
powerandresources,maintaininggoalintegrity,andincreasingitscapabilitiesand
intelligence.
3.Empiricalsafetytestsmaynotbesufficient.Humanoverseersmaynothavethe
capacitytorecognizeproblemsduetothesystem’scomplexity,butalsoitsabilityto
intelligently model and game the oversight mechanism.
4.Formalizinghumanpreferencesishard.Whensuchaformalstatementisfedasthe
goalintopowerfulandintelligentsystems,theyarepronetofailinextremeways.
ThisfailuremodeisatropeinWesternliterature,asperMidas’curse,theSorcerer’s
Apprentice, the Monkey’s Paw, and the maxim to be careful what one wishes for.  67
5.Therearemanywayscontroloralignmentschemescouldcatastrophicallyand
irreversibly fail, and among the most dangerous are those we haven’t thought of yet.

Theaboveframingadoptsthelensof AIaccidentrisks :therisksofundesiredoutcomes
arisingfromaparticular,intentionallydesigned,AIsystem(oftenhighlyintelligent).Thereis
another,relativelyneglectedframing,of AIsystemicrisks :therisksofundesired
outcomes--someofwhichmaybeverytraditional--thatcanemergefromasystemof
competingandcooperatingagentsandcanbeamplifiedbynovelformsofAI.Forexample,AI
couldincreasetheriskofinadvertentnuclearwar,notbecauseofan accidentor misuse ,but
becauseofhowAIcouldrapidlyshiftcrucialstrategicparameters,beforeweareabletobuild
up compensating understandings, norms, and institutions.  68

AIsafetycanthusbeunderstoodasthetechnicalfieldworkingonbuildingtechniquesto
reducetherisksfromadvancedAI.Thisincludestheultimategoalsofsafetyandalignmentof
superintelligentsystems,theintermediategoalsofreducingaccident,misuse,andemergent
risksfromadvancedsystems,aswellasnear-termapplicationssuchasbuildingself-driving
car algorithms that are sufficiently safe, including being resilient to en-masse terrorist hacks.
67SeeSoares,Nate.“EnsuringSmarter-than-HumanIntelligencehasaPositiveOutcome.”TalksatGoogleseries,November20,
2016. https://www.youtube.com/watch?v=dY3zDvoLoao .Bostrom. Superintelligence ,Chapter9. Yudkowsky,Eliezer.
“Difficulties of AGI Alignment.” The Ethics of Artificial Intelligence Conference,NYU ,2016.
https://livestream.com/nyu-tv/ethicsofAI/videos/138893593 .
68Horowitz,MichaelC.,PaulScharre,andAlexVelez-Green.“AStableNuclearFuture?TheImpactofAutonomousSystemsand
ArtificialIntelligence.”WorkingPaper,December2017;Geist,Edward,andAndrewJLohn.“HowMightArtificialIntelligence
Affect the Risk of Nuclear War?” RAND, 2018.  https://www.rand.org/pubs/perspectives/PE296.html .

28

AI Governance: A Research Agenda

Asevidenceoftheimportanceofthisfield,whenAIresearchersweresurveyedaboutthe
likelyoutcomeofsuper-humanAI,thoughthemajoritybelieveitisisverylikelytobe
beneficial,themajorityofrespondentsassignatleasta15%chancethatsuperhumanAI
wouldbe“onbalancebad”orworse,andatleasta5%chanceitwouldbe“extremelybad(e.g.
humanextinction)”.ThegoalofAIsafetyistoprovidetechnicalinsights,tools,andsolutions                69
for reducing the risk of bad, and especially extremely bad, outcomes.

AsAIsystemsaredeployedinevermoresafety-criticalandconsequentialsituations,AI
researchersanddeveloperswillincreasinglyconfrontsafety,ethical,andotherchallenges.
Somesolutionstothesechallengeswillbeone-off,localpatches.Forexample,Google’s
solutiontomisclassifyingimagesofblackpeopleas“gorillas”wastosimplyremove“gorilla”
andsimilarprimatecategoriesfromitsservice.Thiskindofpatchwillnotscaleor                70
generalize.

Wewouldprefertofindsolutionsthataremorefoundationalorgeneralizable,andthusmore
plausiblycontributeto scalably safeandbeneficialAI.Broadly,forparticularsystemswewill
wantthemtohavevariousdesirableproperties,suchasthefollowing(drawingfromEveritt
et al’s 2018 framework):
❖ReliabilityandSecurity ,sothatthesystembehavesasintendedinawiderangeof
situations, including under adversarial attack. 71
❖Corrigibility ,sothatthesystemisoptimallyopentobeingcorrectedbyahuman
overseerifitisnotperfectlyspecified/trained.Candidatemethodsincludeby            72
69 Grace et al. (2017), “When Will AI Exceed Human Performance? Evidence from AI Experts.”
70Simonite,Tom.“WhenItComestoGorillas,GooglePhotosRemainsBlind.”WIRED,January11,2018.
https://www.wired.com/story/when-it-comes-to-gorillas-google-photos-remains-blind/ .
71 Adversarial examples: Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy (2014). “Explaining and Har-
nessingAdversarialExamples.” ArXiv:1412.6572[Stat], March20,2015. https://arxiv.org/abs/1412.6572 .Athalye,Anish,Logan
Engstrom,AndrewIlyas,andKevinKwok.“SynthesizingRobustAdversarialExamples.” ArXiv:1707.07397[Cs] ,July24,2017.
http://arxiv.org/abs/1707.07397 .Brown,TomB.,DandelionMané,AurkoRoy,MartínAbadi,andJustinGilmer.“Adversarial
Patch.” ArXiv:1712.09665[Cs] ,December27,2017. http://arxiv.org/abs/1712.09665 .Nguyen,Anh,JasonYosinski,andJeff
Clune.“DeepNeuralNetworksAreEasilyFooled:HighConfidencePredictionsforUnrecognizableImages.” ArXiv:1412.1897[Cs] ,
December 5, 2014.  http://arxiv.org/abs/1412.1897 .
72NateSoares,BenjaFallenstein,EliezerYudkowsky,andStuartArmstrong.“Corrigibility.”AAAI2015EthicsandArtificial
Intelligence Workshop, 2015. https://intelligence.org/files/Corrigibility.pdf .

29

AI Governance: A Research Agenda
makingtheagentindifferenttoorignorantofinterventions,oruncertainaboutthe              73
reward function.  74
❖Intelligibility ,interpretability,andtransparency,suchasthroughdimensionality
reduction,naturallanguagecommunication,andtechniquesforvisualizingor          75
otherwise understanding what features parts of the learned algorithm are encoding.   76
❖Valuespecification ,relatedtoalignment,formalizingcurrentethicalprinciples,          77 78
inversereinforcementlearningandlearninghumanpreferences,overcoming         79
reward corruption, and measuring and minimizing extreme side effects. 80 81
❖Limitingcapabilities ,suchasthroughboxing,preferring‘oracle’AIs,orbuildingAI             82
services rather than general AI agents. 83
❖Performanceandsafetyguarantees ,suchasformalverificationtoidentifyupper
bounds on the probability of unsafe behaviour or restrictions on exploration policies.

73Orseau, Laurent, and Stuart Armstrong. “Safely Interruptible Agents.” October 28, 2016.
https://intelligence.org/files/Interruptibility.pdf ;Everitt,Tom,DanielFilan,MayankDaswani,andMarcusHutter.
“Self-ModificationofPolicyandUtilityFunctioninRationalAgents.” ArXiv:1605.03142[Cs] ,May10,2016.
http://arxiv.org/abs/1605.03142 ;Armstrong,Stuart,andXavierO’Rourke.“‘Indifference’MethodsforManagingAgent
Rewards.” ArXiv:1712.06365 [Cs] , December 18, 2017.  http://arxiv.org/abs/1712.06365 .
74Hadfield-Menell,Dylan,SmithaMilli,PieterAbbeel,StuartRussell,andAncaDragan.“InverseRewardDesign.”
ArXiv:1711.02827 [Cs] , November 7, 2017.  http://arxiv.org/abs/1711.02827 .
75Mnih,Volodymyr,KorayKavukcuoglu,DavidSilver,AndreiA.Rusu,JoelVeness,MarcG.Bellemare,AlexGraves,etal.
“Human-LevelControlthroughDeepReinforcementLearning.” Nature518,no.7540(February26,2015):529–33.
https://doi.org/10.1038/nature14236 .
76Olah,Chris,AlexanderMordvintsev,andLudwigSchubert.“FeatureVisualization.” Distill2,no.11(November7,2017):e7.
https://doi.org/10.23915/distill.00007 ;Olah,Chris,ArvindSatyanarayan,IanJohnson,ShanCarter,LudwigSchubert,Katherine
Ye,andAlexanderMordvintsev.“TheBuildingBlocksofInterpretability.” Distill3,no.3(March6,2018):e10.
https://doi.org/10.23915/distill.00010 .
77 Bostrom. Superintelligence .
78Hardt,Moritz,EricPrice,andNathanSrebro.“EqualityofOpportunityinSupervisedLearning.” ArXiv:1610.02413[Cs] ,October
7,2016. http://arxiv.org/abs/1610.02413 ;Baum,SethD.“SocialChoiceEthicsinArtificialIntelligence.”SSRNScholarlyPaper.
Rochester, NY: Social Science Research Network, October 2, 2017. https://papers.ssrn.com/abstract=3046725 .
79Christiano,Paul,JanLeike,TomB.Brown,MiljanMartic,ShaneLegg,andDarioAmodei.“DeepReinforcementLearningfrom
HumanPreferences.” ArXiv:1706.03741[Cs,Stat] ,June12,2017. http://arxiv.org/abs/1706.03741 ;Hadfield-Menell,Dylan,Anca
Dragan,PieterAbbeel,andStuartRussell.“CooperativeInverseReinforcementLearning.” ArXiv:1606.03137[Cs] ,June9,2016.
http://arxiv.org/abs/1606.03137 ;Evans,Owain,AndreasStuhlmueller,andNoahD.Goodman.“LearningthePreferencesof
Ignorant,InconsistentAgents.” ArXiv:1512.05832[Cs] ,December17,2015. http://arxiv.org/abs/1512.05832 ;Choi,Jaedeug,and
Kee-EungKim.“InverseReinforcementLearninginPartiallyObservableEnvironments.” JournalofMachineLearningResearch12
(2011):691–730. http://www.jmlr.org/papers/volume12/choi11a/choi11a.pdf;Ng,AndrewY.,andStuartJ.Russell.
“AlgorithmsforInverseReinforcementLearning.”In ProceedingsoftheSeventeenthInternationalConferenceonMachine
Learning , 663–70, 2000.  https://ai.stanford.edu/~ang/papers/icml00-irl.pdf .
80Everitt,Tom,VictoriaKrakovna,LaurentOrseau,MarcusHutter,andShaneLegg.“ReinforcementLearningwithaCorrupted
Reward Channel.” ArXiv:1705.08417 [Cs] , May 23, 2017.  http://arxiv.org/abs/1705.08417 .
81Krakovna,Victoria,LaurentOrseau,MiljanMartic,andShaneLegg.“MeasuringandAvoidingSideEffectsUsingRelative
Reachability.” ArXiv:1806.01186 [Cs, Stat] , June 4, 2018. http://arxiv.org/abs/1806.01186 .
82 Bostrom. Superintelligence , p. 145.
83Drexler,Eric.“Development-orientedmodelsofsuperintelligence:Overviewandtopicaldocuments.”Workinprogress.
http://bit.ly/DrexlerAI .

30

AI Governance: A Research Agenda
 Tosomeextenttheseapproachescanbetrialedanddevelopedinconcretenear-term
settings. 84
3.3 The Implications of AI Safety for AI Governance
ForthepurposesofAIgovernanceitisimportantthatweunderstandthestrategic
parametersrelevanttobuildingsafeAIsystems,includingtheviability,constraints,costs,and
propertiesofscalablysafesystems.Whatisthe safetyproductionfunction ,whichmapsthe
impactofvariousinputsonsafety?Plausibleinputsarecompute,money,talent,evaluation
time,constraintsontheactuators,speed,generality,orcapabilityofthedeployedsystem,and
normsandinstitutionsconducivetoriskreporting.Towhatextentdoweneedtospendtime
orresourcesatvariousstagesofdevelopment(suchasearlyorlate)inordertoachieve
safety?Ifthesafety-performancetrade-offsaremodest,andpoliticaloreconomicreturnsto
absoluteandrelativeperformancearerelativelyinelastic(marginalimprovementsin
performancearenotthatimportant),thenachievingsafeAIsystemsismorelikelytobe
manageable;theworldwillnothavetoresorttoradicalinstitutionalinnovationorother
extremestepstoachievebeneficialAI.If,however,thesafety-performancetrade-offissteep,
orpoliticaloreconomicreturnsarehighlyelasticinabsoluteorespeciallyrelative
performance,thenthegovernanceproblemwillbemuchhardertosolve,andmayrequire
more extreme solutions.

Thereareabroadrangeofimplicitviewsabouthowtechnicallyharditwillbetomakesafe
advancedAIsystems.TheydifferonthetechnicaldifficultyofsafeadvancedAIsystems,as
wellasrisksofcatastrophe,andrationalityofregulatorysystems.Wemightcharacterize
them as follows:

❖Easy :Wecan,withhighreliability,preventcatastrophicriskswithmodesteffort,say
1-10% of the costs of developing the system.
84Amodei,Dario,ChrisOlah,JacobSteinhardt,PaulChristiano,JohnSchulman,andDanMané.“ConcreteProblemsinAISafety.”
ArXiv:1606.06565[Cs] ,June21,2016. http://arxiv.org/abs/1606.06565 ;Leike,Jan,MiljanMartic,VictoriaKrakovna,PedroA.
Ortega,TomEveritt,AndrewLefrancq,LaurentOrseau,andShaneLegg.“AISafetyGridworlds.” ArXiv:1711.09883[Cs] ,
November 27, 2017.  http://arxiv.org/abs/1711.09883 .

31

AI Governance: A Research Agenda
❖Medium: Reliablybuildingsafepowerfulsystems,whetheritbenuclearpowerplants
oradvancedAIsystems,ischallenging.Doingsocostsperhaps10%to100%thecost
of the system (measured in the most appropriate metric, such as money, time, etc.).
➢Butincentivesarealigned .Economicincentivesarealignedsothat
companiesororganizationswillhavecorrectincentivestobuildsufficiently
safesystems.Companiesdon’twanttobuildbridgesthatfalldown,ornuclear
power plants that experience a meltdown.
➢Butincentiveswillbealigned .Economicincentivesarenotperfectlyaligned
today,aswehaveseenwithvariousscandals(oilspills,emissionsfraud,
financialfraud),buttheywillbeafterafewaccidentsleadtoconsumer
pressure, litigation, or regulatory or other responses. 85
➢Butwewillmuddlethrough .Incentivesarenotaligned,andwillneverbe
fully.However,wewillprobablymuddlethrough(gettheriskssmallenough),
as humanity has done with nuclear weapons and nuclear energy.
➢Andotherfactorswillstronglyworkagainstsafety . Strongprofitand
powerincentives,misperception,heterogenoustheoriesofsafety,
overconfidenceandrationalization,andotherpathologiesconspiretodeprive
usofthenecessarypatienceandhumilitytogetitright.Thisviewismost
likelyiftherewillnotbeevidence(suchasrecoverableaccidents)from
recklessdevelopment,andifthesafetyfunctionissteepovermediumlevelof
inputs(“Thiswouldnotbeahardproblemifwehadtwoyearstoworkonit,
once we have the system. It will be almost impossible if we don’t.”).
❖HardorNearImpossible :Buildingasafesuperintelligenceislikebuildingarocket
andspacecraftforamoon-landing,withouteverhavingdoneatestlaunch.Itcosts               86
greater than, or much greater than, 100% of development costs.
❖We don’t  know .

Willwebeabletocorrectlydiagnosethecharacterofthestepsrequiredforsufficientsafety,
indevelopmentanddeployment?Willwebeabletoagreeonacommonsafetypolicy?Will
85  This assumes that recoverable accidents occur with sufficient probability before non-recoverable accidents.
86Yudkowsky,Eliezer.“SoFar:UnfriendlyAIEdition.”EconLog|LibraryofEconomicsandLiberty,2016.
http://econlog.econlib.org/archives/2016/03/so_far_unfriend.html .

32

AI Governance: A Research Agenda
webeabletoverifycompliancewiththatpolicy?Forexample,woulditbepossibleto
separateamachine’sobjectivesfromitscapabilities,asdoingsocouldmakeiteasierfor
non-expertstopoliticallyevaluateasystemandcouldenableverificationschemesthatleak
fewer technical secrets (related to capabilities)?

Greaterinsightintothecharacterofthesafetyproblemwillshedinsightintoanumberof
parametersrelevanttosolvingthegovernanceproblem.Somegovernancearrangementsthat
could depend on the character of the safety problem include:
❖Providing incentives and protections for whistleblowers
❖Representation of AI scientists in decision making
❖Technical verification of some properties of systems
❖Explicit negotiations over the goals of the system

AISafetyworkisbeingdoneatanumberoforganizations,includingDeepMind,OpenAI,
GoogleBrain,theCenterforHumanCompatibleAIandUCBerkeley,theMachineIntelligence
Research Institute, the Future of Humanity Institute, and elsewhere.

33

AI Governance: A Research Agenda

 AI Politics
AIwilltransformthenatureofwealthandpower.Theinterestsandcapabilitiesofpowerful
actorswillbebuffeted,andnewpowerfulactorsmayemerge.Theseactorswillcompeteand
cooperatetoadvancetheirinterests.AdvancedAIislikelytomassivelyincreasethepotential
gainsfromcooperation,andpotentiallossesfromnon-cooperation;wethuswantpolitical
dynamicstobesuchastobemostlikelytoidentifyopportunitiesformutualbenefitandto
identify far in advance joint risks that could be avoided by prudent policy.

Politicaldynamicscouldalsoposecatastrophicrisksshortofhuman-levelAIif,forexample,
theyleadtogreatpowerwarorpromoteoppressivetotalitarianism.Politicaldynamicswill
affectwhatconsiderationswillbemostinfluentialinthedevelopmentof(transformative)AI:
corporateprofit,reflexivepublicopinion,researchers’ethicsandvalues,nationalwealth,
nationalsecurity,stickyinternationalarrangements,orenlightenedhumaninterest.Itisthus
critical that we seek to understand, and if possible, beneficially guide, political dynamics.

AIPoliticslooksathowthechangingtechnicallandscapecouldtransform domesticand
masspolitics , internationalpoliticaleconomy ,and internationalsecurity ,andinturn
howpoliciesbypowerfulactorscouldshapethedevelopmentofAI.Workinthiscluster
benefitsfromexpertiseindomesticpolitics,internationalrelations,andnationalsecurity,
amongotherareas.Itwillinvolvearangeofapproaches,includingtheory(mathematicaland
informal),contemporarycasestudies,historicalcasestudies,closecontactwithandstudyof
the relevant actors, quantitative measurement and statistical analysis, and scenario planning.
4. Domestic and Mass Politics
AIhasthepotentialtoshape,andbeshapedby,domesticandmasspolitics.AsAIandrelated
technologiesalterthedistributionofdomesticpower, formsofgovernmentwillalter.This
couldmeanashiftinpowertowardsactorswiththecapitalandauthoritytodeploypowerful
AIsystems,suchaselites,corporations,andgovernments.Ontheotherhand,AIcouldbe
usedtoenhancedemocracy,forexamplethroughalignedpersonaldigitalassistants,
surveillancearchitecturesthatincreasetheaccountabilityofauthorities,ordecentralized

34

AI Governance: A Research Agenda

 (crypto-economic)coordinationtechnologies.Theimpactofexacerbated inequalityandjob
displacementontrendssuchasliberalism,democracy,andglobalizationcouldbe
substantial.Whatsystemsarepossibleformitigatinginequalityandjobdisplacement,and
willtheybesufficient?Moregenerally, publicopinioncanbeapowerfulforcewhenitis
mobilized.Canweforeseethecontoursofhowpublicopinionislikelytobeactivatedand
expressed?Willcertaingroups--cultures,religions,economicclasses,demographic
categories--havedistinctperspectivesonAIpolitics?Thissetofquestionsisgenerallyless
relevant to short timelines (e.g. AGI comes within 10 years).
4.1 Forms of Government
Domesticpoliticalstructures,suchaswhetheragovernmentis accountablethrough
electionsandis transparentthroughpubliclegislativedebatesandaninformedfreepress,
ariseasacomplexfunctionofmanyfactors,someofwhichwillplausiblybealteredby
advancedAI.Somefactorsthatseemespeciallyimportanttodeterminingthecharacterof
government,andinparticulartheextenttowhichitisliberalanddemocratic,are:1)the
(unequal)distributionofcontrolovereconomicassetsandwealth;(2)surveillance
technologiesandarchitectures;(3)repressiontechnologies;(4)persuasiontechnologies;(5)
personal advisor technologies; (6) collective action technologies.

ResearchonformsofgovernmentwillexamineplausibleAI-driventrendsintheseandother
factors,andevaluatepossiblestrategiesformitigatingadversetrends.Thismattersfor
extremestakesbecause(i)trendsindomesticgovernancespeaktolong-termtrendsin
regime-type(e.g.democracy);(ii)itcouldinfluencethecharacterofkeyactorsinAIstrategy,
suchasthecharacteroftheChineseandUSgovernments;(iii)itwillinformthekindsof
global institutions that will be feasible and their properties.

4.1.1 Inequality
Thereisanextensiveandactiveliteratureoninequalityandgovernment.Thisliterature
shouldbereviewed,andlessonsappliedtoourunderstandingaboutfutureformsof
government, given trends in inequality (see section 4.2 ).

35

AI Governance: A Research Agenda

 4.1.2 Surveillance
TowhatextentwillAIandsensortechnologyenablecheap,extensive,effectivesurveillance?
Itisplausiblethatsufficientinformationaboutanindividual’sbehavior,intent,and
psychology--andofanindividual’ssocialnetwork--willsoonbegeneratedthroughpassive
interactionswithdigitalsystems,suchassearchqueries,emails,systemsforaffectandlie
detection,spatialtrackingthroughMACaddresses,facerecognition,orotherkindsof
individualrecognition.Ifso,afirstordereffectseemstobetoshiftpowertowardsthose
entitieswhoareabletousesuchinformation,plausiblytoreinforcegovernmentauthority,
andthusauthoritariansystems.However,super-surveillancecouldalsoprovebeneficial,
suchasforenablingAIverificationagreementsandforenabling“stabilization”(the
preventionofworld-destroyingtechnology).Inaddition,itmaybepossibletodesign
AI-enabledsurveillanceinwaysthatactuallyreinforceothervaluesandinstitutions,suchas
liberalismanddemocracy.Forexample,itmaybepossibletoattenuatethetypicaltradeoff
betweensecurityandprivacythroughcryptographicallyenabledprivacy-preserving
surveillance. 87

4.1.3 Repression and Persuasion
Profilingofindividuals,mappingofsocialnetworks,ubiquitoussurveillanceandliedetection,
scalableandeffectivepersuasion,andcost-effectiveautonomousweaponscouldallradically
shiftpowertostates.Thesetrendsmayenableastatethatiswillingtodosotomonitorand
disruptgroupsworkingagainstit.Autonomousweaponscouldpermitadictatortorepress
withoutrequiringtheconsentofmilitaryofficersorpersonnel,whichhavehistoricallybeen
onecheckonleaders.Thesetrendsshouldbemapped,understood,theirpotential
consequences studied, and governance safeguards proposed.

87Onecreativeideaistousesecuremultipartycomputationorhomomorphicencryptionwhenanalyzingdataforevidenceof
criminalactivity.Thesecryptographictechnologiesmakeitpossibletoperformsuchanalysiswithouthavingaccesstothe
underlyingdatainanunencryptedform.SeeTrask,Andrew.“SafeCrimeDetection:HomomorphicEncryptionandDeep
Learning for More Effective, Less Intrusive Digital Surveillance.” iamtrask , June 5, 2017.
https://iamtrask.github.io/2017/06/05/homomorphic-surveillance/ ;Garfinkel,Ben.“TheFutureofSurveillanceDoesn’tNeed
toBeDystopian.”TalkatEffectiveAltruismGlobal,June9,2018;andYin,AwaSun.“IntroducingOpenMined:DecentralisedAI.”
Becoming Human: Artificial Intelligence Magazine, August 3, 2017.
https://becominghuman.ai/introducing-open-mined-decentralised-ai-18017f634a3f .
BenGarfinkelprovidesanexcellentreviewofthestate-of-the-artincryptographicsystemsandtheirimplicationsforpolitical,
economic,andsocialinstitutionsinGarfinkel,Ben.“RecentDevelopmentsinCryptographyandPotentialLong-Term
Consequences.” Future of Humanity Institute, 2018.

36

AI Governance: A Research Agenda

 4.1.4 Advisors and Collective Action
AIcouldalsoconceivablyempowercitizens,relativetothestateorelites.Personaladvisor
AIscouldallowindividualcitizenstoengagepoliticallyinamoreinformedmanner,andat
lowercosttothemselves;however,thislevelofAIcapabilityseemslikeitmightbecloseto
AGI(likelytooccurrelativelylateinthesequenceoftransformativedevelopments).AI
systemscouldfacilitatecollectiveaction,suchasifitbecomespossibletoassembleand
mobilizeanovelpoliticalcoalition,andnewpoliticalcleavages,throughscrapingofsocial
mediaforexpressionsofsupportforneglectedpoliticalpositions.Individualscouldexpress
morecomplexpoliticalstrategies,andmoreefficientlycoordinate.Forexample,anAmerican
citizen(inapluralityvotingsystemthatstronglyrewardscoordination)mightwanttostate
thattheywouldvoteforathirdpartycandidateif40%oftherestoftheelectoratealsoagrees
to do so.

OtherkindsofnarrowAIadvisorscouldtransformdomesticpolitics.EfficientAItranslation
couldfacilitatecross-languagecommunicationandcoordination.AIpoliticalfilterscould
exacerbatepoliticalsorting(filterbubbles),orcouldelevatepoliticaldiscoursebyhelping
userstoavoidlow-credibilitynewsandmoreeasilyidentifycounter-arguments.Videoand
audioaffectandsincerity/liedetection,ifeffectiveandtrusted,couldincentivizegreater
sincerity (or self-delusion).  88
4.2 Inequality, Job Displacement, Redistribution
AIseemsverylikelytoincreaseinequalitybetweenpeople(andprobablyalsobetween
countries:see section5 ).Thedigitizationofproducts,becauseoflowmarginalcosts,              89
increaseswinner-take-alldynamics.AIdramaticallyincreasestherangeofproductsthatcan
bedigitized.AIwillalsodisplacemiddle-classjobs,andnear-termtrendsaresuchthatthe
replacementjobsarelowerpaying.Laborshareofnationalincomesisdecreasing;AIis               90
88 Thanks to Carl Shulman for the above.
89Korinek,Anton,andJosephE.Stiglitz.“Artificialintelligenceanditsimplicationsforincomedistributionandunemployment.”
No. w24174. National Bureau of Economic Research, 2017.
https://www8.gsb.columbia.edu/faculty/jstiglitz/sites/jstiglitz/files/w24174.pdf .
90ForareviewofforecastsofAIdisplacementofhumanjobs,seeWinick,Erin.“BusinessImpactEverystudywecouldfindon
what automation will do to jobs,in one chart.” MIT Technology Review ,January 25,2018.
https://www.technologyreview.com/s/610005/every-study-we-could-find-on-what-automation-will-do-to-jobs-in-one-chart/ .
Thethreemoreprominentforecastsarefrom:Nedelkoska,LjubicaandGlendaQuintini.“Automation,SkillsUseandTraining”,
OECD Social,Employment and Migration Working Papers,No.202,OECD Publishing,Paris,2018.
https://www.oecd-ilibrary.org/docserver/2e2f4eea-en.pdf?expires=1527369566&id=id&accname=guest&checksum=F85DCC6

37

AI Governance: A Research Agenda
likelytoexacerbatethis.Ultimately,withhuman-levelAI,thelaborshareofincomeshould               91
becomeeversmaller.Giventhatcapitalismoreunequallydistributedthanlaborvalue,an
increase in capital share of income will increase inequality.

AIseemstobegenerating(orisatleastassociatedwith)newnaturalglobalmonopoliesor
superstarfirms:there’seffectivelyonlyonesearchengine(Google),onesocialnetwork
service(Facebook),andoneonlinemarketplace(Amazon).Thegrowthofsuperstarfirms
plausiblydrivesthedeclininglaborshareofincome.TheseAI(quasi-)monopoliesand             92
associatedinequalityarelikelytoincreasinglybecomethetargetofredistributivedemands.
Anotherrisktoexamine,for“slowscenarios”(scenariosinwhichotherformsof
transformativeAIdonotcomeformanydecades)isofaninternationalwelfare
race-to-the-bottom,ascountriesracewitheachothertoprioritizetheneedsofcapitalandto
minimizetheirtaxburden.Researchinthisareashouldmeasure,understand,andproject
trendsinemploymentdisplacementandinequality.Whatwillbetheimplicationsforthe
policydemandsofthepublic,andthelegitimacyofdifferentgovernancemodels?Whatare
potential governance solutions?  93
D03FB399E86A59357199FABB1 ;Frey,CarlBenedikt,andMichaelA.Osborne.“TheFutureofEmployment:HowSusceptible
Are Jobs to Computerisation?” Technological Forecasting and Social Change 114 (2017): 254-280.
https://www.sciencedirect.com/science/article/pii/S0040162516302244 ;Mankiya,J.,SusanLund,MichaelChui,Jacques
Bughin,JonathanWoetzel,ParulBatra,RyanKo,andSaurabhSanghvi.“Jobslost,jobsgained:Whatthefutureofworkwillmean
for jobs, skills, and wages.” McKinsey & Company, December 2017.
https://www.mckinsey.com/~/media/McKinsey/Featured%20Insights/Future%20of%20Organizations/What%20the%20futu
re%20of%20work%20will%20mean%20for%20jobs%20skills%20and%20wages/MGI-Jobs-Lost-Jobs-Gained-Report-Decemb
er-6-2017.ashx .Oneffortstotheorizeandforecastfuturelabordisplacement,see:Brynjolfsson,Erik,andTomMitchell.“What
can machine learning do? Workforce implications.” Science 358, no. 6370 (2017): 1530-1534.
http://science.sciencemag.org/content/358/6370/1530 .
91Dorn,David,LawrenceF.Katz,ChristinaPatterson,andJohnVanReenen.“ConcentratingontheFalloftheLaborShare.”
AmericanEconomicReview107,no.5(2017):180-85.Autor,David,andAnnaSalomons.“Isautomationlabor-displacing?
Productivity growth,employment,and thelaborshare.”BrookingsPaperson EconomicActivity,2018.
https://www.brookings.edu/wp-content/uploads/2018/03/1_autorsalomons.pdf .
92 Dorn et al, 2017.
93Automation’sthreattothelabormarketmayalreadybeaffectingpolitics.Ananalysisofsurveydatafrom17European
countriesbetween2002and2012findsthatrespondentswhosejobsweremoreautomotableexpressedgreatersupportfor
redistribution(Thewissen,StefanandDavidRueda.“AutomationandtheWelfareState:TechnologicalChangeasaDeterminant
ofRedistributionPreferences.” ComparativePoliticalStudies,2017. https://doi.org/10.1177/0010414017740600 .)Asimilar
studyacross15Europeandemocraciesshowsthatthosemoreexposedtoautomationshocksindicatedgreatersupportfor
nationalistandradical-rightparties(Anelli,Massimo,ItaloColantoneandPieroStanig.“WeWereTheRobots:Automationin
Manufacturing and
VotingBehaviorinWesternEurope.”Workingpaper,2018. http://www.italocolantone.com/research.html .)IntheU.S.,
exposuretoautomationwaspositivelycorrelatedwithsupportforDonaldTrumpinthe2016PresidentialElectionatthe
electoraldistrictlevel(Frey,CarlBenedikt,ChinchihChen,andThorBerger.“PoliticalMachinery:DidRobotsSwingthe2016U.S.
Presidential Election?” Oxford Martin School, July 2018. https://www.oxfordmartin.ox.ac.uk/publications/view/2576 .)

38

AI Governance: A Research Agenda

 4.3 Public Opinion and Regulation
Historicallypublicopinionhasbeenapowerfulforceintechnologypolicy(e.g.bansonGMOs
ornuclearenergy)andinternationalpolitics(e.g.Sputnik).Further,astheseexamples
illustrate,publicopinionisnotsimplyareflectionofeliteinterest.InthecaseofSputnik,the
USintelligencecommunitywaswellawareofSovietprogress,andtheEisenhower
administrationdidnotwanttoengageinaspaceraceandtriedtopersuadetheAmerican
publicthatSputnikwasnotasignificantdevelopment.Andyet,withinmonthsSputnikhad               94
triggeredareorientationofUStechnologypolicy,includingthelegislativeformationofARPA
(todayDARPA).Itcouldthusbehelpfultostudypublicopinionandanticipatemovementsin
publicopinion,ascanbeinformedbyscenariobasedsurveys,andstudyingparticulargroups
whohavebeenexposedtoinstancesofphenomena(suchasemploymentshocks,ornew
formsofsurveillance)thatcouldlateraffectlargerpopulations.Whatkindsofpublic
reactionscouldarise,leadingtooverlyreactivepolicyandregulation?CouldregulatingAI(or
taxingAIcompanies)becomeanewtargetofpoliticalcampaigns,asalreadyseemstobe
happeninginEurope?Thisareaofresearchwillalsohelppolicymakersknowhowbestto
engagepublicopinionwhenaneventoccurs(andingeneral).Itwillalsohelpscholarsto
communicate the results and value of their work.
5. International Political Economy
ThenextsetofquestionsintheAIPoliticsresearchclusterexaminestheinternational
politicaldynamicsrelatingtotheproductionanddistributionofwealth.Economicsuccess
withAIandinformationtechnologyseemstoexhibitsubstantialreturnstoscale(e.g.Google              95
)andagglomerationeconomies(e.g.SiliconValley).Ifthesetrendspersistitcouldleadtoan                 96
(evenmoreextreme)internationalAIoligopoly,whereafewfirmscapturemostofthevalue
fromprovidingAIservices.Arethereanyrelevantaspectstothecompetitivedynamics
94Ryan,AmyandGaryKeeley.“SputnikandUSIntelligence:TheWarningRecord.” StudiesinIntelligence61:3(2017).
https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/csi-studies/studies/vol-61-no-3/pdfs/sputni
k-the-warning-record.pdf .
95Dueinparttothe“virtuouscycle”betweenAIcapabilitieswhichattractscustomers,whichincreasesone’sdata,which
improves one’s AI capabilities, and the high fixed costs of developing AI services and low marginal cost of providing them.
96OnindustryconcentrationinAI,seethefollowingandreferences:Bessen,JamesE.“InformationTechnologyandIndustry
Concentration.”BostonUniv.SchoolofLaw,LawandEconomicsResearchPaperNo.17-41,December1,2017.
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3044730 .

39

AI Governance: A Research Agenda
betweencompanies;forexample,towhatextentareAIinnovationsbeingpatented,are
patentable, or are held as trade secrets?

CountrieslackingAIindustriescurrentlyworrythattheyarebeingshutoutofthemost
rewardingpartoftheglobalvaluechain.SomeintheUSandEuropecurrentlyworrythat                 97
Chinaiscoercively/unfairlyleveragingitsmarketpowertostrategicallyextracttechnical
competencefromWesternfirms,andthiswasarguablythemotivationfortheTrumptrade
war.Theseconcernscouldleadto AImercantilismor AInationalism ,followingfrom              98 99
strategic-tradetheory,wherecountriesdevotesubstantialresourcestoretainingand
developingAIcapabilities,andtosupportingtheirAInationalchampions.Towhatextentare
countries(e.g.Canada)abletointernalizethereturnsontheirAIinvestments,ordoestalent
inevitablygravitatetowardsandbenefittheexistingleadersinAI(e.g.SiliconValley)?              100
Whatlessonsemergefromexaminingthepartialanalogiesofothergeneralpurpose
technologiesandeconomywidetransformationssuchascomputerization,electrification,and
industrialization?

CountriesandcompaniesaresearchingforotherwaystoeconomicallybenefitfromtheAI
transformedeconomy.Theyaresearchingforrewardingnodesinthevaluechaininwhich
theycanspecialize.Countriesareexaminingpolicyleverstocapturemoreoftherentsfrom
AIoligopolies,andaspiretobuilduptheirownAIchampions(suchastheEUrulingsagainst
AppleandGoogle,andChina’sexclusionofGoogleandFacebook).Howsubstantialofan
advantagedoesChinahave,ascomparedwithotheradvanceddeveloped(mostlyliberal
democratic)countries,initsabilitytochannelitslargeeconomy,collectandsharecitizen
data,andexcludecompetitors?WhatstepscouldandwouldtheU.S.taketoreinforceitslead?
WhatarethepossibilitiesandlikelydynamicsofaninternationaleconomicAIrace?Isit
97On“highdevelopmenttheory”asappliedtodataflowsandassets,see:Weber,Steven.“Data,development,andgrowth.”
Business and Politics (2017): 1-27.
https://www.cambridge.org/core/journals/business-and-politics/article/data-development-and-growth/DC04765FB73157C8
AB76AB1742ECD38A .
98Barboza,David.“HowThisU.S.TechGiantIsBackingChina’sTechAmbitions.”NewYorkTimes,August4,2017.
https://www.nytimes.com/2017/08/04/technology/qualcomm-china-trump-tech-trade.html .
99Hogarth, Ian. “AI Nationalism.” Ian Hogarth (blog), June 13, 2018.
https://www.ianhogarth.com/blog/2018/6/13/ai-nationalism .
100RecentopeningoftwoDeepMindsatellitesinCanada,andthe Pan-CanadianAIStrategysuggestthatotherinformedactors
believe non-central locations are worth investing in.

40

AI Governance: A Research Agenda
 plausiblethatcountrieswouldsupportdomesticoralliedconsortiaofAIcompanies,soasto
better compete in industries that appear to be naturally oligopolistic?

Technologicaldisplacementwillimpactcountriesdifferentially,andcountrieswilladopt
differentpolicyresponses.Whatwillthosebe?Ifredistributingwealthandretraining
becomesaburdenonprofitablecompanies,couldtherebeAIcapitalflightandan
internationalrace“tothebottom”ofprovidingaminimaltaxburden?Ifso,couldthe
internationalcommunitynegotiate(andenforce)aglobaltaxsystemtoescapethisperverse
equilibrium?OrareAIassetsandmarketssufficientlytaxinelastic(e.g.territoriallyrooted)
as to prevent such a race-to-the-bottom?

ResearchoninternationalpoliticaleconomyismostrelevantforscenarioswhereAIdoesnot
(yet)provideastrategicmilitarybenefit,asonceitdoesthelogicofinternationalsecuritywill
likelydominate,oratleastheavilyshape,economicconsiderations.However,manyIPE
relatedinsightsequallyapplytotheinternationalsecuritydomain,sothereisvaluein
studying these common problems framed in terms of IPE.
6. International Security
AIandrelatedtechnologiesarelikelytohaveimportantimplicationsfornationaland
internationalsecurity.ItisalsoplausiblethatAIcouldhavestrategicandtransformative
militaryconsequencesinthenearandmedium-term,andthatthenationalsecurity
perspectivecouldbecomedominant.First,studyingthe near-termsecuritychallengesis
helpfulforunderstandingthecontextoutofwhichlonger-termchallengeswillemerge,and
enableustoseedlong-termbeneficialprecedents.Longer-term,ifgeneralAIbecomes
regardedasacriticalmilitary(oreconomic)asset,itispossiblethatthestatewillseekto
control ,close ,andsecuritizeAIR&D.Further,thestrategicandmilitarybenefitsofAImay
fuelinternational racedynamics .Weneedtounderstandwhatsuchdynamicsmightlook
like, and how such a race can be avoided or  ended .

41

AI Governance: A Research Agenda

 6.1 Near-term Security Challenges
InthecomingyearsAIwillposeahostofnovelsecuritychallenges.Theseinclude
internationalanddomesticusesofautonomousweapons,andAI-enabledcyber-operations,
malware,andpoliticalinfluencecampaigns(“activemeasures”).Manyofthesechallenges
looklike“lite”versionsofpotentialtransformativechallenges,andthesolutionstothese
challengesmayserveasafoundationforsolutionstotransformativechallenges.Tothe              101
extentthenear-termandtransformativechallenges,ortheirsolutions,aresimilar,itwillbe
usefulforustobeawareofandengagewiththem.Forarecommendedsyllabusofreadings
on AI and International Security, see:
❖Zwetsloot,Remco.“ArtificialIntelligenceandInternationalSecuritySyllabus.”Future
of Humanity Institute, 2018. ( link ).
Some specific references worth looking at include:
❖Brundage,M.,S.Avin,etal.“TheMaliciousUseofArtificialIntelligence:Forecasting,
Prevention, and Mitigation.” Future of Humanity Institute, 2018. [ PDF ].
❖Horowitz,Michael,PaulScharre,GregoryC.Allen,KaraFrederick,AnthonyChoand
EdoardoSaravalle.“ArtificialIntelligenceandInternationalSecurity.”Centrefora
New American Security, 2018. ( link ).
❖Scharre,P. ArmyofNone:AutonomousWeaponsandtheFutureofWar .NewYork:W.
W. Norton & Company, 2018.
❖Horowitz,M.“ArtificialIntelligence,InternationalCompetition,andtheBalanceof
Power.” Texas National Security Review , 2018.  [ link ] [ PDF ].
6.2 Control, Closing, and Securitization
BasicAIR&Discurrentlyconductedintheopen:researchershaveastronginteresttopublish
theiraccomplishmentstoachieverecognition,andthereisastrongethosofscientific
openness.SomeAIR&Dissemi-closed,conductedinprivatefor-profitspaces;however,this
tendstonotbegeneralAIR&D,butinsteadapplicationsofexistingtechniques.Thiscould
plausiblychange,ifAIbecomesperceivedascatastrophicallydangerous,strategicmilitary,or
101Forexample,wecananalyzehowtransnationalself-governanceregimesofprivatecompanieshaveemergedandwhythese
effortshavesucceededorfailed.ThisisparticularlyrelevantasseveralAIcompanieshavealreadyintroducedself-governance
measures as well. Fischer, Sophie-Charlotte. 2018. “Reading List - Industry Self-Regulation/Security Governance”..

42

AI Governance: A Research Agenda
 evenstrategiceconomic.TotheextentanAIraceislikelyandcatastrophicrisksare
increasinginacloserace,asopposedtoonewheretheleaderhasalargelead,itwould
arguablybepreferableforAIleaderstotakestepstopreventtheircapabilitiesfromdiffusing
to others.

WhatarethedifferentmodelsforcompletelyorpartiallyclosingAIresearchorassets(like
compute)?Whataretheirprosandcons? Atwhatpointwouldandshouldthestatebe                 102
involved?Whatarethelegalandothertoolsthatthestatecouldemploy(orareemploying)to
closeandexertcontroloverAIcompanies?Withwhatprobability,andunderwhat
circumstances,couldAIresearchanddevelopmentbe securitized --i.e.,treatedasamatterof
nationalsecurity--atorbeforethepointthattransformativecapabilitiesaredeveloped?How
mightthishappenandwhatwouldbethestrategicimplications?Howareparticularprivate
companieslikelytoregardtheinvolvementoftheirhostgovernment,andwhatpolicy
optionsareavailabletothemtonavigatetheprocessofstateinfluence?Howareresearchers
likelytobeinvolved?Canwelearnfromthestudyoftheattemptedclosingandcontrolof
other technologies?
6.3 Race Dynamics
AdvancedAIcouldconveyextremepowerandwealthtoitspossessors.Ifso,andinparticular
ifitisexpectedtoconveystrategicmilitaryoreconomicbenefits,thenitisplausiblethatan
(international)racedynamiccouldemerge.Thedefiningfeatureofatechnologyraceisthat
therearelargegainsfromrelativeadvantage.Insuchacircumstanceactorshavestrong
incentivestotrade-offagainstothervalues(likesafety,transparency,accountability,
democracy)andopportunities,inordertoincreasetheprobabilityofgainingadvantage.In
particular,aworryisthatitmaybeclosetoanecessaryandsufficientconditionforAIsafety
andalignmentthattherebeahighdegreeofcautionpriortodeployingadvancedpowerful
systems;however,ifactorsarecompetinginadomainwithlargereturnstofirst-moversor
relative advantage, then they will be pressured to choose a sub-optimal level of caution.

102Bostrom,Nick.“StrategicImplicationsofOpennessinAIDevelopment.” GlobalPolicy ,February2017.
http://onlinelibrary.wiley.com/doi/10.1111/1758-5899.12403/fullor http://www.nickbostrom.com/papers/openness.pdf ;
Krakovna,Victoria.“ClopenAI:OpennessindifferentaspectsofAIDevelopment.” DeepSafety (blog),August1,2016.
https://vkrakovna.wordpress.com/2016/08/01/clopen-ai-openness-in-different-aspects-of-ai-development/ .

43

AI Governance: A Research Agenda
Researchonracedynamicsinvolvesalargesetofquestionsandapproaches.Wewillneedto
integrateanddevelopmodelsoftechnology/armsraces.Whatarethedistinctivefeatures             103
ofanAIrace,ascomparedwithotherkindsofraces?Whatrobustpredictionscanwemake
aboutthatsubfamilyofraces?Underwhatconditionsarethoseracesmostdangerousor
destructive?Specifically,aplausibleandimportantpropositionisthatracesaremore
dangerousthesmallerthemarginoftheleader;isthisarobustconclusion? Howdo                104
openness,accessibilityoftheresearchfrontier,first-moveradvantages,insecurecompute,
andotherfactorsaffectracedynamics?GivenmodelsofAIinnovation,howconfidentcana
leadteambeabouttheperformanceofitsrivals,andthatitwillbeabletosustainaknown
lead?GivenmodelsofAIsafety(suchastheperformance-safetytradeoffsandthe
time-schedule for safety investments), what is the expected risk incurred by race dynamics?

Therearealsoquestionsaboutthestrategiesforretainingaleadorcatchingup.Arethere
toolsavailabletotheleadingteamthatwillallowittoretainalead?Forexample,coulda
teamretainitsleadbyclosingoffitsresearch?Whatdifferencedoesitmakeiftheleading
team is a state, or closely supported by a state?

Thepotentialforcoalitionswithinaracemeritsstudy.Whatarethepossibilitiesforalliances
betweenleadinggroupsorstatestohelpthemretaintheirlead?Inlightofstates’interestsin
strongAIsystems,currentinternationalagreements,andhistoricrelationships,what
configurations of state coalitions are likely and under what circumstances?

Historicalprecedentsandanalogiescanprovideinsight,suchasconsiderationofthearms
raceforandwithnuclearweapons,otherarmsraces,andpatentandeconomictechnology
races.Whataboutanalogiestootherstrategicgeneralpurposetechnologiesandmore
gradualtechnologicaltransformations,like industrialization,electrification,and
computerization? In what ways do each of these fail as analogies?
103ForamodelonrisksfromAIraces,seeArmstrong,Stuart,NickBostrom,andCarlShulman.“Racingtotheprecipice:amodel
of artificial intelligence development.”  AI & Society 31, no. 2 (2016): 201-206.
Thebroadermodelingliteraturerelevanttoracesislarge,includingeconomicmodelsofauctions,patentraces,andmarket
competition.
104Foramodelofatechnologyracewhichismostintensewhentheracersarefaraway,seeHörner,Johannes.“APerpetualRace
to Stay Ahead.” Review of Economic Studies (2004) 71, 1065–1088.
http://users.econ.umn.edu/~holmes/class/2007f8601/papers/horner.pdf .

44

AI Governance: A Research Agenda

FinallyitwouldbevaluabletotheorizethelikelystagesofanAIraceandtheircharacteristics
(tempo,danger,securityconsequences).Canwemapcurrentbehaviorontothisframework?
Whatisthecurrentdistributionofcapabilities,talent,andinvestment? Towhatextentdo               105
existingpolicymakersandpublicsperceiveorinvokearacelogic? Whatkindsofevents                106
couldsparkorescalatearace,suchas“Sputnikmoments”forpublics oran               107
“Einstein-Szilard letter” for leaders? 108
6.4 Avoiding or Ending the Race
GiventhelikelylargerisksfromanAIrace,itisimperativetoexamine possibleroutesfor
avoidingracesorendingoneunderway .Thepoliticalsolutionstoglobalpublicbadsare,in
increasingexplicitnessandinstitutionalization:norms,agreements(“softlaw”),treaties,or
institutions.Thesecanbebilateral,multilateral,orglobal.Normsinvolvearoughmutual
understandingaboutwhat(observable)actionsareunacceptableandwhatsanctionswillbe
imposedinresponse.Implicitnormshavetheadvantagethattheycanarisewithoutexplicit
consent,butthedisadvantagethattheytendtobecrude,andarethusofteninadequateand
mayevenbemisdirected. Ahardenedformofinternationalnormsiscustomarylaw,              109
thoughabsentarecognizedinternationaljudiciarythisisnotlikelyrelevantforgreat-power
cooperation.  110

Diplomaticagreementsandtreatiesinvolvegreaterspecificationofthedetailsofcompliance
andenforcement;whenwellspecifiedthesecanbemoreeffective,butrequiregreaterlevels
105 This is also asked in the Technical Landscape .
106Cave,Stephen,andSeánS.ÓhÉigeartaigh.“AnAIRaceforStrategicAdvantage:RhetoricandRisks.”In AAAI/ACMConference
on Artificial Intelligence, Ethics and Society , 2018.
http://www.aies-conference.com/wp-content/papers/main/AIES_2018_paper_163.pdf .
107AlphaGosofarbeingtheclosestthingtoaSputnikmoment,thoughthatmostlyforpeopleinChina,Japan,SouthKorea,and
other cultures where Go is esteemed.
108Cf.Grace,Katja.“LeóSzilárdandtheDangerofNuclearWeapons:ACaseStudyinRiskMitigation.”TechnicalReport.Berkeley,
CA: Machine Intelligence Research Institute, October 2015.  https://intelligence.org/files/SzilardNuclearWeapons.pdf  .
109JosephNyeadvocatesforcyber-norms.Nye,JosephS.“ANormativeApproachtoPreventingCyberwarfare.”ProjectSyndicate,
March 13, 2017.
https://www.project-syndicate.org/commentary/global-norms-to-prevent-cyberwarfare-by-joseph-s--nye-2017-03 .However,
thecaseofcyberweaponsmayalsopointtosometechnologiesthataremuchmorelimitedintheirpotentialtobecontrolled
througharmscontrolagreements.Foranintroduction,cf.Borghard,EricaD.,andShawnW.Lonergan.“WhyAreThereNoCyber
Arms Control Agreements?” Council on Foreign Relations, January 16, 2018.
https://www.cfr.org/blog/why-are-there-no-cyber-arms-control-agreements .
110Cf.Williamson,Richard.“HardLaw,SoftLaw,andNon-LawinMultilateralArmsControl:SomeComplianceHypotheses.”
Chicago Journal of International Law  4, no. 1 (April 1, 2003).  https://chicagounbound.uchicago.edu/cjil/vol4/iss1/7 .

45

AI Governance: A Research Agenda
ofcooperationtoachieve.Institutions,suchastheWTO,involveestablishingabureaucracy
withtheabilitytoclarifyambiguouscases,verifycompliance,facilitatefuturenegotiations,
andsometimestheabilitytoenforcecompliance.Internationalcooperationoftenbeginswith
norms, proceeds to (weak) bilateral or regional treaties, and consolidates with institutions.

SomeconjecturesaboutwheninternationalcooperationintransformativeAIwillbemore
likelyarewhen:(1)thepartiesmutuallyperceiveastronginterestinreachingasuccessful
agreement(greatrisksfromnon-cooperationorgainsfromcooperation,lowreturnson
unilateralsteps);(2)whenthepartiesotherwisehaveatrustingrelationship;(3)whenthere
issufficientconsensusaboutwhatanagreementshouldlooklike(whatcomplianceconsists
of),whichismorelikelyiftheagreementissimple,appealing,andstable;(4)when
complianceiseasily,publicly,andrapidlyverifiable;(5)whentherisksfrombeingdefected
onarelow,suchasifthereisalong“breakouttime”,alowprobabilityofapowertransition
becausetechnologyisdefensedominant,andnear-termfuturecapabilitiesarepredictably
non-transformative;(6)theincentivestodefectareotherwiselow.Comparedtoother
domains,AIappearsinsomewayslessamenabletointernationalcooperation--conditions
(3),(4),(5),(6)--butinotherwayscouldbemoreamenable,namely(1)ifthepartiescometo
perceiveexistentialrisksfrom unrestrictedracingandtremendousbenefitsfrom
cooperating,(2)becauseChinaandtheWestcurrentlyhavearelativelycooperative
relationshipcomparedtootherinternationalarmsraces,andtheremaybecreativetechnical
possibilitiesforenhancing(4)and(5).Weshouldactivelypursuetechnicalandgovernance
research today to identify and craft potential agreements.

Third-Party Standards, Verification, Enforcement, and Control
OnesetofpossibilitiesforavoidinganAIarmsraceistheuseofthirdpartystandards,
verification,enforcement,andcontrol.Whataretheprospectsforcooperationthroughthird
partyinstitutions?Thefirstmodel,almostcertainlyworthpursuingandfeasible,isan
international“safety”agencyresponsiblefor“establishingandadministeringsafety
standards.”Thisiscrucialtoachievecommonknowledgeaboutwhatcountsascompliance.              111
Thesecond“WTO”or“IAEA”modelbuildsonthefirstbyalsoverifyingandrulingon
111 As per Article II of the IAEA Statute.

46

AI Governance: A Research Agenda
non-compliance,afterwhichitauthorizesstatestoimposesanctionsfornoncompliance.The
thirdmodelisstrongerstill,endowingtheinstitutionwithsufficientcapabilitiestoenforce
cooperationitself.Thefourth,“AtomicDevelopmentAuthority”model,involvestheagency
itselfcontrollingthedangerousmaterials;thiswouldinvolvebuildingaglobalAI
developmentregimesufficientlyoutsidethecontrolofthegreatpowers,withamonopolyon
this(militarily)strategictechnology.Especiallyinthefourthcase,butalsofortheweaker
models,greatcarewillneedtogointotheirinstitutionaldesigntoassurepowerfulactors,
and ensure competence and good motivation.

Suchthirdpartymodelsentailaseriesofquestionsabouthowsuchinstitutionscouldbe
implemented.Whataretheprospectsthatgreatpowerswouldgiveupsufficientpowertoa
globalinspectionagencyorgoverningbody?Whatpossiblescenarios,agreements,tools,or
actionscouldmakethatmoreplausible?Whatdoweknowabouthowtobuildgovernment
thatisrobustagainstslidingintototalitarianismandothermalignantforms(see section4.1 )?
Whatcanwelearnfrom similarhistoricalepisodes,suchasthefailureofthe
Acheson-LilienthalReportandBaruchPlan,thesuccessofarmscontroleffortsthatled
towardsthe1972Anti-BallisticMissile(ABM)Treaty,andepisodesofattemptedstate             112
formation?

Theremayalsobeotherwaystoescapetherace.Couldonesideformawinningor
encompassingcoalition?Couldoneorseveralracersengageinunilateral“stabilization”ofthe
world,withoutriskingcatastrophe?ThesectionAIIdealGovernancediscussesthedesirable
properties of a candidate world hegemon.

112Adler,Emanuel.“TheEmergenceofCooperation:NationalEpistemicCommunitiesandtheInternationalEvolutionoftheIdea
of Nuclear Arms Control.” International Organization  46, no. 1 (1992): 101–45.

47

AI Governance: A Research Agenda
 AI Ideal Governance
TheTechnicalLandscapeseekstounderstandthetechnicalpossibilitiesandconstraintsof
AIdevelopment. AIPoliticsseekstounderstandhowdifferentactorswillcompeteand
cooperatetoachievetheirobjectivesrelatedtopowerfulAI. AIIdealGovernancefocuseson
cooperativepossibilities:ifwecouldsufficientlycooperate,whatmightwecooperateto
build?AIIdealGovernanceexaminespotentialglobalarrangementsforgoverningwhatkinds
ofAIaredevelopedanddeployed,bywhom,forwhatpurposes,andwithwhatconstraints.In
particular,thisclusterseekstoidentifyidealmodelsofglobalgovernance.Whatare
humanity’scommon values ,andwhatarrangementscouldbestsatisfyourdistinctgoals?
Whatorganizationalprinciplesand institutionalmechanismsexisttobestpromotethose?
Theseage-oldquestionsneedtobeinvestigatedwithrenewedvigor.Wemaysoonneedto
implementourbestanswer.AdvancedAIcouldalsodramaticallyaltertherelevant
parameters of the question, rendering prior insights less relevant.

Thisresearchclusterfocusesonidealizedglobalgovernance,forseveralreasons.Itis
importanttodevotethoughttoarticulatingwhatwearetryingtoachieve(1)sothatweare
bestabletosteereventsindesirabledirectionsand(2)tofacilitatecooperationby
coordinatingonanappealingshared vision .Insodoingitwillbeimportanttodevelop
effectivemeansofcommunicatingthebountyofpotentialbenefitsfromcooperation,andthe
existentialdangersfromrivalrousbehavior.Weneedtomakesureweunderstandthe
distinctconcernsandworldviewsofpeopleandelitesfromdifferentbackgrounds,sothatour
governanceproposalsaremostlikelytoresonategloballyandtobeincentive-compatible
withpowerfulstakeholders.Weneedtothinkpragmaticallyaboutinstitutionaldesign--what
shouldbetheconstitutionalfoundation;whoisentitledtomakewhatdecisions,underwhat
conditions,bywhatvotingrules;whatinformationistransmittedtowhom--toensurethat
any acceptable ideal vision is politically stable.

ThefindingsfromthisresearchclusterwillbecrucialforadvisingthegovernanceofAIfirms
and countries today and in the future. This is so for two reasons.

48

AI Governance: A Research Agenda
 (1)Thegovernanceproblemsthatwearefacingtodayandthatwewillfaceinthefuture
overlapextensively,withtheprimarydifferencesbeing(i)thescopeofintereststobe
represented,(ii)thepotentialneedtocompeteinsomebroadermilitary-economicdomain,
and(iii)thestakes.Toillustratethesimilarities,considerhowthegovernanceofan
internationalAIcoalitionwillideallyhavesomeconstitutionalcommitmenttoacommon
good,willhaveinstitutionalmechanismsforassuringtheprincipals(e.g.thepublicsand
leadersoftheincludedcountries)thattheregimeiswell-governed,andforcredibly
communicatingalackofthreattootherparties.Infact,ifweareabletocraftasufficiently
appealing,realistic,self-enforcing,robustmodelofAIgovernance,thiscouldserveasa
beacon,toguideusoutofarivalrousequilibrium.Theproblemthenreducestooneof
sequencing: how do we move from the present to this commonly appealing future?
(2)Wewouldideallyliketoembedintoourgovernancearrangementstoday,whenthestakes
arerelativelylow,theprinciplesandmechanismsthatwewillneedinthefuture.For
example,giventemporaldiscounting,diminishingmarginalutility,anduncertaintyabout
whowillpossessthewealth,itmaybepossibletodaytoinstitutionalizecollective
commitments for redistributing wealth in the future. 113

Thisresearchclusteristheleastdevelopedinthisdocument,andwithinthecommunityof
people working on AI governance.
7. Values and Principles
AIidealgovernanceaspirestoenvision,blueprint,andadvanceidealinstitutionalsolutions
forhumanity’sAIgovernancechallenges.Whatarethecommonvaluesandprinciplesaround
whichdifferentgroupscancoordinate?Whatdovariousstakeholders(publics,cultural
groups,AIresearchers,elites,governments,corporations)wantfromAI,inthenear-termand
long-term?Whatarethebestwaysofmediatingbetweencompetinggroupsandbetween
conflictingvalues?Whatdolong-termtrends--suchasfromdemographics,secularization,
globalization,liberalism,nationalism,inequality--implyaboutthevaluesofthese
stakeholders over medium and long timelines?
113Bostrom,Nick,AllanDafoe,andCarrickFlynn.“PublicPolicyandSuperintelligentAI:AVectorFieldApproach.”Futureof
Humanity Institute, 2018. http://www.nickbostrom.com/papers/aipolicy.pdf

49

AI Governance: A Research Agenda

Givenwhatwearestilllearningaboutourselvesandourvalues,isitpossibletoanticipatethe
directionthatourvaluesaremovingin,orthedirectiontheyshouldmovein?Given
uncertaintyaboutourcommonvaluesandwhatshouldbeourvalues,arethereprincipleswe
canemploythatwill“learnwithus”overtimeandpreventusfrommakinglargemistakes?
Bostrom,Dafoe,andFlynn(2018)offerasetofpolicydesideratathatgaindistinctive
importanceinaworldofsuperintelligentAI,including:expeditiousprogress,AIsafety,
conditionalstabilization,non-turbulence,universalbenefit,magnanimity,continuity,
first-principlesthinking,wisdom,speedanddecisiveness,andadaptability.Thereappearto
betwocrucial(meta-)principlesinthepresentworld,andtheyareintension:(1) security
(AI safety, conditional stabilization) and (2) autonomy  (freedom, continuity, sovereignty).  114

Thisworkrequiresscholarsofethicsandmorality,psychology,globalpublicopinion,culture,
and religion.
8. Institutions and Mechanisms
Theprevioussectioninvolvesspecifyingtheinterestsofthestakeholdersthatagovernance
systemshouldmeet,aswellastheoverallgoalsofthesystem.Thissectionthenseeksto
develop institutions that can successfully achieve these interests and goals.

Wewantourgovernanceinstitutionstobecapableofprovidingsecurity,ensuringsafety
fromnon-alignedAI,andotherwisestabilizingtechnologicaldevelopmenttopreventnew
extremerisks.ThismayrequirecentralizedcontroloverAIdevelopment,orextensive
surveillanceofAIprojectswithreadyabilitytoshutthemdown.It’spossiblesuchsafety
couldbeachievedthroughacooperatingmultipolarworld,butitmayrequireconcentration
ofpowerandauthority.Whataretheleastinfringingpossiblestabilizationarrangements?
WhatcapabilitiesmayAIenablethatcouldhelpuswiththis,howprobablearethey,andwhat
could be done to increase their probability?
114TheAsilomarAIPrinciples(https://futureoflife.org/ai-principles/)offersastatementofsomebroadlyappealingprinciples.
Metzinger outlines reasons for and components of a Global AI Charter.
https://www.blogs.uni-mainz.de/fb05philosophie/files/2018/03/Metzinger_2018_Global_Artificial_Intelligence_Charter_PE_61
4.547.pdf

50

AI Governance: A Research Agenda

Wewantourgovernanceinstitutionstoberesilienttodriftandhijacking.Twopolesofthe
riskspacearetotalitariancaptureandtyrannyofthemajority.Topreventtotalitarian
captureandtyrannyofthemajority,tovaryingextentsandinvaryingcombinations,
countriesthroughouttheworldhaveemployed:regular,free,andfairelections;protected
rightsforpoliticalexpression;ruleoflawandanindependentjudiciary;divisionofpower;
constraints on state power; constitutionally protected rights; federalism.

Theproblemofhowtobuildinstitutionsforgoverningapolityisacorepartofthefieldsof
politicalscienceandpoliticaleconomy.Themoremathematicaltheoreticalcornerofthis
spaceisoftencalledpublicchoice,socialchoice,or(byeconomists)politicaleconomy.
PoliticalscientistsincomparativepoliticsandAmericanpoliticsextensivelystudythe
propertiesofdifferentpoliticalsystems.Scholarsinpoliticalscienceandpoliticaltheory
studythedesignofconstitutions.Giventhecentralityofthisproblemtothesefields,andtheir
existingexpertise,substantialeffortshouldbespentlearningfromthemandrecruitingthem,
ratherthantryingtoreinventgoodgovernance.Nevertheless,atthepresenttimethe
application of these disciplines to the problems of AI governance remains neglected.
9. Positive Visions
Whiletheaboveisdirectedtodevisingafeasiblemodelofideallong-runAIgovernance,itis
unlikelytogenerateasimplesolution(anytimesoon).However,itcouldbeextremely
beneficialtohaveasimple,compelling,broadlyappealingvisionofthebenefitsfrom
cooperation,tohelpmotivatecooperation.Webelievethatboththepotentialbenefitsfrom
safedevelopmentandthepotentialdownsidesfromunsafedevelopmentarevast.Giventhat
perspective,itisfoolishtosquabbleoverrelativegains,ifdoingsoreducesthechancesof
safedevelopment.Howcanwesimply,clearly,evocativelycommunicatethatvisionto
others? 115

115 “Paretotopia”?

51

AI Governance: A Research Agenda
 Appendix A: Forecasting Desiderata
(This relates to section 2.3 ).

1.Wewantourforecastingtargetstobeindicatorsfor relevant achievements.This
includestargetsthatserveas(leading)indicatorsforimportanteconomiccapabilities,
suchasacapabilitythatwouldposeasubstantialemploymentthreattoalargegroup
ofpeople.Itincludesindicatorsforimportant security capabilities,suchasinpotent
AIcyber-offense,surveillanceandimageryintelligence,orliedetection.Itincludes
indicatorsforimportant technicalachievements,suchasthosethatarethoughttobe
crucialstepsonthepathtomoretransformativecapabilities(e.g.AGI),thosethatare
centraltomanyproblemareas,orthosethatwouldotherwisesubstantiallyaccelerate
AI progress.  116
2.Wewantthemtobe accurateindicators,asopposedtonoisyindicatorsthatarenot
highlycorrelatedwiththeimportantevents.Specifically,whereEistheoccurrenceor
nearoccurrenceofsomeimportantevent,andYiswhetherthetargethasbeen
reached,wewantP(notY|notE)~1,andP(Y|E)~1.Anindicatormayfailtobe
informativeifitcanbe“gamed”inthattherearewaysofachievingtheindicator
withouttheimportanteventbeingnear.Itmaybeanoisyindicatorifitdependson
otherwiseirrelevantfactors,suchaswhetheratargethappenstotakeonsymbolic
importance as the focus of research.
3.Wewantthemtobe wellspecified :theyareunambiguousandpubliclyobservable,
sothatitwillnotbecontroversialtoevaluatewhetherEhastakenplace.Thesecould
beeithertargetsbasedonacommonlyagreedobjectivemetricsuchasan
authoritativemeasureofperformance,orasubjectivetargetlikelytoinvolve
agreement across judges. Judges will not ask later: “what did you mean”? 117
4.Wewantthemtobesomewhat near-termprobable :weshouldnotbetooconfident
inthenear-termaboutwhethertheywilloccur.Iftheyallhavetinyprobabilities
116TheGoodJudgmentProjectsometimesreferstoanindicatorthatisrelevantasonethatis diagnostic ofabiggerissuethatwe
care about.
117Tetlockhascalledthisthe“ClairvoyanceTest”:ifyouaskedaclairvoyantaboutyourforecastingquestion,wouldtheybeable
toansweryouorwouldtheyrequireclarificationonwhatyoumeant.SeeTetlock,PhilipE.,andDanGardner.Superforecasting:
The art and science of prediction. Random House, 2016, and
https://www.edge.org/conversation/philip_tetlock-how-to-win-at-forecasting

52

AI Governance: A Research Agenda
(<1%)thenwewillnotlearnmuchafternotseeinganyofthemresolve.Thecloser                 118
theprobabilityofaforecastingeventandasetofpredictionsisto50%,overagiven
timeframe,themorewewilllearnaboutforecastingability,andtheworld,overthat
time frame.
5.Weideallywantthemtobe epistemicallytemporallyfractal :wewantthemtobe
suchthatgoodforecastingperformanceonnear-termforecastsisinformativeofgood
forecastingperformanceonlong-termpredictions.Near-termforecastingtargetsare
morelikelytohavethispropertyastheydependoncausalprocessesthatarelikelyto
continue to be relevant over the long-term.
6.Wewantthemtobe jointlymaximallyinformative .Thismeansthatweideallywant
asetoftargetsthatscorewellontheabovecriteria.Awayinwhichthiscouldnotbe
soisifsometargetsarehighlystatisticallydependentonothers,suchasifsomeare
logicallyentailedbyothers.Anotherheuristichereistoaimforforecastingtargets
that  exhaustively cover the different causal pathways to relevant achievements.
118Thoughweshouldlearnalotfromseeingonesuchunexpectedeventoccur.Thus,sucha“long-shot”targetwouldbea
worthwhileforecastingtargettoapersonwhoassignsintermediatesubjectiveprobabilityofitoccurring,evenifeveryoneelse
in the community is confident it will (not) occur.

53

A comprehensive document about the research agenda at the Governance of AI Program. This is really long and covers a lot of ground so I'm not going to summarize it, but I highly recommend it, even if you intend to work primarily on technical work.