The case for strong longtermism

The case for stronglongtermismHilary Greaves and William MacAskillGlobal Priorities Institute | June 2021GPI Working Paper No . 5-2021

The Case for Strong Longtermism1. Introduction22. Precisifying strong longtermism32.1 Axiological strong longtermism (ASL)32.2 Benefit ratio (BR) and ASL43. The size of the future64. Tractability of significantly affecting the farfuture104.1 Influencing the choice among persistent states104.2 Mitigating risks of premature human extinction104.3 Influencing the choice among non-extinction persistentstates134.4 Uncertainty and ‘meta’ options155. Strong longtermism about individual decisions166. Robustness of the argument177. Cluelessness197.1 Simple cluelessness207.2 Conscious unawareness207.3 Arbitrariness227.4 Imprecision237.5 Ambiguity aversion238. Fanaticism259. Deontic strong longtermism2610. Summary and conclusions30Appendix31References321
1. IntroductionAstrikingfactaboutthehistoryofcivilisationisjusthowearlyweareinit.Thereare5000yearsofrecordedhistorybehindus,buthowmanyyearsarestilltocome?Ifwemerelylastaslongasthetypicalmammalianspecies,westillhaveover200,000yearstogo(Barnoskyetal.2011);therecouldbeafurtheronebillionyearsuntiltheEarthisnolongerhabitableforhumans(WolfandToon2015);andtrillionsofyearsuntilthelastconventionalstarformations(AdamsandLaughlin1999:34).Evenonthemostconservativeofthesetimelines,wehaveprogressedthroughatinyfractionofhistory.Ifhumanity’ssagawereanovel,wewould be on the very first page.Normally,wepayscantattentiontothisfact.Politicaldiscussionsarenormallycenteredaroundthehereandnow,focusedonthelatestscandalorthenextelection.Whenapundittakesa“long-term”view,theytalkaboutthenextfiveortenyears.Withtheexceptionsofclimatechangeandnuclearwaste,weessentiallyneverthinkabouthowouractionstodaymight influence civilisation hundreds or thousandsof years hence.Webelievethatthisneglectoftheverylong-runfutureisaseriousmistake.Analternativeperspectiveisgivenbylongtermism,accordingtowhichweshouldbeparticularlyconcernedwithensuringthatthefarfuturegoeswell(MacAskillMS).Inthisarticlewegofurther,arguingforstronglongtermism:theviewthatimpactonthefarfutureisthemostimportantfeatureofouractionstoday.Wewilldefendbothaxiologicalanddeonticversionsofthisthesis.Humanity,today,islikeanimprudentteenager.Themostimportantfeatureofthemostimportantdecisionsthatateenagermakes,likewhatsubjecttostudyatuniversityandhowdiligentlytostudy,isnottheenjoymenttheywillgetintheshortterm,buthowthosedecisions will affect the rest of their life.Thestructureofthepaperisasfollows.Section2setsoutmorepreciselythethesiswewillprimarilydefend:axiologicalstronglongtermism(ASL).Thisthesisstatesthat,inthemostimportantdecisionsituationsfacingagentstoday,(i)everyoptionthatisnear-bestoverallisnear-bestforthefarfuture,and(ii)everyoptionthatisnear-bestoveralldeliversmuchlargerbenefits in the far future than in the near future.Weprimarilyfocusonthedecisionsituationofasocietydecidinghowtospenditsresources.Weusethecost-effectivenessofantimalarialbednetdistributionasanapproximateupperboundonattainablenear-futurebenefitsperunitofspending.Towardsestablishingalowerboundonthehighestattainablefar-futureexpectedbenefits,section3arguesthatthereis,inexpectation,avastnumberofsentientbeingsinthefutureofhuman-originatingcivilisation.Section4thenargues,bywayofexamplesinvolvingexistentialrisk,thattheprojectoftryingtobeneficiallyinfluencethecourseofthefarfutureissufficientlytractableforASL(i)andASL(ii)tobetrueoftheabovedecisionsituation.Section5arguesthatthesameclaimsandargumentsapplyequallytoanindividualdecidinghowtospendresources,andanindividual2
choosingacareer.Weclaimthesecollectivelyconstitutethemostimportantdecisionsituations facing agents today, so that ASL follows.The remainder of the paper explores objections andextensions to our argument.Section6arguesthatthecaseforASLisrobusttoseveralplausiblevariationsinaxiology,concerningriskaversion,prioritytotheworseoff,andpopulationethics.Section7addressestheconcernthatwearecluelessabouttheverylong-runeffectsofouractions.Section8addressestheconcernthatourargumentturnsproblematicallyontinyprobabilitiesofenormous payoffs.Section9turnstodeonticstronglongtermism.Weoutlineanargumenttotheeffectthat,accordingtoanyplausiblenon-consequentialistmoraltheory,ourdiscussionofASLalsosuffices to establish an analogous deontic thesis.Section 10 summarises.Theargumentinthispaperhassomeprecedentintheliterature.NickBostrom(2003)hasarguedthattotalutilitarianismimpliesweshouldmaximisethechancethathumanityultimatelysettlesspace.NickBeckstead(2013)argues,fromasomewhatbroadersetofassumptions,that“whatmattersmost”isthatwedowhat’sbestforhumanity’slong-termtrajectory.Inthispaper,wemaketheargumentforstronglongtermismmorerigorous,andweshowthatitfollowsfromamuchbroadersetofempirical,moralanddecision-theoreticviews. In addition, our argument in favour ofdeonticstrong longtermism is novel.Webelievethatstronglongtermismisoftheutmostimportance:thatifsocietycametoadopttheviewswedefendinthispaper,muchofwhatweprioritiseintheworldtodaywouldchange.2. Precisifying strong longtermism2.1 Axiological strong longtermism (ASL)Stronglongtermismcouldbemadepreciseinavarietyofways.First,sincewedonotassumeconsequentialism,wemustdistinguishbetweenaxiologicalanddeonticclaims.Letaxiological(resp.deontic)stronglongtermismbethethesisthatfar-futureeffectsarethemostimportant determinant of the value of our options(resp. of what we ought to do).Itremainsimprecisewhat“mostimportantdeterminant”means.Takingtheaxiologicalcasefirst, in this paper we consider the following moreprecise thesis:1Axiologicalstronglongtermism(ASL):Inthemostimportantdecisionsituationsfacingagents today,(i) Every option that is near-best overall is near-bestfor the far future.(ii)Everyoptionthatisnear-bestoveralldeliversmuchlargerbenefitsinthefarfuture than in the near future.1We discuss deontic strong longtermism in section9.3
Wherecondition(i)holds,onecanidentifythenear-bestoptionsbyfocussinginthefirstinstanceonlyonfar-futureeffects.If(aswebelieve,butwillnotarguehere)theanalogousstatementregardingnear-futureeffectsisnotalsotrue,thatsuppliesonesenseinwhichfar-futureeffectsare“themostimportant”.Wherecondition(ii)holds,theevaluationofnear-best options is primarily driven by far-futureeffects. That supplies another such sense.Insections3-5,wewillarguethatclauses(i)and(ii)ofASLholdofparticulardecisionsituations:thoseofasocietydecidinghowtospendmoneywithnorestrictionsasto‘causearea’,anindividualmakingtheanalogousdecision,andindividualcareerchoice.Becausethesedecisionsituationshaveparticularlygreatsignificanceforthewell-beingofbothpresentandfuturesentientbeings,weclaim,theyarethemostimportantsituationsfacedbyagentstoday.Therefore,stronglongtermismfollows,evenifASL(i)and(ii)donotholdofany other decision situations.Throughout,“thefarfuture”meanseverythingfromsometimetonwards,wheretisasurprisinglylongtimefromthepointofdecision(say,100years).“Thenearfuture”meansthetimefromthepointofdecisionuntiltimet.Wewillinterpretboth“near-bestoverall”and“near-bestforthefarfuture”intermsofproportionaldistancefromzerobenefittothemaximum available benefit, and “much larger” in termsof a multiplicative factor.Asweintendit,ASLisnotdirectlyconcernedwiththeobjectivevalueofoptionsandtheiractualeffects.Rather,termslike“near-best”and“benefits”relatetotheexantevalueofthoseoptions,giventheinformationavailableatthetimeofdecision,andtheirprospectsforaffectingthenearorfarfuture.Exantevaluemaybeexpectedvalue,butthestatementofASL does not presuppose this.Sinceitrefersto“benefits”,ASLmakessenseonlyrelativetoastatusquooption:benefitsareincreasesinvaluerelativetothestatusquo.Asabove,ourprimaryexampleswillbecasesofdecidinghowtospendsomeresource(eithermoneyortime).Forconcreteness,wewillthentakethestatusquotobeasituationinwhichtheresourcesinquestionaresimplywasted.However,otherplausiblechoicesofstatusquowouldbeunlikelytosignificantlyaffectourargument, and the argument does not require that thestatus quo be special in any deep sense.ASLmakesonlycomparativeclaims.Wedonotclaim,andnordowebelieve,thatoptionscannotdeliverlargebenefitswithoutbeingnear-bestforthefarfuture,orthatavailablenear-futurebenefitsaresmallinanyabsolutesense.Ourclaimisratherthatavailablebenefitsfor the far future are many times larger even thanthis.2.2 Benefit ratio (BR) and ASLOurargumentforASLproceedsviatheintermediateclaimthatthefollowingpropertyholdsof the decision situations in question:Benefitratio(BR):Thehighestfar-futureexantebenefitsthatareattainablewithoutnetnear-futureharmaremanytimesgreaterthanthehighestattainablenear-futureexantebenefits.4
WeproveintheAppendixthatifBRholdsofagivendecisionsituation,then(firstly)sodoesASL(ii),and(secondly)ASL(i)holdsofacertainrestrictionofthatdecisionsituation.(Therestrictioninvolvesremovinganyoptionsthatdonetexpectednear-futureharm;thisrestriction is innocuous in the context of our argument.)EvaluatingBR,andhenceASL,requiresquantitativeanalysis;anyparticularquantitativeanalysisrequiresstrongevaluativeassumptions.Tothisend,wewilltemporarilymakeaparticular,plausiblebutcontroversial,setofevaluativeassumptions.Section6,however,showsthatvariousplausiblewaysofrelaxingtheseassumptionsleavethebasicargumentintact.Onecontroversialassumptionthatmaybeessential,concerningthetreatmentofverysmall probabilities, is discussed in section 8.Theinessentialassumptionsinquestionincludethefollowing.First,wewillidentifytheexantevalueofanoptionwithitsexpectedvalue:theprobability-weightedaverageofthepossiblevaluesitmightresultin.Second,wewillidentifyvaluewithtotalwelfare:thatis,wewillassumeatotalutilitarianaxiology.Third,andanear-corollaryofthelatter,wewillassumetime-separability.Thelatterallowsustoseparatelydefinenear-futureandfar-futurebenefits:overallvalueisthensimplythesumofnear-futurevalueandfar-futurevalue,wherethese in turn depend only on near-future (respectively,far-future) effects.Foraroughupperboundonnear-futureexpectedbenefitsinthecontextofasocietyspendingmoney,weconsiderthedistributionoflong-lastinginsecticide-treatedbednetsinmalarialregions,whichsavesalifeonaverageforaround$4000.Each$100thereforesavesonaverage 0.025 lives in the near future (GiveWell 2020a).2Wecannotarguethatthisistheactionwiththeverylargestnear-futurebenefits.Inparticular,thoughitseemshardtobeatthiscost-effectivenesslevelviaanyinterventionthatisbackedbyrigorousevidence,itmightbepossibletoachievehighershort-termexpectedbenefitsviasomesubstantiallymorespeculativeroute.Afullexaminationofthecaseforstrong3longtermismwouldinvolveinvestigationofthis,andthecorrespondingsensitivityanalysis.However,evenquitelargeupwardadjustmentstothefigureweuseherewouldleaveourargument largely unaffected.Weemphasisethatwearenotconsideringthelong-runknock-oneffectsofbednetdistribution.Itispossible,forallwesayinthispaper,thatbednetdistributionisthebestwayof making the far future go well, though we thinkthis unlikely.4
4Itwouldamounttoa“surprisingandsuspiciousconvergence”betweennear-futureandfar-futureoptimisation(Lewis 2016).3AsisterorganisationtoGiveWell,OpenPhilanthropy,hastriedhardtofindhuman-centricinterventionsthathave more short-term impact, and has struggled (Berger2019).Theremightbemorecost-effectiveinterventionsfocusedonpreventingthesufferingofanimalslivinginfactoryfarms(Bollard2016).Weleavethisasideinordertoavoidgettingintoissuesofinter-speciescomparisons;again, there is a corresponding need for sensitivityanalysis.2FollowingGiveWell(2018b),wewillassumethattheshort-termbenefitsoftheinterventionsthatdothemostshort-run good would scale proportionately even ifvery large amounts of money were spent.5
Wewillargueinsection4that,forasociety’sdecisionabouthowtospenditsresources,thelowerboundonattainablefar-futureexpectedbenefitsismanytimeshigherthanthisupperboundfornear-futureexpectedbenefits,andthereforeBRholdsofthisdecisionsituation.Section 5 discusses related decision situations facingindividuals.3. The size of the futureThereis,inexpectation,avastnumberoflivesinthefutureofhumancivilisation.Any5estimateofjusthow“vast”isofcourseapproximate.Nonetheless,wewillargue,existingwork supports estimates that are sufficiently largefor our argument to go through.Thereareseveraltechniquesonecanuseforestimatingtheexpectednumberoffuturebeings.Letusstartwiththequestionoftheexpecteddurationofhumanity’sfutureexistence,temporarily setting aside questions of how large thepopulation might be at any future time.Firstly,onemightuseevidenceregardingtheageofourspeciestogroundjudgmentsontheannualriskofextinctionfromnaturalcauses,andextrapolatefromthere.GiventhatHomosapienshasexistedforover200,000years,Snyder-Beattieetal.(2019:2)therebyestimatethattheexpectedfuturelifespanofhumanityisatleast87,000years,asfarasnaturalcausesof extinction are concerned.Secondly,onemightundertakereferenceclassforecasting(KahnemanandLovallo1993;Flyvbjerg2008).Here,thelifespansofothersufficientlysimilarspeciesserveasbenchmarks.Estimatesoftheaveragelifespanofmammalianspecies(resp.hominins)arebetween0.5and6millionyears(resp.around1millionyears)(Snyder-Beattieetal.2019:6).Thusreferenceclassforecasting,naivelyapplied,suggestsatleast1millionyearsfortheexpectedfutureduration of humanity.Bothoftheseestimates,however,ignorethefactthathumanstodayarehighlyatypical.Humanitytodayissignificantlybetterequippedtosurviveextinction-levelthreatsthaneitherotherspeciesare,orthanourownspecieswasinthepast,thankstoacombinationoftechnologicalcapabilitiesandgeographicaldiversity.Thereforearangeofsubstantiallyhigherbenchmarksisalsorelevant:forinstance,thefrequencyofmassextinctionevents(1inevery30-100millionyears(Snyder-Beattieetal.2019:7)),andthetimeoverwhichtheEarthremains habitable for humans (around 1 billion years(Adams 2008:34)).Theabovefiguresconcerntheexpecteddurationofhumanity’sfuture.Sinceweareinterested in the expectednumberof future beings,we also need to consider population size.Weagainconsiderseveralbenchmarks.First,theUNDepartmentofEconomicandSocialAffairs(2019:6)projectsthattheglobalpopulationwillplateauataround11billionpeoplebytheyear2100.Second,thelargemajorityofestimatesoftheEarth’s“carryingcapacity”–thatis,itslong-runsustainablepopulation,basedonrelativelyconservativeassumptionsaboutfuturetechnologicalprogress–areover5billion,andsometimessubstantiallyhigher5Wewilluse‘human’toreferbothtoHomosapiensandtowhateverdescendantswithatleastcomparablemoral status we may have, even if those descendantsare a different species, and even if they are non-biological.6
(Cohen1998:342;BerghandRietveld2004:197).Third,forpredictingthefurtherfuture,wemightextrapolatefromthehistoricaltrendofhumanpopulationincreasingovertime.Giventhistrend,itisatleastplausiblethatcontinuedtechnologicaladvanceswillenableanevenlargerfuturepopulationuptosomemuchhigherplateaupoint(say,1trillion),evenifwecannot currently foresee the concrete details of howthat might happen (Simon 1998).Importantly,itistheexpectednumberoffuturebeings,notthemedian,thatisrelevantforourpurposes.Inadditiontothepossibilityofnumberslikethehigherbenchmarksindicatedabove,itisofcoursealsopossiblethatthefuturedurationand/orpopulationsizeofhumanityaremuchsmaller.However,theeffectsofthesepossibilitiesontheexpectednumberare6highlyasymmetric.Evena50%credencethatthenumberoffuturebeingswillbezerowoulddecreasetheexpectednumberbyonlyafactoroftwo.Incontrast,acredenceassmallas1%thatthefuturewillcontain,forexample,1trillionbeingspercenturyfor100millionyears(ratherthan10billionpercenturyfor1millionyears)increasestheexpectednumberbyafactor of 100.Wemustalsoconsidertwomoreradicalpossibilitiesthat,whileveryuncertain,couldgreatlyincreasethedurationandfuturepopulationsizesofhumanity.Thefirstconcernsspacesettlement.Therearecurrentlynoknownobstaclestotheviabilityofspacesettlement,andsomescientificinvestigationssuggestingitsfeasibilityusingknownscience(SandbergandArmstrong2013;Beckstead2014).IfhumanitylivesnotonlyonEarthbutalsoonotherplanets—inourownsolarsystem,elsewhereintheMilkyWay,orinothergalaxiestoo—thenterrestrialconstraintsonfuturepopulationsizedisappear,andastronomicallylargerpopulationsbecomepossible.Evenifweonlysettlethesolarsystem,civilisationwouldhaveover5billionyearsuntiltheendofthemainsequencelifetimeoftheSun(Sackmannetal.1993:462;SchröderandSmith2008:157-8),andwewouldhaveaccesstoovertwobilliontimesasmuchsunlightpoweraswedonow(Stix2002:6;SarbuandSebarchievici2017:16).IfweareabletowidelysettletherestoftheMilkyWay,thenwecouldaccesswellover250millionrockyhabitable-zoneplanets(Brysonetal.2021:22),eachofwhichhasthepotentialtosupporttrillionsoflivesoverthecourseoftheirsun’slifetimes.Moreover,aninterstellarcivilisationcouldsurviveuntiltheendofthestelliferousera,ontheorderoftentrillionyearshence(AdamsandLaughlin1999).Ifweconsiderpossiblesettlementofthebillionsofothergalaxies accessible to us, the numbers get dramaticallylarger again.Thesecondradicalpossibilityisthatofdigitalsentience:thatis,consciousartificialintelligence(AI).Theleadingtheoriesofphilosophyofmindsupporttheideathatconsciousnessisnotessentiallybiological,andcouldbeinstantiateddigitally(Lewis1980;Chalmers1996:ch.9).AndthedramaticprogressincomputingandAIoverjustthepast70yearsshouldgiveusreasontothinkthatifso,digitalsentiencecouldwellbearealityinthe6Onduration:technologicalprogressbringsnotonlyprotectionagainstexistingextinctionrisks,butalsonovelsourcesofextinctionrisk(Ord2020:esp.chs.4and5).Onpopulationsize:thetendencyforrichersocietiestohavelowerfertilityrateshasledsometoconjecturethathumanpopulation,afterplateauingaround2100,mightsignificantlydeclineintotheindefinitefuture,ahigh“carryingcapacity”notwithstanding(BrickerandIbbitson2019).7
future.Itisalsoplausiblethatsuchbeingswouldhaveatleastcomparablemoralstatustohumans (Liao 2020), so that they count for the purposesof the arguments in this paper.7Considerationofdigitalsentienceshouldincreaseourestimatesoftheexpectednumberoffuturebeingsconsiderably,intwoways.First,itmakesinterstellartravelmucheasier:itiseasiertosustaindigitalthanbiologicalbeingsduringverylong-distancespacetravel(Sandberg2014:453).Second,digitalsentiencecoulddramaticallyincreasethenumberofbeingswhocouldlivearoundonestar:digitalagentscouldliveinamuchwidervarietyofenvironments(Sandberg2014:453),andcouldmoreefficientlyturnenergyintoconsciouslife(Bostrom 2003:309).Onemightfeelscepticalaboutthesescenarios.Butgiventhattherearenoknownscientificobstaclestothem,itwouldbeoverconfidenttobecertain,ornear-certain,thatspacesettlement,ordigitalsentience,willnotoccur.Imaginethatyoucouldpeerintothefuture,andtherebydiscoveredthatEarth-originatingcivilisationhasspreadacrossmanysolarsystems.Howsurprisedwouldyoube,comparedtohowsurprisedyouwouldbeifyouwonthe lottery?Tomovetowardsparticularnumbers,weconsiderthreespecificfuturescenarios,takenfromNewberry(2021a),wherecivilisationis:(i)Earthbound;(ii)limitedtotheSolarSystem;and(iii)expandedacrosstheMilkyWay.Ineachcase,Newberrymakesaconservativeestimateofthecarryingcapacityofcivilisationinthatscenario,ontheassumptionsthatdigitallifeisandisnotpossible,givingsixscenariosinall.Healsoprovidesabest-guessestimateofthedurationofcivilisationinthatscenario.Thesescenariosarenotmeanttoexhaustthepossibilityspace,buttheygiveanindicationofthepotentialmagnitudesoffuturepopulationsize:ScenarioDuration (centuries)Carrying capacity(lives per century)Number of futurelivesEarth (mammalianreference class)10410101014Earth (digital life)10410141018Solar System10810191027Solar System (digitallife)10710231030Milky Way101110251036Milky Way (digitallife)101110341045
7We return to the likelihood of artificial superintelligencein section 4.3.8
Toarriveatanoverallestimateoftheexpectednumberoffuturepeople,onewouldfurtherneedtoestimateprobabilitiesforscenariossuchasthoseabove(andforallotherscenarios).However,sincethenumberoflivesinthefutureaccordingtodifferentpossiblescenariosisspreadovermanyordersofmagnitude,inanysuchexpectedvaluecalculation,ittendstobethe“largest”scenarioinwhichonehasanynonzerocredencethatdrivestheoverallestimate.Evena0.01%credencethatbiologicalhumanitysettlestheMilkyWayatcarryingcapacity,forexample,contributesatleast1032totheexpectednumberoffuturebeings.Preciselyhowone’sremainingcredenceisspreadamong“smaller”scenariosthenmakesverylittledifference.Becauseofthis,webelievethatanyreasonableestimateoftheexpectednumberoffuturebeingsisatleast1024.(Infact,webelievethatanyreasonableestimatemustbesubstantiallyhigherthanthis;sincehighernumberswouldmakelittledifferencetotheargumentsofthispaper,however,wewillnotpressthatcasehere.)However,wearealsosympathetictotheconcernthatifthisistheonlyestimateweconsider,thecaseforstronglongtermismwouldbedrivenpurelybytinycredencesinhighlyspeculativescenarios.Wewillthereforealsoconsidertheextenttowhichthesameargumentswouldgothroughonsomevastlymoreconservative estimates, as follows:Expected number of future beingsMain estimate1024Low estimate1018Restricted estimate1014Ourlowestimate(1018)corresponds,forinstance,toa0.0000001%credenceintheSolarSystem(biologicallife)scenario,withzerocredenceineitherdigitalsentienceormorewide-rangingspacesettlement.Ourrestrictedestimate(1014)correspondstotheaboveestimateforEarthboundlife,withzerocredenceinanylarger-populationscenario(includingbothdigitalsentienceandanyspacesettlement).Intheargumentsthatfollow,thereaderisinvited to substitute her own preferred estimate throughout.WewillarguethatBR(andhenceASL)holdsofsociety’sdecisionsituationevenonourrestricted estimate, and clearly holds by a largemargin on our main estimate.4. Tractability of significantly affecting the farfutureThefar-futureeffectsofone’sactionsareusuallyhardertopredictthantheirnear-futureeffects.Mightitbethattheexpectedinstantaneousvaluedifferencesbetweenavailableactionsdecaywithtimefromthepointofaction,anddecaysufficientlyfastthatinfactthenear-futureeffectstendtobethemostimportantcontributortoexpectedvalue?Ifthatwereso, then neither BR nor ASL would hold.9
Thisisanaturalreasontodoubtstronglongtermism.Wewillcallitthewashing-outhypothesis.8Weagreethatthewashing-outhypothesisistrueofsomedecisionsituations.However,weclaim that it is false of our society’s decision situation.Giventheargumentofsection2,ourtaskistoshowthatthereexistsatleastoneoptionavailabletosocietywiththepropertythatitsfar-futureexpectedbenefitsaresignificantlygreaterthanthenear-futureexpectedbenefitsofbednetdistribution(thatis,recall:0.025livessavedper$100spent).Wewillconsiderexamplesintwocategories:mitigatingextinctionrisk, and positively shaping the development of artificialsuperintelligence.4.1 Influencing the choice among persistent statesHereisanabstractstructurewhich,insofarasitisinstantiatedintherealworld,offersarecipe for identifying options whose effects willnot wash out.ConsiderthespaceSofallpossiblefine-grainedstatestheworldcouldbeinatasinglemomentoftime(thatis,thespaceofallpossibleinstantaneousmicrostates).Onecanpicturethehistoryoftheuniverseasapaththroughthisspace.LetapersistentstatebeasubsetAofSwiththepropertythat,giventhedynamicsoftheuniverse,iftheinstantaneousstateoftheworldisinA,thentheexpectedtimeforwhichitremainsinAisextremelylong.Nowsupposethattherearetwoormoresuchpersistentstates,differingsignificantlyfromoneanotherinvalue.Supposefurtherthattheworldisnotyetinanyofthestatesinquestion,butmightsettleintooneortheotherofthestatesinquestionintheforeseeablefuture.Finally,supposethatthereissomethingwecandonowthatchangestheprobabilitythattheworldendsupinabetterratherthanaworsepersistentstate.Then,asaresultofthepersistencethatis built into the definition, the effects of theseactions would not wash out at all quickly.Theempiricalquestioniswhetherthereare,intherealworld,anyoptionsavailablethatinstantiate the structure just described. We claimthat there are.4.2 Mitigating risks of premature human extinctionThenon-existenceofhumanityisapersistentstateparexcellence.Tostatetheobvious:thechancesofhumanityre-evolving,ifwegoextinct,areminiscule.Onlyslightlymoresubtly,theexistenceofhumanityisalsoapersistentstate:whilewefacesignificantrisksofpremature extinction, as argued in section 3, humanity’sexpectedpersistence is vast.Thesepersistentstateshaveunequalexpectedvalue.Assumingthatonaveragepeoplehavelivesofsignificantlypositivewelfare,accordingtototalutilitarianismtheexistenceof9humanityissignificantlybetterthanitsnon-existence,atanygiventime.Combiningthiswith
9We return to this assumption in section 6.8Itisimportantheretodistinguishbetweenexanteandexpostversionsofthewashing-outclaim.Theexpostversionisfalse,asisestablishedbytheliteratureoncluelessness;cf.section7.1.However,itisthemuchmoreplausibleex antewashing-out claim that is relevantto the arguments of this paper.10
thefactthatbothstatesarepersistent,prematurehumanextinctionwouldbeastronomicallybad.Correspondingly,evenanextremelysmallreductioninextinctionriskwouldhaveveryhighexpectedvalue(Bostrom2013:18).Forexample,evenifthereare‘only’1014livestocome(asonourrestrictedestimate),areductioninnear-termriskofextinctionbyonemillionthofonepercentagepointwouldbeequivalentinvaluetoamillionlivessaved;onourmainestimateof1024expectedfuturelives,thisbecomestenquadrillion(1016)livessaved.Asisincreasinglyrecognised,asanempiricalmatteroffact,therearethingswecoulddothatwouldreducethechanceofprematurehumanextinctionbyanon-negligibleamount.Asaresult,althoughpreciseestimatesoftherelevantnumbersaredifficult,thefar-futurebenefitsofsomesuchinterventionsseemtocompareveryfavourably,bytotalutilitarianlights,tothehighest available near-future benefits.Thedetectionandpotentialdeflectionofasteroidsprovidesarelativelyrobustexampleofsuchanintervention.ThisinvolvesscanningtheskiestoidentifyasteroidsthatcouldpotentiallycollidewithEarthand,ifonewerefound,investingtheresourcestotrytodeflectit,and/ortopreparebunkersandfoodstockpilestohelpussurviveanimpactwinter.Mostoftheexpectedcostshereareindetection,becausethecostsofdeflectionandpreparationareonlypaidintheveryunlikelyeventthatonedoesdetectanasteroidonacollisioncoursewithEarth.In1996,NASAcommencedtheSpaceguardSurvey,amulti-decadeplantotracknear-Earthobjectswiththeaimofidentifyinganyonimpacttrajectories.Atatotalcostof$71 million(USD)by2012,theSpaceguardSurveyhadtrackedover90%ofasteroidsofdiameter1kmor more in near-Earth orbit, and all asteroids ofdiameter 10km or more over 99% of the sky.Itisnotcertainthatalargeasteroidcollisionwouldcausehumanbeingstogoextinct.Weassumeastatusquoriskofhumanextinction,conditionalontheimpactofa10km+asteroid,of1%.Itisalsofarfromcertainthatwecoulddeflecta10km+asteroid,evenifweknewitwasonacollisioncourse.However,itisfarfromcertainthatwecouldnot,and,asabove,thereareotheractionswecouldtaketoprotectagainsttheextinctionrisk.Weassumeherethatifsuchanobjectweredetectedtobeonacollisioncourse,ourdeflectionandpreparationeffortswouldreduceextinctionriskbyaproportional5%.Theassumptionsinthisparagraphfollow Newberry (2021b), and seem fairly conservative.Puttingthesenumberstogether,weestimatethattheSpaceguardSurvey,onaverage,reducedextinctionriskbyatleast5×10-16per$100spent.Onourmainestimateoftheexpectednumberoffuturebeings,thisamountsto500additionalmillionlives;thisdecreasesto500or0.05 lives on our low and restricted estimates, respectively.Ofcourse,weshouldexpectfurtherworkonasteroidstohavelowercost-effectiveness,becauseofdiminishingmarginalreturns.However,theopportunityremainssignificant.Theremainingriskofa10km+asteroidcollisioninthenext100yearshasbeenestimatedat1in150million(Ord2020:71).Ithasbeenestimatedthatthecosttodetectwithnear-certainty11
anyremainingasteroidsofgreaterthan10kmdiameterwouldbeatmostafurther$1.2billion(Newberry2021b).Onourmain(resp.low,restricted)estimateoftheexpectednumberoffuturebeings,every$100ofthiswould,onaverage,resultin300,000(resp.0.3,0.00003)additionallives.Thisexamplethereforesupportsstronglongtermismonourmainandlowestimates,thoughnotontherestrictedestimate.Organisationswhoseworkmitigatesriskofextinctionfromasteroidimpacts,andwhichwouldbenefitfromsubstantiallymorefunding,include the Planetary Society and the B612 Foundation.Whileasteroiddefenseisamongthemoreeasilyquantifiedareasofextinctionriskreduction,itisfarfromtheonlyone,orthemostsignificant(Ord2020:ch.3).Anotherpossibilityconcernsglobalpandemics.Suchapandemiccouldbenaturalorman-made,withthelatterbeingparticularlyconcerning(Posner2004:75-84;Rees2018:sec.2.1;Ord2020).Inparticular,progressinsyntheticbiologyisveryrapid(MengandEllis2020),anditislikelythatwewillsoonbeabletodesignman-madeviruseswithveryhighcontagiousnessandlethality.Ifsuchpathogenswerereleased(whetherdeliberatelyorbyaccident(Shulman2020;Ord2020:129–131))inthecourseofmilitarytensions,orbyaterroristgroup,thereisarealpossibilitythattheycouldkillasufficientnumberofpeoplethatthehumanspecieswould not recover.Inarecentpaper,MilletandSnyder-Beattie(2017)usethreedistinctmethodstogenerateestimatesoftheriskofanextinction-levelpandemicinthenext100years.Theresultingestimatesrangefrom1in600,000to1in50.TheauthorsfurtherusefiguresfromtheWorldBanktogenerateaveryconservativeestimatethat$250billionofspendingonstrengtheninghealthcaresystemswouldreducethechanceofsuchextinction-levelpandemicsthiscomingcentury by at least a proportional 1%.10Takingthegeometricmeantoaverageacrossthetwomethodsthatgeneratethelowerestimatesforextinctionrisk,weobtainariskofabout1in22,000ofextinctionfromapandemicoverthenext100years.Ifweusetheabovefigureof$250billiontoreducethe11riskby1%,andassumethattheriskreductionoccursthroughoutthenext100yearsbutonlyinthattimeperiod,theneach$100ofsuchspendingwould,inexpectation,increasethenumberoffuturebeingsby200million(respectively,200,0.02)onourmain(resp.,low,restricted)estimate.Accordingtothesecalculations,thefar-futurebenefitswouldtherebysignificantlyexceedthenear-futurebenefitsofbednetdistributiononourmainandlowestimatesofthesizeofthefuture,thoughnotonourrestrictedestimate.Organisations
11Weusethegeometricratherthanthearithmeticmeanbecausetheestimatesinquestionarespreadacrossseveralordersofmagnitude;thearithmeticmeaneffectivelydeferstothehighestestimateonthequestionoforderofmagnitude.Usingthearithmeticmeanwouldleadtoresultsthatarestillmorefavourabletostronglongtermism.Similarly,wedisregardMilletandSnyder-Beattie’s“Model1”because,astheauthorsnote,thismodelisflawedinimportantrespects;includingthismodelwouldalsostrengthenthecaseforstronglongtermism.10TwowaysinwhichMilletandSnyder-Beattie’sestimateisparticularlyconservativeare(i)thatthe$250bnfigureisattheextremeupperendofanticipatedcostsfortheinterventiontheydiscuss,and(ii)thattheinterventioninquestionconcernsanextremelybroad-basedapproachtobiosecurity,notspecificallyoptimisingfor extinction risk reduction.12
workingonthesethreatsincludetheJohnHopkinsCenterforHealthSecurity,theNuclearThreat Initiative’s biosecurity program, and GryphonScientific.4.3 Influencing the choice among non-extinction persistentstatesAsecondwayofpositivelyimpactingthelongrunisbyimprovingthevalueofthefutureconditionalontheexistenceofaverylargenumberoffuturesentientbeings.Forconcreteness,wefocusononewayofdoingthis:positivelyshapingthedevelopmentofartificialsuperintelligence(ASI),thatis,artificialsystemsthatgreatlyexceedthecognitiveperformance of humans in virtually all domains ofinterest.12TheideathatthedevelopmentofsufficientlyadvancedartificialintelligencecouldproveakeyturningpointinhistorygoesbacktotheearlycomputerpioneersAlanTuring(1951)andI.J.Good(1966).Ithasmorerecentlybeenthesubjectofwiderconcern.Therearetwo13classes of long-term worry.ThefirstisfromAI-takeoverscenarios(Bostrom2014;Russell2019).Thisworryisthat,oncewebuildahuman-levelartificialintelligence,itwouldbeabletorecursivelyself-improve,designingever-betterversionsofitself,quicklybecomingsuperintelligent.Fromthere,inordertobetterachieveitsaims,itwilltrytogainresources,andtrytopreventthreatstoitssurvival.Itwouldthereforebeincentivisedtotakeovertheworldandeliminateorpermanentlysuppresshumanbeings.BecausetheASI’scapabilityissomuchgreaterthanthat of humans, it would probably succeed in theseaims.Thesecondworryisfromentrenchmentscenarios(MacAskillMS).IfanauthoritariancountrywerethefirsttodevelopASI,withasufficientlead,theycouldusethistechnologicaladvantagetoachieveworlddomination.Theauthoritarianleadercouldthenquashanyideologicalcompetition.AnAIpoliceforcecouldguaranteethatpotentialrebellionsareprevented;anAIarmywouldremoveanypossibilityofacoup.Andiftheleaderwantedhisideologytopersistindefinitely,hecouldpasscontrolofsocietyontoanASIsuccessorbeforehisdeath.Tothisend,hecouldhard-codethegoalsoftheASItomatchhisown,havetheASIlearnhisgoalsfromhisspeechandbehaviour,oreven‘mindupload’,scanninghisbrainand having it digitally emulated (Sandberg and Bostrom2008; Sandberg 2013).Ineitherofthesescenarios,oncepowerovercivilisationisinthehandsofanASI,thiscouldpersistaslongascivilisationdoes(RiedelMS).DifferentversionsoftheASI-controlledfuturesarethereforepersistentstateswithsignificantlydifferingexpectedvalue,sothatwehaveanotherinstantiationofthestructureoutlinedinsection4.1.Theruler-ASIcouldmonitoreveryaspectofsociety.Anditcouldreplicateitselfindefinitely,justaseasilyaswecanreplicatesoftwaretoday;itwouldbeimmortal,freedfromthebiologicalprocessof13ThoseconcernedincludeleadingmachinelearningresearcherssuchasStuartRussell(2019)andShaneLegg(2008:sec.7.3),philosopherssuchasNickBostrom(2014),EliezerYudkowsky(2013),TobyOrd(2020:138–152)andRichardNgo(2020),physicistssuchasMaxTegmark(2017:ch.4)andStephenHawking(2018: ch. 9), and tech entrepreneurs such as ElonMusk (2014), Sam Altman (2015) and Bill Gates (Statt2015).12Otherareasonemightconsiderhereincludeaffectingthevaluesthattheworldconvergeson(Reese2018),orreducing the risk of a totalitarian world government(Caplan2008).13
aging.Thevalueoftheresultingworldwoulddependinconsiderablepartonthegoalsoftheruler-ASI.Thoughextinctionrisksinvolvedramaticreductionsinthesizeofthefuturepopulation,theseAIscenariosneednot.IntheclassicstatementoftheAI-takeoverscenario,theASIgoesontosettlethestarsinpursuitofitsgoals(Bostrom2014:100).Similarly,ifanauthoritarianleadertransferredpowertoanASI,theytoomightwanttheircivilisationtobelarge,populousandlong-lasting.Inparticular,forawidevarietyofgoals(suchasbuildingthegrandestpossibletemples,doingresearch,or,inatoyexampleBostrom(2014:123-4)givestoillustratethegeneralphenomenonofmisalignedAI,maximisingthenumberofpaperclips),acquiringmoreresourceshelpswithachievementofthesegoals,whichmotivatessettlingthestars.And,inordertofulfillthesegoals,apopulousworkforcewouldbeinstrumentallyvaluable.Inexpectation,thenumberoffuturebeings,inthesescenarios,isvery large.Now,thisworkforcemightconsistalmostentirelyofAIs.But,aswenotedinsection3,therearereasonstothinkthatsuchbeingswouldhavemoralstatus,andthereforehowwellorpoorlytheirliveswentwouldbeofmoralconcern,relevanttotheargumentsofthispaper.And,atleastontheauthoritarian-takeoverscenarios,therulermightwishtohaveaverylargenumber of human followers, too.TherearetwostrandsofworkaimedatreducingrisksfromASI.First,AIsafetyresearch,whichaimstoensurethatAIsystemsdowhatweintendthemtodo(Amodeietal.2016).SuchworkisconductedbyorganisationssuchasBerkeley’sCenterforHuman-CompatibleAI,theMachineIntelligenceResearchInstitute,andlabswithinGoogleDeepMindandOpenAI.Second,policywork,inparticulartoensureacooperativeapproachbetweencountriesandcompanies:forexample,byThePartnershiponAI,theCentrefortheGovernance of AI, and the Center for New AmericanSecurity.Despitethiswork,ASIsafetyandpolicyarestillextremelyneglected.Forexample,theOpenPhilanthropyProjectistheonlymajorfoundationwiththeseissuesasakeyfocusarea;itspendsunder$30millionperyearonthem(OpenPhilanthropy2020).TheAIsafetyteams14at OpenAI and DeepMind are small.Thereisnohardquantitativeevidencetoguidecost-effectivenessestimatesforAIsafetywork.Expertjudgment,however,tendstoputtheprobabilityofexistentialcatastrophefromASIat1-10%.Giventhesesurveyresultsandtheargumentswehavecanvassed,wethink15
15Graceetal.(2018)asked352leadingAIresearcherstogiveaprobabilityonthesizeofexistentialriskarisingfromthedevelopmentof‘human-levelmachineintelligence’;themedianestimatewas5%.Asurveyamongparticipantsataconferenceonglobalcatastrophicriskssimilarlyfoundthemedianestimatetobe5%(SandbergandBostrom2008).Onewouldexpectaselectioneffecttobeatworkinsurveysofthosewhohavechosentowork on existential risk, but not so (or not strongly)for the survey of AI researchers.14Neglectednessiscrucialtotheargumentofthispaper.Wouldstronglongtermismstillbetrueif,forexample,10%ofglobalGDPwerealreadyspentonthemostvaluablelong-term-orientedinterventions?Eveniftrue,woulditstillbesignificantlyrevisionarycomparedtoanear-termistapproach,aswehaveclaimeditisatthecurrent margin? We aren’t sure. Our claim here isonly that the world today is clearly far below thisoptimum.14
thatevenahighlyconservativeassessmentwouldassignatleasta0.1%chancetoanAI-drivencatastrophe(asbadasorworsethanhumanextinction)overthecomingcentury.Wealsoestimatethat$1 billionofcarefullytargetedspendingwouldsufficetoavoidcatastrophicoutcomesin(attheveryleast)1%ofthescenarioswheretheywouldotherwiseoccur.Ontheseestimates,$1 billionofspendingwouldprovideatleasta0.001%absolutereductioninexistentialrisk.Thatwouldmeanthatevery$100spenthad,onaverage,animpactasvaluableassavingonetrillion(resp.,onemillion,100)livesonourmain(resp.low, restricted) estimate – far more than the near-futurebenefits of bednet distribution.4.4 Uncertainty and ‘meta’ optionsThereisalotofuncertaintyinthenumberswehavegiven,eveninthemostscientificallyrobustcaseofasteroiddetection.Wewillgivethisissueamorethoroughtreatmentinthenextsection,arguingagainstvariouswaysinwhichonemightworryitunderminesourargument.Onethingthatuncertaintycansupport,however,isapreferencefordifferenttypesofstrategytoimprovethefarfuture.Ratherthandirectlytryingtoinfluencethefarfuture,onecouldinsteadtrytoinvestindecision-relevantresearch,orinvestone’sresourcesforuseatalaterdate.Thepossibilityofeitherofthesestrategiesstrengthensourargumentconsiderably.Toseethis,letussuppose,forthesakeofargument,thatno‘first-order’intervention(suchasthosewediscussedinsections4.2-3)delivershigherfar-futureexpectedbenefitsthanthehighestavailablenear-futureexpectedbenefits,relativetothecredencesthatareappropriateinthepresentstateofinformation.Suppose,however,thatitishighlylikelythatconditionalonsufficientadditionalinformation,atleastoneoftheproposedinterventions,oranothersuchintervention(notyetconsidered)inasimilarspirit,wouldhavemuchhigherfar-futurebenefits,relativetotheupdatedcredences,thanthebestavailablenear-futurebenefits.Thensocietymightfundresearchintothecost-effectivenessofvariouspossibleattemptstoinfluencethefarfuture.Providedthatsubsequentgovernmentsorphilanthropistswouldtakeduenoteoftheresults,this‘meta-option’couldeasilyhavemuchgreaterfar-futureexpectedbenefitsthanthebestavailablenear-futureexpectedbenefits,sinceitcoulddramaticallyincreasetheexpectedeffectivenessoffuturegovernmentalandphilanthropicaction(allrelative tocurrentlyappropriate credences) .Acomplementarypossibilityisthatratherthanspendingnow,societycouldsaveitsmoneyforalatertime(Christiano2014;MacAskill2019;Trammell2020).Thatis,itcouldsetupasovereignwealthfund,withalongtermistmission.Thisfundwouldpayoutwhenevertherebecomesavailablesomeactionthatwillsufficientlybenefitthefarfuture(inexpectation),whetherthatisduringthelifetimesofcurrentcitizensorlater.Therewouldbesomeannualriskoffuturegovernmentsbeingmisalignedandusingthemoneypoorly,butthisriskcould15
bemitigatedviaconstitutionalenshrinementofthemission,andwouldbecompensatedbythe fact that the fund would benefit from compoundreturns of investment.16Theseconsiderationsshowthatthebarthat‘intractability’objectionstoourargumentmustmeetisveryhigh.ForBRtofailtoholdonsuchgrounds,everyoptionavailabletosocietymusthavenegligibleeffectonthefarfuture’sexpectedvalue.Moreover,itmustbenear-certainthattherewillbenosuchactionsinthefuture,andthatnosuchactionscouldbediscovered through further research. This constellationof conditions seems unlikely.5. Strong longtermism about individual decisionsSofarwehavediscussedwhatisbestforasocietytodo,sometimesreferringtowhatbillionsof dollars would be able to achieve. But what aboutindividuals?Webelieveourargumentsapplytoindividualsinmuchthesamewaytheyapplytosocietyasawhole.SupposeShivaniisanindividualphilanthropist,decidingwheretospendhermoney.Naively,wemightthinkofShivaniasmakingacontributiontoasteroiddetection,pandemicpreparedness,orAIsafetythatisproportionaltoherresources.If$1billioncandecreasethechanceofanasteroidcollisionthiscenturyby1in120,000,then$10,000candecreasethechanceofanasteroidcollisionby1in12billion.Becausetheindividual’sabilitytocontributetoshort-termgoodwouldalsodecreaseproportionally,perhapstheargumentgoesthrough in just the same way.This“naive”argumentis,inourview,approximatelycorrect.Weforeseethreewaysofresisting it.First,onecouldclaimthatprivateindividualsaremuchmorelimitedintheiroptions,tosuchanextentthatShivanicandonothingtodecreaserisksfromasteroids,pandemics,orAI.However,thisissimplynottrue.Multipleorganisationsworkingontheserisks,including16Plausibly,thegainsfromtheinvestmentwouldoutweightheriskofvalue-driftofthefund:thehistoricalrealrateofreturnonriskyinvestments(suchasstocksandhousing)wasaround7%duringtheperiod1870-2015(Jordàetal.2019:1228).Itseemsreasonabletoexpectsubstantiallylowerreturnsinthefuture;butevenifso,theywouldstillbesignificantlyhigherthantheriskoffuturegovernmentsmisusingthefunds;evena90%probabilityofafuturegovernmentmisusingthefundsoverthenextcenturywouldamounttoonly2%annualrisk.Thereissomeprecedentforsuccessfullong-lastingtrustsinthecharitablesector.IntheUStheJohnClarkeTrustwasfoundedin1676(Ochs,2019);intheUK,King’sSchool,Canterburywasestablishedin597(HouseofCommonsPublicAdministrationSelectCommittee,2013).In1790BenjaminFranklininvested£1000foreachofthecitiesofBostonandPhiladelphia:¾ofthefundswouldbepaidoutafter100years,andtheremainderafter200years.By1990,thedonationhadgrowntoalmost$5millionforBostonand$2.3millionforPhiladelphia(Isaacson2003:473–474).Theoldestsimilargovernmentfundsdatebacktothemid-19thcentury:Texas’sPermanentSchoolFundwasfoundedin1854(TexasEducationAgency2020),anditsPermanentUniversityFundwasfoundedin1876(UniversityofTexasSystem2021).Iftheannualchanceoffailureofsuchfundswereashighas2%,thenthechanceoftheTexasPermanentSchoolFundpersistinguntilthepresentdaywouldbeoneinthirty,andthechanceoftheKing’sSchoolpersistinguntilthepresentdaywouldbeoneintentrillion.Thisdoesnotmerelyappeartobeaselectioneffect:toourknowledge,itisnotthecasethattherehavebeenverylargenumbersofattemptedlong-lastinggovernmentfundsthathavefailed.Thissuggests that 2% is a conservatively high estimateof the annual risk of failure.16
mostofthosewementionedabove,acceptfundingatallscalesfromprivateindividuals,andwould scale up their activity in response.Second,onecouldclaimthatthereareincreasingreturnstoscale,sothattheimpactofasmalldonationismuchlessthantherelevantfractionoftheimpactofalargedonation.Thisisanopenpossibility,butitseemssignificantlymorelikelythattherearefairlystronglydiminishingreturns,hereaselsewhere.Thisisforboththeoreticalandempiricalreasons.17Theoretically:sinceinterventionsvaryintheirexantecost-effectiveness,arationalaltruisticactorwillfundthemostcost-effectiveinterventionfirst,beforemovingtothenext-mostcost-effectiveintervention,andsoon.Empirically,diminishingreturnshavebeenobservedacross many fields (e.g. Cassman et al. 2002:134;Arnold et al. 2018; Bloom et al. 2020).Third,onecouldclaimthat,onceweconsidertheactionsofindividualswithsmalleramountsofresources,theprobabilityofsuccessfromdirectingthoseresourcestolong-termorientedinterventionsbecomessolowthatexpectedutilitytheorygivesthewrongrecommendations.We discuss this issue in section 8.Whatofindividualdecisionsaboutwheretodirectone’slabour,ratherthanone’smoney?Webelievethatmuchthesameargumentsapplyhere.SupposethatAdamisayounggraduatechoosinghiscareerpath.Adamcanchoosetotraineitherasadevelopmenteconomist,orasanAIsafetyresearcher.WhiletherearedifferencesbetweenAdam’sdecisionsituationandShivani’s(MacAskill2014),therearealsoimportantsimilarities.Inparticular,theconsiderationsthatmakeitbetterinexpectationforShivanitofundAIsafetyratherthandevelopingworldpovertyreductionsimilarlyseemtomakeitbetterinexpectationforAdamtotrainasanAIsafetyresearcherratherthanasadevelopmenteconomist.6. Robustness of the argumentInourinitialpresentationoftheargument,wehaveattimesassumedexpectedtotalutilitarianism,forsimplicity.Thisraisesanimportantquestionofhowwideaclassofaxiologies will support axiological strong longtermism.First,whatifinsteadofmaximisingexpectedtotalwelfare,thecorrectaxiologyisriskaverse?Thisinfactseemstostrengthenthecaseforstronglongtermism:thefar-future18interventionswehavediscussedaremattersofmitigatingcatastrophicrisks,andingeneralterms,riskaversionstrengthensthecaseforriskmitigation(Mogensen,MacAskillandGreavesMS).Withonlyminormodifications,similarremarksapplyif,insteadofreplacing
18Onthestandardaccount,toberiskaverseistohaveutilitybeaconcavefunctionoftotalwelfare(Pratt1964:127;O’DonoghueandSomerville2018:93).Somehavearguedthatthestandardaccountisinadequate(Rabin2000;Buchak2013:30).Onrisk-weightedexpectedutilitytheory,riskaversionisrepresentedbyariskfunctionthattransformstheexpectedutilityfunction(Quiggin1982;QuigginandWakker1994;Buchak2013).The differences between these accounts are unimportantfor our purposes.17Relatedly,itseemsthatinsofarasscaledoesmakeadifference,ASL(i)and(ii)aremorelikelytobetrueofdecision situations involving smaller sums of money,not less likely.Increasing-returns phenomena are discussed by Pierson(2000).17
riskneutralitywithriskaversion,wereplaceappealstoutilitarianisminourargumentwith(ex post) prioritarianism.Second,iftheonlymeansofpositivelyinfluencingthefarfuturewereviareducingtheriskofextinction,thecaseforstronglongtermismmightrelyoncontroversialviewsinpopulationethics,suchastotalism,onwhichtheabsenceofalargenumberofhappyfuturebeingsmakesthingsmuchworse.Butmanyaxiologieswillnotagreethatprematureextinctionisextremelybad.Inparticular,person-affectingapproachestopopulationethicstendtoresistthatclaim.Accordingtothespiritofaperson-affectingapproach,prematureextinctionisinitselfatworstneutral:ifhumanitygoesprematurelyextinct,thentheredoesnotexistanypersonwhoisworseoffasaresultofthatextinction,and,accordingtoaperson-affectingprinciple,itfollowsthattheresultingstateofaffairsisnotworse.Thefar-futurebenefitsofextinctionriskmitigationmaythereforebeatthebestnear-futurebenefitsonlyconditionaloncontroversial population axiologies.19However,risksfromASIareunlikeextinctioninthisrespect:therewillbealargepopulationinthefutureeitherway,andwearesimplyaffectinghowgoodorbadthosefuturelivesare.Theideathatit’sgoodtoimproveexpectedfuturewell-beingconditionalontheexistenceofalargeandroughlyfixed-sizefuturepopulationisrobusttoplausiblevariationsinpopulation-ethical assumptions.20Third,theexampleofASIriskalsoensuresthatourargumentgoesthroughevenif,inexpectation,thecontinuationofcivilisationintothefuturewouldbebad(AlthausandGloor2018;ArrheniusandBykvist1995:ch.3;Benatar2006).Ifthisweretrue,thenreducingtheriskofhumanextinctionwouldnolongerbeagoodthing,inexpectation.ButintheAIlock-inscenarioswehaveconsidered,therewillbealong-lastingcivilisationeitherway.ByworkingonAIsafetyandpolicy,weaimtomakethetrajectoryofthatcivilisationbetter,whether or not it starts out already ‘better thannothing’.Onefeatureofexpectedutilitarianismthatisnear-essentialtoourargumentisazerorateofpuretimepreference.Withevenamodestpositiverateofpuretimepreference(ase.g.on“discountedutilitarian”axiologies),theargumentwouldnotgothrough.Ourassumptionofa
20“Narrow”person-affectingapproachesdisagree,sincetheyregardtwostatesofaffairsasincomparablewheneverthosestatesofaffairshavenon-identicalpopulations(Heyd1988).However,suchapproachesareimplausible,forpreciselythisreason.Similarly,theoriesonwhichanytwostatesofaffairswithnon-equinumerouspopulationsareincomparable(BaderMS)areimplausible.Whencomparingdifferentsizedpopulations,a“wide”person-affectingapproachwilltypicallymapthesmallerpopulationtoasubsetofthelargerpopulation,andcomparewell-beingperson-by-personaccordingtothatmapping(Meacham2012);thesetheories will tend to agree with total utilitarianismon the evaluation of the AI catastrophes we discuss.Forsimilarreasons,wealsodonotconsiderheretheincomparabilitythatisintroducedbya“criticalrange”view (Blackorby, Bossert and Donaldson 1996).19Itisnotimmediatelyclearpreciselywhataperson-affectingapproachwillsayaboutthevalueofextinctionriskmitigation,sincetheusualformulationsofthosetheoriesdonotspecifyhowthetheoriesdealwithrisk,anditisnotimmediatelyclearhowtoextendthemtocasesthatdoinvolverisk.Thomas(2019)exploresanumberof possibilities.
18
zerorate,however,matchesaconsensusthatisalmostuniversalamongmoralphilosophers,and also reasonably widespread among economists.21Thisisofcoursenowherenearanexhaustivelistofpossibledeviationsfromexpectedtotalutilitarianism.Weconsidersomeotherdeviationsbelow,inthecourseofdiscussingcluelessnessandfanaticism.Ourconclusionisthatthecaseforstronglongtermismisatleastfairlyrobusttovariationsinplausibleaxiologicalassumptions;weleavetheinvestigationofother possible variations for future research.7. CluelessnessSection4focussedonworriesaboutourabilitiestoaffectthefarfuture.Adistinctfamilyofworriesismoredirectlyepistemic,andinvolvestheideathatwearecluelessbothaboutwhatthefarfuturewillbelike,andaboutthedifferencesthatwemightbeabletomaketothatfuture.Perhapsthebeingsthatarearoundwillbeveryunlikehumans.Perhapstheir22societies,iftheyhaveanythingthatcanbecalledasocietyatall,willbeorganizedinenormouslydifferentways.Fortheseandotherreasons,perhapsthekindsofthingsthatareconducivetothewell-beingoffar-futurecreaturesareverydifferentfromthekindsofthingsthatareconducivetoourwell-being.Givenallofthis,canwereallyhaveanyclueaboutthefar-future value of our actionseven in expectation?Wetakeitforgrantedthatwecannotknowwhatthefarfuturewillbelike.But,sincetheargumentofsections2-6hasalreadybeenconductedintermsofexpectedvalue,lackofknowledgecannotgroundanyobjectiontotheargument.Theobjectionmustinsteadbesomething else.Infact,thereareseveralquitedistinctpossibilitiesinthevicinityofthe“cluelessness”worry.Inthepresentsection,weaddressfiveoftheseobjections,relatingtosimplecluelessness,conscious unawareness, imprecision, arbitrariness,and ambiguity aversion.
22Since“washing-out”concernswhetherweareabletoaffectthefarfutureinexpectation,thistoohasanepistemicaspect,sothatthedistinctionbetweentheconcernsofsection4andthosediscussedhereisnotcompletelyclear(Tarsney2019).Nonetheless,theissuesraisedseemsufficientlydifferenttowarrantaseparatetreatment.21Azerorateofpuretimepreferenceisendorsedby,interalia,Sidgwick(1890),Ramsey(1928),Pigou(1932),Harrod(1948),Solow(1974),Cline(1992),Cowen(1992),Stern(2007),Broome(2008),Dasgupta(2008),Dietz,Hepburn,andStern(2008),BuchholzandSchumacher(2010),andGollier(2013).Inarecentsurveyofacademiceconomistswithexpertiseonthetopicofsocialdiscounting,38%ofrespondentsagreedwiththis“Ramsey-Stern view” (Drupp et al. 2018:119). Greaves(2017) provides a survey of the arguments on bothsides.Evenamongphilosophers,theconsensusagainstdiscountingfuturewell-beingisnotuniversal.Inparticular,someplausiblemodelsofpartialitysuggestassigninggreatereffectivemoralweighttoone’sowncontemporariesthantofar-futurepeople(Setiya2014;Mogensen2019).However,eventhesemodelsseemunlikelytorecommendsufficientdiscountingtounderminetheargumentforlongtermism(Mogensen2019:sec.6).19
7.1 Simple cluelessnessOurconcerniswithrelativelyweightydecisions,suchashowtodirectsignificantphilanthropicfunding.Butitisilluminatingtocomparethesetofarmoretrivialdecisionsituations, such as a choice of whether or where togo shopping on a given day.Eveninthelattercases,manyhaveargued,wecanbeallbutcertainthatourchoicewillhavehighlysignificantconsequencesexpost–farmoresignificantthanthemorepredictablenearer-termeffects.Thereasonsforthisincludethetendencyforeventrivialactionsto23affecttheidentitiesoffuturepersonsfarintothefuture.However,whencomparingquitetrivialalternatives,wecanhavenoideawhichofthetwowillturnouttobesuperiorvis-à-visthese deeply unpredictable very far future effects.Somehavearguedthatthesefactsundermineanyattempttobasedecisionsonconsiderationsoftheoverallgoodevenintrivialeverydaydecisioncontexts(e.g.Lenman2000).WeagreewithGreaves(2016)thatthisconcernisoverblown:inthecontextofrelativelytrivialeverydaydecisions,atleast,thedeeplyunpredictablefar-futureeffectsplausiblycanceloutforthepurposeofcomparingactionsinexpectedvalueterms.Consequently,thereisnoobjectionheretobasingthesedecisionsonanexpected-valueassessmentofnearer-future,more foreseeable effects.Aswehavearguedinsection4,however,decisionsabouthowtospendphilanthropicfundingaredisanalogousinthisrespect.WearenotdiscussingthepossibilitythateitherfundingAIsafetyresearchornotfundingitmightlead,aschancehasit,tothebirthofanadditionalunusuallygoodorbadpersonseveralcenturies’hence.Rather,wearediscussingthepossibilitythatfundingAIsafetymighthaveitsintendedeffectofmakingAIsafer.Whiletherearecertainlysevereuncertaintiesinsuchwork,itwouldbeoverlypessimistictoinsistthatsuccessisnomorelikelythancounterproductivity.Considerationsofsuch‘simple’cluelessness therefore do nothing to undermine theargument for strong longtermism.7.2 Conscious unawarenessTheexpectedvalueapproachweassumedinsection3isintendedasasubjectivedecisiontheory:thatis,itutilizesonlymaterialthatisaccessibletothedecision-makeratthetimeofdecision.Inparticular,therefore,thereisanimplicitassumptionthattheagentherselfisinapositiontograspthestates,actsandconsequencesthatareinvolvedinmodellingherdecision.Butperhapsthisisnottrue.Consider,forexample,would-belongtermistsintheMiddleAges.Itisplausiblethattheconsiderationsmostrelevanttotheirdecision–suchasthebenefitsofscience,andthereforetheenormousvalueofeffortstohelpmakethescientificandindustrialrevolutionshappensooner–wouldnothavebeenontheirradar.Rather,they23See e.g. Lenman (2000), Greaves (2016). We agreewith this claim, but our argument does not rely onit.20
mightinsteadhavebackedattemptstospreadChristianity,perhapsbyviolence:aputativeroute to value that, by our more enlightened lightstoday, looks wildly off the mark.Thesuggestion,then,isthatourcurrentpredicamentisrelevantlysimilartothatofourmedievalwould-belongtermists.Perhapsthereareactionsavailabletousthatwould,ifwewereabletothinkitallthroughinfulldetail,thendeliverhighexpectedbenefitsforthefarfuture.Butweknow,ifonlybyinductionfromhistory,thatwehavenotthoughtthingsthroughinallrelevantdetail.Perhapswetherebyhavegoodreasontorejectsubjectiveexpected-valueanalysis,andusesomequitedifferentformofdecisionanalysistoassessfar-future effects – in which case, all bets are asyet off regarding what the conclusion will be.Thisistheissueofconsciousunawareness–knowingthatoneisunawareofmanyrelevantconsiderations,mereawarenessofwhichwouldinfluenceone’sdecision-making.Followingmuchoftherecentliteratureonthistopic,however,ourviewisthatconsciousunawarenessdoesnotoccasionanyparticularlysignificantrevisionoftheBayesianframework,forthreereasons.First,weknowthatweoperatewithcoarse-grainedmodels,andthatthereasonsforthisincludeunawarenessofsomefine-grainings.Ofcourse,failuretoconsiderkeyfine-grainingsmightleadtodifferentexpectedvaluesandhencetodifferentdecisions,butthisseemspreciselyanalogoustothefactthatfailuretopossessmoreinformationaboutwhichstateinfactobtainssimilarlyaffectsexpectedvalues(andhencedecisions).Sinceourquestioniswhich actions areex anterational, both kinds offailure are beside the point.Second,weknowwearelikelytobeomittingsomeimportantpossiblestatesofnaturefromourmodelaltogether.Butconsciousnessofthiscanbemodelledbyinclusionofa“catchall”state:“alltheotherpossibilitiesIhaven’tthoughtof”.Again,conceptualisingpartsofthisstateinmoreexplicittermsmightchangesomeexpectedvalueassessments,butagainthisdoesnothingtounderminetheexanterationalityofdecisionstakenonthebasisofone’sexisting assessments.24Third,whilethebestoptionsmightwellbeonesthathavenotoccurredtous,thatdoesnothingtoimpugntherationalityofassessmentsofthosepossibleoptionsthathaveoccurredtous.Andourargumentforstronglongtermism,recall,requiresonlyalowerboundonattainable far-future expected benefits.Wedonotclaim(nordowebelieve)thatissuesofconsciousunawarenesshavenoeffectonwhatthereasonablecredencesandvaluesinagivendecisionsituationare.Thepointisratherthattheseissuesneednotoccasionanydeepstructuralchangetotheanalysis.Ourfurtherclaimisthatthenumberswehavesuggestedinsection4arereasonableaftertakingissuesofconscious unawareness into account.24Thefirsttypeofunawarenessisunawarenessofpossiblerefinements,thesecondisunawarenessofpossibleexpansions(Bradley 2017: sec. 12.3; Stefánsson andSteele forthcoming: sec. 3.2).21
7.3 ArbitrarinessAnobviousandpotentiallytroublingfeatureofourdiscussioninsection4isthepaucityofobjectiveguidanceforthekeyvaluesandprobabilities.Thisseemstocontraststarklywith,forinstance,theusualimpactevaluationsfortheshort-termbenefitsofbednetdistribution,which can be guided by relatively hard evidence (GiveWell2020b).Thisgivesrisetothreedistinct,thoughrelated,concernswiththestandardBayesianapproachthatwehaveused.Thefirstissimplythattheprobabilitiesand/orvaluesinthiscasearetooarbitraryforourargumenttocarryanyweight.Thesecondisthatincaseswhereanypreciseassignmentswouldbethisarbitrary,itisinappropriatetohaveprecisecredencesandvaluesatall.Thethirdisthatinsuchcases,theappropriatedecisiontheoryisambiguityaverse,andthatthismightunderminetheargumentforstronglongtermism.Weaddresstheseconcernsinturn.The“arbitrariness”objectionisthatevenifarationalagentmusthavesomeprecisecredenceandvaluefunctions,thereissolittlebywayofrationalrestrictiononwhichprecisefunctionsarepermissiblethattheargumentforstronglongtermismislittlemorethananassertionthatthe authors’ own subjective probabilities are onesrelative to which this thesis is true.Wehavesomesympathywiththisobjection.However,thereisadistinctionbetweentherebeingnowatertightargumentagainstsomecredencefunctionontheonehand,andthatcredencefunctionbeingreasonableontheother.Eveninthepresentstateofinformation,inourviewcredence-valuepairssuchthattheargumentforstronglongtermismfailsareunreasonable.If,forinstance,onehadcredencessuchthattheexpectednumberoffuturepeoplewasonly1014,thestatusquoprobabilityofcatastrophefromAIwasonly0.001%,andtheproportionbywhich$1billionofcarefulspendingwouldreducethisriskwasalsoonly0.001%,thenonewouldjudgespendingonAIsafetyequivalenttosavingonly0.001livesper$100–lessthanthenear-futurebenefitsofbednets.Butthisconstellationofconditionsseems unreasonable.However,wenotethatthisissueiscontentious.Weregardthequantitativeassessmentofthecrucial far-future-related variables as a particularlyimportant topic for further research.7.4 ImprecisionImpreciseapproachesrepresentanagentbyaclassofpairsofprobabilityandvaluefunctions–arepresentor–ratherthanasinglesuchpair.Thenaturalinterpretationisthatthesecorrespondtoincompleteorderingsofoptions:oneoptionisbetterthananother,forinstance,ifandonlyifthefirsthashigherexpectedvaluethanthesecondonallprobability-valuepairsin the representor.2525Bewley(2002),Dubra,Maccheroni,andOk(2004),andGalaabaatarandKarni(2013)providerepresentationtheorems linking such representations to incompleteorderings.22
ASLinvolvescomparingexantefar-futurebenefitswithexantenear-futureortotalbenefits.Ifimprecisionisafeatureofrationalevaluationatall,itisplausiblyaparticularlyprominentfeatureofevaluationoffar-futureconsequences.Soperhaps,foranyoption(includingtheoneswehavediscussedabove),anyreasonablerepresentorcontainsatleastsomeelementsaccordingtowhichthefar-futurebenefitsofthisoptionarenohigherthanthenear-futurebenefits of bednet mitigation?ItissomewhatcomplextosayhowoneshouldevaluateASLinthecontextofsuchimprecision.(Forinstance:ShouldwesimplyevaluateASLitselfrelativetoeachelementoftherepresentorinturn,andsupervaluatetoarriveatanoverallverdict?Orshouldweseektodefinesubsententialtermslike“near-best”inthecontextofrepresentors?Ifthelatter,howexactly?)Thegeneralidea,though,isthatonewayoranother,ifthepossibilityinthelastsentence of the preceding paragraph is realised, thenASL is at least not determinately true.Ourreplytotheimprecisioncritiqueisverysimilartoourreplytothearbitrarinesscritique.Whilewedonottakeastandonwhetherornotanyimprecisionofvaluationiseitherrationallypermissibleorrationallyrequired(Elga2010),wedon’tourselvesthinkthatanyplausibledegreeofimprecisioninthecaseathandwillunderminetheargumentforstronglongtermism.Forexample,wedon’tthinkanyreasonablerepresentorevencontainsaprobabilityfunctionaccordingtowhicheffortstomitigateAIrisksaveonly0.001livesper$100inexpectation.Thisdoesseemlessclear,however,thantheclaimthatthisisnotareasonable precise credence function.7.5 Ambiguity aversionInemployingthestandardBayesianmachinery,wehavebeenassumingambiguityneutrality.Incontrast,anambiguity-aversedecisiontheoryfavoursgamblesthatinvolvemoreratherthanlessobjectivelyspecifiedprobabilities,otherthingsbeingequal(MachinaandSiniscalchi 2014).Empirically,peoplecommonlydemonstrateambiguityaversion.Suppose,forexample,thatoneurncontains50redballsand50blackballs,andasecondurncontainsbothredandblackballsinunknownproportion(Ellsberg1961).Ifoneisambiguityaverse,onemightstrictlyprefertobetontheriskyurn,whereoneknowstheprobabilityofwinning,regardlessofwhichcolouroneisbettingon.Thispreferenceseemsinconsistentwithexpectedutilitytheory, but is widespread (Trautmann and Kuilen 2015).Itmightseematfirstsightthatambiguityaversionwouldunderminethecaseforstronglongtermism.Incontemplatingoptionslikethosediscussedinsection4,oneneedstosettleone’scredencethatsomegiveninterventiontoreduceextinctionrisk,ortoincreasethesafetyofASI,wouldleadtoalargepositivepayoffinthefarfuture.Butagain,thereseemssignificantarbitrarinesshere.Incontrast,impactevaluationsforthenear-futurebenefitsofbednetdistributionseemtoinvolvemuchmorepreciselyboundedprobabilities.Mightan23
ambiguity-aversedecisiontheory,then,takeasubstantiallydimmerviewofthefar-futurebenefits of existential risk mitigation, and henceof strong longtermism?Our answer is ‘no’, for two reasons.First,whetherornotambiguityaversionhasanyprospectofunderminingtheargumentforstronglongtermismdepends,inthefirstinstance,onwhethertheagentinquestionisambiguityaversewithrespecttothestateoftheworld,orinsteadwithrespecttothedifferenceonemakesoneselftothatstate.Theaboveargument-sketchimplicitlyassumedthelatter.But,ifoneisgoingtobeambiguityaverseatall,itseemsmoreappropriateforan26altruisttobeambiguityaverseintheformersense(MacAskill,Mogensen,GreavesandThomasMS).Anditisfarfromclearthatactionsseekingtoimprovethefarfutureincreaseambiguitywithrespecttothestateoftheworld.Itisalreadyextremelyambiguous,forinstance,howmuchnear-termextinctionriskhumanityfaces.Weseenoreasontothinkthat27thislatterambiguityisincreased,ratherthandecreasingorremainingthesame,by,forexample, funding pandemic preparedness.28Second,althoughitispsychologicallynatural,andcorrespondinglywidespread,ambiguityaversionisanywayirrational.Hereweagreewithafairlywidespreadconsensus;wehavenothing to add to the existing debate on this question.29Weconcludethatthepossibilityofambiguityaversiondoesnotunderminetheargumentforstrong longtermism.8. FanaticismOneobviouspointofcontrastbetweentheparadigmexamplesofwaystoattainhighnear-futurevs.far-futureexpectedbenefitsisthattheformertendtoinvolvehighprobabilitiesofrelativelymodestbenefits,whereasthelattertendtoinvolvetinyprobabilitiesofenormousbenefits.Indiscussingactionsaimedatmitigatingextinctionrisk,forinstance,weconcededthatitisveryunlikelythatanysuchactionmakesanysignificantdifference;theargumentforprioritizingsuchactionsnonethelessischaracteristicallythatiftheydomakeasignificant difference, they might make a truly enormousone.Evenamongthosewhoaresympatheticingeneraltoexpectedutilitytheory,manybalkatitsapparentimplicationsforcasesofthislattertype.Supposeyouarechoosingbetweena“safe
29Seee.g.(Al-Najjar&Weinstein2009)forasurveyofargumentsthatambiguityaversionisirrational.Rowe&Voorhoeve (2018) and Stefánsson & Bradley (2019) defendits rationality.28WeinvestigatetheissuesoutlinedinthisparagraphinmoredepthinMogensen,MacAskillandGreaves(MS).27Beardetal.(2020,AppendixA)andSandbergandBostrom(2008)bothpresentawiderangeofestimatesfrom around 1% to 50%, from (respectively) a literaturereview and a conference participant survey.26ToseethedistinctioninEllsberg’stwo-urnssetting,supposethatinthestatusquo,oneissettoreceive$100ifftheambiguousurndeliversaredball.Supposeone’schoiceisbetweenwhethertoaddtothatbackgroundgambleabetonablackballbeingdrawnfromtheriskyurn,orinsteadfromtheambiguousurn.Prettyclearly,ambiguityaversioninthestandardsensewillrecommendthelatter(sinceonethenfaceszeroambiguityoverall), notwithstanding the fact that the benefitdelivered by one’s actionis more ambiguous in thiscase.24
option”ofsavingathousandlivesforsureanda“riskyoption”thatgivesaoneinatrillionchanceofsavingaquintillionlives.Theexpectednumberoflivessavedisathousandtimesgreaterfortheriskyoption.Unlesstheutilityfunctionisverynon-linearasafunctionoflivessaved,correspondingly,theexpectedutilityofthelatteroptionisalsolikelytobegreater.Yet,ifyouchoosetheriskygamble,itisoverwhelminglylikelythatathousandpeoplewilldie, for no gain.30Intuitively,itseemsatleastpermissibletosavethethousandinthiscase.Ifso,thismightsuggestthatwhileexpectedutilitytheoryisagoodapproachtochoiceunderuncertaintyinmoreordinarycases,itfailsincasesinvolvingextremelylowprobabilitiesofextremelylargevalues.Onemight,then,seeka“non-fanatical”decisiontheory–onethatdoesnotrequiretheagenttosacrificearbitrarilymuch,withprobabilityarbitrarilyclosetoone,in“fanatical”pursuitofanextremelyunlikelybutenormouslylargerpayoff.Mightanon-fanaticaldecisiontheory undermine the case for strong longtermism?Weregardthisasoneofthemostplausiblewaysinwhichtheargumentforstronglongtermismmightfail.Ourviewisthatatpresent,thequestioncannotbeconfidentlysettled,sinceresearchintothepossibilityofanon-fanaticaldecisiontheoryiscurrentlyembryonic.However,initialresultssuggestthatavoidingfanaticismmightcomeattoohighaprice.BecksteadandThomas(2020),forinstance,considerasequenceofgambles.Thefirstgambledeliversalargebutrelativelymodestbenefitwithcertainty.Thelastgambledeliversanenormouslylargebenefitwithextremelysmallprobability,andzerobenefitotherwise.Thesetwogamblesarelinkedbyasequenceinwhicheachgambleoffersonlyaveryslightlylowerprobabilityofwinningthanthepreviousgamble,andinvolvesamuchbetterbenefitifonedoeswin.Thissequence-schemaillustratesthatanytransitivetheorythatisnotfanaticalmustinsteadbeworryingly“timid”:inatleastonepairwisecomparisonofadjacentgambles,evenanarbitrarilylargeincreaseinthevalueofapositivepayofffailstocompensateforanyarbitrarilysmalldecreaseinitsprobability.AsBecksteadandThomasgoontoshow,suchtimidityinturnleadstoimplausiblyextremeformsofriskaversioninsomecases,andtoparticularlyimplausibleformsofdependenceofoption-assessmentsonassessmentsofcausally isolated aspects of the state of affairs.Acomplementaryreplyisthatinanycase,theprobabilitiesinvolvedintheargumentforlongtermismmightnotbesufficientlyextremeforanyplausibledegreeofresistanceto“fanaticism”tooverturntheverdictsofanexpectedvalueapproach,atleastatthesocietallevel.Forexample,itwouldnotseem“fanatical”totakeactiontoreduceaone-in-a-millionriskofdying,asoneincursfromcycling35milesordriving500miles(respectively,bywearingahelmetorwearingaseatbelt(DepartmentofTransport2020)).Butitseemsthatsocietycanpositivelyaffecttheverylong-termfuturewithprobabilitieswellabovethis30A similar example is that of Pascal’s Mugging (Bostrom2009).25
threshold.Forinstance,insection4.3,wesuggestedalowerboundofonein100,000onaplausiblecredencethat$1billionofcarefullytargetedspendingwouldavertanexistentialcatastrophe from artificial intelligence.Thingsarelessclearontheindividuallevel.If,forexample,$10billioncanreducetheriskofextinction(oracomparablybadoutcome)byonein100,000,andanindividualphilanthropistmakesa$10,000contributionwitheffectsproportionaltothat,thenthephilanthropistwouldreduceextinctionriskbyoneintenbillion.Atthislevel,weareunlikelytofindcommonplacedecisionsrelyingonthatprobabilitythatwewouldregardasnon-fanatical.31So,ifoneisinclinedtotakeseriouslythefanaticismworry,despitetheproblemswith‘timidity’,itmaybethattheprobabilitiesinquestionareproblematicallysmallontheindividual level, but not at the social level.Ourinclinationistothinkthatourintuitionsonthesocietallevelarecorrect,andthatourintuitionsaroundhowtohandleverylowprobabilitiesareunreliable.Thelatterhassomesupportfromthepsychologicalliterature(KahnemanandTversky1979:282-83;Erevetal.2008).Wethereforetentativelyconcludethatconsiderationsoffanaticismdonotunderminetheargument for strong longtermism.9. Deontic strong longtermismInsection2,wedistinguishedbetweenaxiologicalanddeonticversionsofstronglongtermism.Sofar,ourdiscussionhasfocusedexclusivelyonthecasefortheaxiologicalclaim.The deontic analog to ASL isDeonticstronglongtermism(DSL):Inthemostimportantdecisionsituationsfacingagentstoday,(i) One ought to choose an option that is near-bestfor the far future.(ii)Oneoughttochooseanoptionthatdeliversmuchlargerbenefitsinthefarfuturethan in the near future.JustasASLconcernsexanteaxiology,the‘ought’inDSListhesubjectiveought:theonethatismostrelevantforaction-guidance,andisrelativetothecredencesthatthedecision-maker ought to have.32
32Itiswidelyagreedthateitheritisusefultodistinguishbetweenobjectiveandsubjectivesensesof‘ought’(Ewing1948:118-22;Brandt1959:360-7;Russell1966;Parfit1984:25;Portmore2011;Dorsey2012;Olsen2017;Gibbard2005;Parfit2011),or‘ought’isunivocalandsubjective(Prichard1932;Ross1939:139;Howard-Snyder2005;Zimmerman2006;Zimmerman2008;Mason2013).Ourdiscussionpresupposesthatone31Oneexceptionmightbeputtingonaseatbeltforaone-miledrive.Ifdoingsodecreasesone’schanceofafatalaccidentbyafactorofone-third,thentheseatbeltreducesone’sriskofdeathbyaboutoneinabillion.Butperhaps this is not our reason for wearing seatbeltsfor short journeys.26
Withoutassumingconsequentialism,DSLdoesnotimmediatelyfollowfromASL.Webelieve,however,thatourargumentforASLnaturallygroundsacorrespondingargumentforDSL. This is because of the followingstakes-sensitivityargument:(P1)Whentheaxiologicalstakesareveryhigh,therearenoseriousside-constraints,andthepersonalprerogativesarecomparativelyminor,oneoughttochooseanear-best option.(P2)Inthemostimportantdecisionsituationsfacingagentstoday,theaxiologicalstakesareveryhigh,therearenoseriousside-constraints,andthepersonalprerogatives are comparatively minor.(C)So,inthemostimportantdecisionsituationsfacingagentstoday,oneoughttochoose a near-best option.DSL follows from the conjunction of (C) and ASL.The stakes-sensitivity argument is obviously valid.Are its premises true?(P1)appealstoonlyaverymoderateformofstakes-sensitivenon-consequentialism.Itallowsthattheremaybesomeactionsthatarealwayspermissibleorprohibited,nomatterhowgreattheaxiologicalstakes:forexample,perhapsoneisalwayspermittedtosavethelifeofone’schild;orperhapsoneisalwaysprohibitedfromtorturinganotherperson.Anditonlyentailsthat comparatively minor prerogatives are overriddenwhen the stakes are very high.33Itishighlyplausiblethatthereshouldbeatleastthismuchstakes-sensitivity.Thelackofstakes-sensitivityisacommonobjectiontoKant'snotoriousviewthatevenifafriend'slifedependsonit,oneshouldnottellalie(Kant1996).Turningtoprerogatives,in“emergencysituations”likewartime,ordinaryprerogatives—forinstance,toconsumeluxuries,tolivewithone’sfamily,andeventoavoidsignificantriskstoone’slife—arequiteplausiblyoverridden.Nagel(1978)observesthatpublicmoralitytendstobemoreconsequentialistincharacterthanprivatemorality;onenaturalpartialexplanationforthis(thoughnottheoneemphasisedbyNagelhimself)isthatinpubliccontextssuchasgovernmentalpolicydecisions, the axiological stakes tend to be higher.Weforeseetwolinesofresistanceto(P1).First,onecouldrejecttheideaof“thegood”altogether(Thomson2008:sec.1.4).Onthisview,thereissimplynosuchthingasaxiology.It’sclearthatourargumentasstatedwouldnotberelevanttothosewhoholdsuchviews.Butsuchaviewmuststillbeabletoexplainthefactthat,incaseswherethereisahugeamountatstake,comparativelyminorconstraintsandprerogativesgetoverridden.Itseemslikelythat
33(P1) is very similar to Singer’s claim that “If itis in our power to prevent something very bad fromhappening,without thereby sacrificing anything morally significant,we ought, morally, to do it”(Singer 1972:231).ofthesedisjunctsiscorrect.Aminorityofauthorsholdsthat‘ought’isunivocalandobjective(Moore1903:199-200,229-30;Ross1930:32;Thomson1986:177-79;Graham2010;Bykvist2011).Similarly(butlessdiscussed),onemightbeskepticalofthenotionofexanteaxiology;again,ourdiscussionofASLhaspresupposed that any such skepticism is misguided.27
anysuchexplanationwillresultinsimilarconclusionstothosewehavedrawn,viasimilararguments.Secondly,andmoreplausibly,perhapsonlysomesortsofaxiologicalconsiderationsarerelevanttodeterminingwhatweoughttodo.Weconsidertwowaysinwhichthisideamightundermine our argument.First,onanon-aggregationistview,comparativelysmallexantebenefitstoindividualsarenotrelevanttodeterminingwhatoneoughttodo,evenifthebenefitsapplytoanenormousnumber of people (Scanlon 1998:235 ; Frick 2015 ; Voorhoeve 2014).Second,perhapsaxiologicalconsiderationscannotoutweighnon-consequentialistconsiderationswhentheaxiologicalconsiderationsinvolvealteringtheidentitiesofwhocomes into existence (Parfit 1984: ch. 16).However,bothlinesofthoughtriskprovingtoomuch.Let’sfirstconsiderthenon-aggregationistresponse.ConsideraBriton,duringWWII,decidingwhethertofight;orsomeonedebatingwhethertovoteintheircountry’sgeneralelection;orsomeonedecidingwhethertojoinanimportantpoliticalprotest;orsomeonedecidingwhethertoreducetheircarbonfootprint.Ineachcase,theexantebenefitstoanyparticularotherpersonaretiny.Butinatleastsomesuchcases,it’sclearthattheagentisrequiredtoundertaketherelevantaction,andthemostnaturalexplanationofwhyisbecausetheaxiologicalstakesaresohigh.34Second,considerthenon-identityresponse.It’sclearthatgovernmentsoughttotakesignificantactiontofightclimatechange.Butalmostalloftheexpecteddamagesfromclimatechangecomefromitsimpactsonthosewhoareyettobeborn.What’smore,any35policydesignedtomitigateclimatechangewillalsoaffecttheidentitiesofthoseunbornpeople.Endorsingthenon-identityresponsewouldthereforeriskrejectingtheideathatwelfaristconsiderationsgenerateanyobligationsforsocietytodaytofightclimatechange,evenwhileacceptingthatclimatechangewillsignificantlyandavoidablyreducewelfareinexpectation for centuries to come. That position isclearly incorrect.Turningnowto(P2):The‘high-stakes’aspectofthispremiseisjustifiedinpartonthebasisoftheargumentsofsections3-4.Atleastonourmainandlowestimatesoftheexpectedsizeofthefuture,inthedecisionsituationswe’vediscussed,notonlyarethebestoptionsthosethathavethenear-bestfar-futureconsequences,buttheyaremuchbetterthanthoseoptionswhose far-future consequences are nowhere near best.Atthesametime,atleastformostmembersofrichcountries,thedecisionsituationswe’vediscussedarethosewherethepersonalprerogativesarearguablycomparativelyminor,and
35Forexample,theSternReviewpredictsthevastmajorityofdamagestooccurafter2100inboth“baseline”and “high climate” scenarios (Stern 2007:178, fig.6.5d).34Noneoftheseexamples,however,involvesforegoinganopportunitytosavemanylivesofidentifiedpeople.Inthisrespect,ourexamplesareperhapsrelevantlydissimilartoadecisionbetweenspendingtobenefitthefarvs. the near future. We thank an anonymous refereefor pressing this reply.28
wheretherearenoserioussideconstraints.Thisisclearestinthecasesofindividualdecisionsaboutwheretodirectone’saltruisticspending(holdingfixedthetotalsizeofone’s“altruisticbudget”),andaboutcareerchoice.Thedecisiontogivetoorganisationsthatwillpositivelyinfluencethefarfutureratherthanorganisationsmoregearedtowardsimprovingthenearfuture,ortoworkinacareerthatisparticularlybeneficialforthelong-termfuture,mightwellinvolvesomesacrifices.Buttheyarenotclosetothesortsofsacrificeswhere36theremightbeabsoluteornear-absoluteprerogatives.Similarly,thesearenotcircumstanceswhereoneisrequiredtoviolateside-constraintsinordertoachievethenear-bestlong-termoutcome.Theslightlylessclearcasesarethoseinvolvingindividualdecisionsaboutthetotalsizeofone’s“altruisticbudget”(vs.“personalbudget”),andsocietaldecisionsabouthowmanyresourcestodevotetoimprovingtheprospectsforthefarfuture(vs.thenearfuture,includingthelifetimesofpresentpeople).Here,itremainstruethatnoseriousside-constraintsneedbeinvolved.Onemightworry,though,thathereourargumentwillbetoodemanding:mightitimplythatwe,individuallyorasasociety,oughttodevotemostofourresourcestoimprovingthefarfuture,atthelargeexpenseofourownprudentialinterests?Asinthediscussionofdemandingnessinthecontextofglobalpoverty,arangeofresponsestothisconcernispossible.Wehavenothingtoaddtotheexistingliteratureondemandingness(e.g.Kagan1984;Mulgan2001;Hooker2009).Wewillsimplynotethat37evenif,forexample,thereisanabsolutecaponthetotalsacrificethatcanbemorallyrequired,itseemsimplausiblethatsocietytodayiscurrentlyanywherenearthatcap.Thesameremarkappliestoatleastthevastmajorityofindividualsinrichcountries.Weoughttobe doing a lot more for the far future than we currentlyare.3810. Summary and conclusionsThepotentialfutureofcivilisationisvast.Onceweappreciatethis,itbecomesplausiblethatimpact on the far future is the most important featureof our actions today.Stronglongtermismwouldbefalseinaworldthathadsufficientlyweakcausalconnectionsbetweenthenearandthedistantfuture,suchthatitwastoodifficulttosignificantlyinfluence
38Mightourargumentsgofurtherthanthis,andjustifyatrocitiesinthenameofthelong-termgood?PerhapstheFrenchRevolutionhadgoodlong-termconsequences,intermsofbringingaboutamoreliberalanddemocraticworld: does strong longtermism, if so, justify theguillotine?Wedonotthinkso,foratleasttworeasons.Thefirstisthat,forsuchseriousside-constraints,somethingclosertoabsolutismornear-absolutismbecomesmuchmoreplausible(or,atleast,ittakesmorethanmereexantegoodnesstojustifyviolationofthoseside-constraints).Thesecondisthat,inalmostallcases,whenthereissomeoptionavailablethatpromotesthelong-termgoodwhileviolatingaseriousside-constraint,therewillbesomealternativeoptionavailablethatachievesasimilaramountoflong-termgoodwithoutviolatingthatside-constraint. Liberal democracy could have beenachieved in France without the Reign of Terror.37Mogensen (2020) discusses specifically the relationshipbetween demandingness and longtermism.36There are, however, reasons to think that these sacrificesare not as great as we might initially suppose(MacAskill, Mogensen and Ord 2018).
29
thecourseoftheverylong-runfuture.However,wehaveargued,theworldwefindourselvesin today does not have this feature.Wepresentedourcentralcaseintermsof(i)atotalutilitarianaxiologyand(ii)anexpectedvaluetreatmentofdecision-makingunderuncertainty.However,weargued,plausibledeviations from either or both of these assumptionsdo not undermine the core argument.Thispapermainlyfocussedonthedecisionsituationsofasocietyorindividualconsideringhowtospendmoneywithoutconstraintsastocausearea,andofanindividual’scareerchoice.Wearguedthatthesearesituationswherewecaninexpectationsignificantlyinfluencethefarfuture.Preciselybecauseofthis,theyareamongthemostimportantdecisionsituations we face, and axiological strong longtermismfollows.Inourownview,theweakestpointsinthecaseforaxiologicalstronglongtermismaretheassessmentofnumbersforthecost-effectivenessofparticularattemptstobenefitthefarfuture,theappropriatetreatmentofcluelessness,andthequestionofwhetheranexpectedvalueapproachtouncertaintyistoo“fanatical”inthiscontext.Theseissuesinparticularwould benefit from further research.Inadditiontoaxiologicalissues,wealsodiscussedthecounterpartdeonticissues.Wesuggestedthatdeonticstronglongtermismmightwellbetrueevenifconsequentialismisfalse,onthegroundsthat(i)thestakesinvolvedareveryhigh,(ii)aplausiblenon-consequentialisttheoryhastobesensitivetotheaxiologicalstakes,becomingmoreconsequentialistinoutputastheaxiologicalstakesgethigher,and(iii)inthekeydecisionsituations,anycountervailingconstraintsand/orprerogativesarecomparativelyminor.Quiteplausibly,intheworldasitistoday,themostimportantdeterminantsofwhatweoughttodoarise from our opportunities to affect the far future.Itispossible,butfarfromobvious,thatfar-futureimpactsarealsomoreimportantthannear-futureimpactsinamuchwiderclassofdecisionsituations:forinstance,decisionsaboutwhetherornottohaveachild,andgovernmentpolicydecisionswithinarelativelynarrow‘causearea’.Insofarastheyare,stronglongtermismcouldpotentiallysetamethodologyforfurtherworkinappliedethicsandappliedpoliticalphilosophy:foreachissueinthesesubfields,onecouldidentifythepotentialfar-futureeffectsfromdifferentactionsorpolicies,andthenworkthroughhowthesebearontheissueinquestion.Theanswersmightsometimesbe surprisingly revisionary.AppendixWe claimed in the main text that (BR) entails:(a)thatASL(i)holdsofarestrictionofsociety’sdecisionsituation,obtainedbyremovingany options involving net expected short-term harmfrom the choice set; and(b)that ASL(ii) holds of society’s decision situation.Here, we make these claims precise, and supply theproofs for them.30
Terminology and notationForanyoption,let,,respectivelydenote’snear-future,far-futureand𝑥𝑁(𝑥)𝐹(𝑥)𝑉(𝑥)𝑥overallbenefits.Let,,respectivelybethehighestavailablenear-future,far-futureand𝑁\*𝐹\*𝑉\*overallbenefits.Letbethehighestfar-futurebenefitthatisavailablewithoutnet𝐹'short-term harm.Weinterpretboth“near-bestoverall”and“near-bestforthefarfuture”intermsofproportionaldistancefromzerobenefittothemaximumavailablebenefit,and“muchlarger”intermsofamultiplicativefactor.Thereis,ofcourse,flexibilityontheprecisevaluesofthefactorsinvolved.Wethereforeconsiderthefollowingprecisificationsofourkeyclaims,carrying free parameters:BR(n):.𝐹'≥𝑛𝑁\*:Everyoptionthatdeliversoverallbenefitsofatleast𝐴𝑆𝐿𝑖(ϵ𝑂, ϵ𝐹)(1−ϵ𝑂)𝑉\*delivers far-future benefits of at least.(1−ϵ𝐹)𝐹\*:Everyoptionthatdeliversoverallbenefitsofatleastdelivers𝐴𝑆𝐿(ϵ𝑂,𝑟)(1−ϵ𝑂)𝑉\*far-future benefits that are at least r times itsown near-future benefits.Inwhatfollows,weproveclaims(a)and(b)forspecifiedrelationshipsbetweentheparameter values.Precisificationofclaim(a).Weclaim(moreprecisely)thatifBR(n)holdsofagivendecisionsituation,thenforany,holdsoftherestricteddecisionϵ𝑂∈[0,1]𝐴𝑆𝐿(ϵ𝑂,ϵ𝑂+1𝑛) situation(withanyoptionsinvolvingnetshort-termharmremoved).Forexample,if,𝑛=10theneveryoptionthatdeliversatleast90%ofavailableoverallexpectedbenefitsdeliversatleast80%ofavailablefar-futureexpectedbenefits,onceanyoptionsinvolvingnetshort-termharm are ruled out.Proof.SupposethatBR(n)holds.Sincefar-futurebenefitisattainablewithoutnear-future𝐹'netharm,theoverallbestoptionmustdelivertotalbenefitsofatleast;soanynear-best𝐹'optionmustdelivertotalbenefitsofatleast.ButbyBR(n),themaximum(1−ϵ𝑂)𝐹'attainablenear-futurebenefitisatmost.Therefore,anynear-bestoptionmustdeliver𝐹'𝑛far-futurebenefitsofatleast.Butinthisdecisionsituation,(1−ϵ𝑂−1𝑛)𝐹'𝐹'=𝐹\*(since near-future net harm is here ruled out).Precisificationofclaim(b).Weclaim(moreprecisely)thatifBR(n)holdsthenforany,ASL(ii)alsoholds.Forexample,if,theneveryϵ𝑂∈[0,1](ϵ𝑂,(1−ϵ𝑂)𝑛−1)𝑛=1031
optionthatdeliversatleast90%ofavailableoverallexpectedbenefitsdeliversatleast8times as much far-future as near-future expected benefit.Proof.Letbe any option that is near-best overall.Then𝑥𝑉(𝑥)≥(1 − ϵ𝑂)𝑉\*by definition of near-best≥(1 − ϵ𝑂)𝐹'since, by hypothesis,is achievable without short-term𝐹'harmBut, so it follows that𝑉(𝑥)=𝑁(𝑥)+𝐹(𝑥)𝐹(𝑥)≥(1 − ϵ𝑂)𝐹'−𝑁(𝑥)≥(1 − ϵ𝑂)𝐹'−𝑁\*≥((1−ϵ𝑂)𝑛−1)𝑁\*by BR(n)≥((1−ϵ𝑂)𝑛−1)𝑁(𝑥)ReferencesFredC.Adams(2008).Long-TermAstrophysicalProcesses.InNickBostromandMilanCirkovic (eds.)Global Catastrophic Risks.Oxford:Oxford University Press.FredC.Adams&G.Laughlin(1999).TheFiveAgesoftheUniverse:InsidethePhysicsofEternity. Free Press.NabilI.Al-Najjar&JonathanWeinstein(2009).TheAmbiguityAversionLiterature:ACritical Assessment.Economics & Philosophy,25(3):249–284.DavidAlthausandLukasGloor(2018).ReducingRisksofAstronomicalSuffering:ANeglectedPriority.FoundationalResearchInstitute.Availableathttps://foundational-research.org/reducing-risks-of-astronomical-suffering-a-neglected-priority/SamAltman(2015).Machineintelligence,part1.Availableathttps://blog.samaltman.com/machine-intelligence-part-1DarioAmodei,ChrisOlah,JacobSteinhardt,PaulChristiano,JohnSchulmanandDanMané(2016). Concrete Problems in AI Safety.https://arxiv.org/abs/1606.06565StuartArmstrongandAndersSandberg(2013).Eternityinsixhours:Intergalacticspreadingof intelligent life and sharpening the Fermi paradox.Acta Astronautica89:1-13.32
RenéArnold,J.ScottMarcus,GeorgiosPetropoulosandAnnaSchneider(2018).Isdatathenewoil?Diminishingreturnstoscale.29thEuropeanRegionalConferenceoftheInternational Telecommunications Society.GustafArrheniusandKristerBykvist(1995).FutureGenerationsandInterpersonalCompensations:MoralAspectsofEnergyUse.UppsalaPrintsandPreprintsinPhilosophy21, Uppsala Universitet.Ralf Bader (ms). Neutrality and Conditional Goodness.AnthonyBarnosky,NicholasMatzke,SusumuTomiya,GuinevereWogan,BrianSwartz.,TiagoQuental,CharlesMarshall,JennyMcGuire,EmilyLindsey,KaitlinMaguire,BenMersey&ElizabethFerrer.(2011).HastheEarth’ssixthmassextinctionalreadyarrived?Nature471(7336):51.TobiasBaumann(2018).Anintroductiontoworst-caseAIsafety.https://s-risks.org/an-introduction-to-worst-case-ai-safety/SimonBeard,ThomasRowe,andJamesFox(2020).Ananalysisandevaluationofmethodscurrently used to quantify the likelihood of existentialhazards.Futures115:102469.NickBeckstead(2013).OntheOverwhelmingImportanceofShapingtheFarFuture.PhDthesis,RutgersUniversity.Availableathttps://rucore.libraries.rutgers.edu/rutgers-lib/40469/PDF/1/play/NickBeckstead(2014).Willweeventuallybeabletocolonizeotherstars?Notesfromapreliminaryreview.https://www.fhi.ox.ac.uk/will-we-eventually-be-able-to-colonize-other-stars-notes-from-a-preliminary-review/NickBecksteadandTerujiThomas(2020).AParadoxforTinyProbabilitiesandEnormousValues.GPIWorkingPaper10-2020.https://globalprioritiesinstitute.org/nick-beckstead-and-teruji-thomas-a-paradox-for-tiny-probabilities-and-enormous-values/DavidBenatar(2006).BetterNevertoHaveBeen:TheHarmofComingintoExistence.Oxford: Clarendon Press.AlexanderBerger(2019).GiveWell’sTopCharitiesAre(Increasingly)HardtoBeat.OpenPhilanthropyProject.Availableathttps://www.openphilanthropy.org/blog/givewells-top-charities-are-increasingly-hard-beat. Accessed 26 January 2021.Truman F. Bewley (2002). Knightian decision theory.Part I.Decisions in Economics andFinance,25(2):79–110.CharlesBlackorby,WalterBossertandDavidDonaldson(1996).Quasi-orderingsandpopulation ethics.Social Choice and Welfare13:129–150.NicholasBloom,CharlesI.Jones,JohnVanReenen,andMichaelWebb(2020).AreIdeasGetting Harder to Find?American Economic Review110(4):1104-44.LewisBollard(2016).InitialGrantstoSupportCorporateCage-freeReforms.OpenPhilanthropyProject.Availableat33
https://www.openphilanthropy.org/blog/initial-grants-support-corporate-cage-free-reforms#Corporate\_cage-free\_campaigns\_are\_extremely\_cost-effective.Accessed26January 2021.NickBostrom(2003).AstronomicalWaste:TheOpportunityCostofDelayedTechnologicalDevelopment.Utilitas15(3):308–314.Nick Bostrom (2009). Pascal’s Mugging.Analysis69(3):443–445.NickBostrom(2013).ExistentialRiskPreventionasGlobalPriority.GlobalPolicy4(1):15–31.NickBostrom(2014).Superintelligence:Path,Dangers,Strategies.Oxford:OxfordUniversity Press.RichardBradley(2017).DecisionTheorywithaHumanFace.Cambridge:CambridgeUniversity Press.Richard Brandt (1959).Ethical Theory. Englewood Cliffs,N.J.: Prentice-Hall.DarrellBrickerandJohnIbbitson(2019).EmptyPlanet:TheShockofGlobalPopulationDecline.New York: Crown Publishing Group.John Broome (2008). The ethics of climate change.Scientific American298:96-102.StevenBryson,MichelleKunimoto,RaviKopparapu,...&KhadeejahZamudio,(2021).TheOccurrenceofRockyHabitable-zonePlanetsaroundSolar-likeStarsfromKeplerData.The Astronomical Journal, 161(1):36.Lara Buchak (2013).Risk and Rationality. Oxford:Oxford University Press.WolfgangBuchholzandJanSchumacher.2010.Discountingandwelfareanalysisovertime:Choosing the η.European Journal of Political Economy26(3):372–385.KristerBykvist(2011).HowtoDoWrongKnowinglyandGetAwaywithIt.InNeither/Nor.PhilosophicalPapersDedicatedtoErikCarlsonontheOccasionofHisFiftiethBirthday,31–47.UppsalaPhilosophicalStudies58.Uppsala:DepartmentofPhilosophy, Uppsala University.BryanCaplan(2008).Thetotalitarianthreat.InNickBostromandMilanCirkovic(eds.)Global Catastrophic Risks.Oxford: Oxford UniversityPress.KennethG.Cassman,AchimR.Dobermann,andDanielT.Walters(2002).Agroecosystems,Nitrogen-UseEfficiency,andNitrogenManagement.AMBIO:AJournaloftheHuman Environment31(2):132-140.DavidJ.Chalmers(1996).TheConsciousMind:InSearchofaFundamentalTheory.Oxford: Oxford University Press.PaulChristiano(2014).Wecanprobablyinfluencethefarfuture.RationalAltruist.https://rationalaltruist.com/2014/05/04/we-can-probably-influence-the-far-future/WilliamR.Cline(1992).TheEconomicsofGlobalWarming.Washington,DC:InstituteforInternational Economics.34
JoelE.Cohen(1995).HowManyPeopleCantheEarthSupport?NewYork;London:Norton.TylerCowen(1992).Consequentialismimpliesazerorateofintergenerationaldiscount.InP.Laslett&J.S.Fishkin(eds.),JusticeBetweenAgeGroupsandGenerations.YaleUniversity Press.ParthaDasgupta(2008).Discountingclimatechange.JournalofRiskandUncertainty37:141–169.DepartmentofTransport(2020).ReportedroadcasualtiesinGreatBritain:provisionalresults2019.Availableathttps://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment\_data/file/904698/rrcgb-provisional-results-2019.pdf.Accessed 28th May 2021.SimonDietz,CameronJ.HepburnandNicholasStern(2008).Economics,ethicsandclimatechange.InKaushikBasuandRaviKanbur(eds.),ArgumentsforaBetterWorld:EssaysinHonourofAmartyaSen.VolumeII:Society,InstitutionsandDevelopment,365–386. Oxford: Oxford University Press.DaleDorsey(2012).ObjectiveMorality,SubjectiveMoralityandtheExplanatoryQuestion.Journal of Ethics and Social Philosophy6(3):1–24.MoritzA.Drupp,MarkFreeman,BenGroomandFrikkNesje.(2018).DiscountingDisentangled.American Economic Journal: EconomicPolicy10:109-134.Juan Dubra, Fabio Maccheroni & Efe A. Ok (2004). Expectedutility theory without thecompleteness axiom.Journal of Economic Theory115(1):118–133.Adam Elga (2010). Subjective Probabilities Shouldbe Sharp.Philosopher’s Imprint10(5):1–11.DanielEllsberg(1961).Risk,Ambiguity,andtheSavageAxioms.TheQuarterlyJournalofEconomics75(4):643–669.IdoErev,IraGlozman,andRalphHertwig(2008).Whatimpactstheimpactofrareevents.Journal of Risk and Uncertainty36(2):153–177.A.C. Ewing (1948).The Definition of Good.London:Routledge & Kegan Paul.BentFlyvbjerg(2008).Curbingoptimismbiasandstrategicmisrepresentationinplanning:Reference class forecasting in practice.EuropeanPlanning Studies16(1):3-21.JohannFrick(2015).ContractualismandSocialRisk.Philosophy&PublicAffairs43(3):175-223.Tsogbadral Galaabaatar & Edi Karni (2013). SubjectiveExpected Utility With IncompletePreferences.Econometrica81(1):255–284.Allan Gibbard (2005). Truth and Correct Belief.PhilosophicalIssues15:338–350.GiveWell(2018).EstimatingtheFundingGapsforDistributionofAntimalarialNetsandSeasonalMalariaChemoprevention.Availableathttps://www.givewell.org/international/technical/programs/malaria-funding-gaps#Wh35
at\_would\_it\_cost\_to\_deliver\_nets\_to\_everyone\_who\_needed\_them.Accessed9Feb2021.GiveWell(2020a).GiveWell’sCost-EffectivenessAnalyses.Availableathttps://www.givewell.org/how-we-work/our-criteria/cost-effectiveness/cost-effectiveness-models. Accessed 26 January 2021.GiveWell(2020b).OurCriteriaforTopCharities.Availableathttps://www.givewell.org/how-we-work/criteria#Criteria.Accessed 28 May 2021.ChristianGollier(2013).PricingthePlanet’sFuture:TheEconomicsofDiscountinginanUncertain World. Princeton, NJ: Princeton UniversityPress.IrvingJ.Good(1966).SpeculationsConcerningtheFirstUltraintelligentMachine.InFranzL. Alt and Morris Rubinoff (eds.),Advances in Computers6. Academic Press, 31–88.KatjaGrace,JohnSalvatier,AllenDafoe,BaobaoZhang,andOwainEvans(2018).WhenWillAIExceedHumanPerformance?EvidencefromAIExperts.JournalofArtificialIntelligence Research62:729-754.PeterA.Graham(2010).InDefenseofObjectivismaboutMoralObligation.Ethics121:88-115.Hilary Greaves (2016). Cluelessness.Proceedings ofthe Aristotelian Society116(3):311-339.HilaryGreaves(2017)DiscountingforPublicPolicy:ASurvey.EconomicsandPhilosophy33(3):391–439.Roy F. Harrod (1948).Towards a Dynamic Economics.London: Macmillan.Stephen Hawking (2018).Brief Answers to the Big Questions.London: John Murray Press.DavidHeyd(1988).Procreationandvalue:Canethicsdealwithfuturityproblems?Philosophia18(2-3):151–170.BradHooker(2009).TheDemandingnessObjection.InTimothyChappell(ed.),TheProblem of Moral Demandingness. London: Palgrave McMillan.HouseofCommonsPublicAdministrationSelectCommittee(2013).TheroleoftheCharityCommissionand“publicbenefit”:Post-legislativescrutinyoftheCharitiesAct2006.ThirdReportofSession2013–14.VolumeI:Report,togetherwithformalminutes,oralandwrittenevidence.Availableathttps://publications.parliament.uk/pa/cm201314/cmselect/cmpubadm/76/7602.htmFrances Howard-Snyder (2005). It’s the Thought thatCounts.Utilitas17:265–281.WalterIsaacson(2003).BenjaminFranklin:AnAmericanLife.NewYork,NY:Simon&Schuster.ÒscarJordà,KatharinaKnoll,DmitryKuvshinov,MoritzSchularick,andAlanMTaylor(2019).TheRateofReturnonEverything,1870–2015.TheQuarterlyJournalofEconomics134(4):1225–1298.ShellyKagan(1984).DoesConsequentialismDemandTooMuch?RecentWorkontheLimits of Obligation.Philosophy & Public Affairs13(3):239–254.36
DanielKahnemanandDanLovallo(1993).TimidChoicesandBoldForecasts:ACognitivePerspective on Risk Taking.Management Science39(1):17–31.DanielKahnemanandAmosTversky(1979).ProspectTheory:AnAnalysisofDecisionunder Risk.Econometrica47(2):263–291.ImmanuelKant(1996).OnaSupposedRighttoLiefromPhilanthropy.InMaryGregor(transl./ed.)ImmanuelKant:PracticalPhilosophy.NewYork:CambridgeUniversityPress.DanielKokotajloandAlexandraOprea(2020).CounterproductiveAltruism:TheOtherHeavy Tail.Philosophical Perspectives34:134-163.Shane Legg (2008).Machine Super Intelligence. PhDthesis, University of Lugano.JamesLenman(2000).ConsequentialismandCluelessness.Philosophy&PublicAffairs29(4):342–370.DavidLewis(1980).MadpainandMartianpain.ReadingsinthePhilosophyofPsychology1: 216-222.GregoryLewis(2016).Bewaresurprisingandsuspiciousconvergence.TheEffectiveAltruismForum.Availableathttps://forum.effectivealtruism.org/posts/omoZDu8ScNbot6kXS/beware-surprising-and-suspicious-convergenceS.MatthewLiao(2020).TheMoralStatusandRightsofArtificialIntelligence.InS.MatthewLiao(ed.),EthicsofArtificialIntelligence.NewYork:OxfordUniversityPress,480–498.WilliamMacAskill(2014).Replaceability,CareerChoice,andMakingaDifference.EthicalTheory and Moral Practice17:269–283.WilliamMacAskill(2019).Whenshouldaneffectivealtruistdonate?GPIWorkingPaper8-2019.https://globalprioritiesinstitute.org/william-macaskill-when-should-an-effective-altruist-donate/William MacAskill (MS).What We Owe The Future. BasicBooks.WilliamMacAskill,AndreasMogensen,andTobyOrd(2018).GivingIsn’tDemanding.InPaulWoodruff(ed.),TheEthicsofGiving:Philosophers'PerspectivesonPhilanthropy. Oxford: Oxford University Press, 178–203.WilliamMacAskill,AndreasMogensen,TerujiThomasandHilaryGreaves(MS).Difference-making consequentialism.Mark J. Machina and Marciano Siniscalchi (2014). Ambiguityand ambiguity aversion. InHandbook of the Economics of Risk and UncertaintyVol. 1. Amsterdam: NorthHolland, 729–807.ElinorMason(2013).ObjectivismandProspectivismAboutRightness.JournalofEthicsandSocial Philosophy7(2):1–21.37
ChristopherMeacham(2012).Person-affectingviewsandsaturatingcounterpartrelations.Philosophical Studies158(2):257–287.FankangMengandTomEllis(2020).Theseconddecadeofsyntheticbiology:2010–2020.Nature Communications11:5174.PiersMillettandAndrewSnyder-Beattie(2017).ExistentialRiskandCost-EffectiveBiosecurity.Health Security15(4):373–383.Andreas Mogensen (2019). The Only Ethical Argumentfor Positive Delta?GPI WorkingPaper5-2019.https://globalprioritiesinstitute.org/andreas-mogensen-the-only-ethical-argument-for-positive-delta-2/Andreas Mogensen (2020). Moral Demands and the FarFuture.Philosophy andPhenomenological Research. https://doi.org/10.1111/phpr.12729Andreas Mogensen, William MacAskill, and Hilary Greaves(MS). Risk Aversion, AmbiguityAversion, and Longtermism.G.E. Moore (1903).Principia Ethica.Cambridge: CambridgeUniversity Press.Tim Mulgan (2001).The Demands of Consequentialism.Oxford: Oxford University Press.ElonMusk(2014).One-on-onewithElonMusk.TalkattheMITAeroAstro1914-2014CentennialSymposium.Availableathttps://aeroastro.mit.edu/videos/centennial-symposium-one-one-one-elon-muskThomasNagel(1978).Ruthlessnessandpubliclife.InStuartHampshire(ed.),PublicandPrivate Morality. Cambridge: Cambridge UniversityPress.TobyNewberry(2021a).Howmanylivesdoesthefuturehold?GlobalPrioritiesInstituteTechnicalReportT2-2021.https://globalprioritiesinstitute.org/how-many-lives-does-the-future-hold-toby-newberry-future-of-humanity-institute-university-of-oxford/TobyNewberry(2021b).Howcost-effectiveareeffortstodetectnear-Earth-objects?GlobalPrioritiesInstituteTechnicalReportT1-2021.https://globalprioritiesinstitute.org/how-cost-effective-are-efforts-to-detect-near-earth-objects-toby-newberry-future-of-humanity-institute-university-of-oxford/RichardNgo(2020).AGISafetyfromFirstPrinciples.https://drive.google.com/file/d/1uK7NhdSKprQKZnRjU58X7NLA1auXlWHt/viewTedO'DonoghueandJasonSomerville(2018).ModelingRiskAversioninEconomics.TheJournal of Economic Perspectives32(2):91-114.AlyssaOchs(2019).WhatIstheOldestCharitableTrustintheU.S.,andWhatDoesItFund?InsidePhilanthropy.Availableathttps://www.insidephilanthropy.com/home/2019/4/21/what-is-the-oldest-charitable-trust-in-the-us-and-what-does-it-fundKristianOlsen(2011).ADefenseoftheObjective/SubjectiveMoralOughtDistinction.TheJournal of Ethics21(4):351–373.38
OpenPhilanthropyProject(2020).GrantsDatabase.Availableathttps://www.openphilanthropy.org/giving/grants. Accessed26 January 2021.TobyOrd(2019).TheMoralImperativeTowardCost-EffectivenessinGlobalHealth.InHilaryGreavesandTheronPummer(eds.),EffectiveAltruism:PhilosophicalIssues.Oxford: Oxford University Press.TobyOrd(2020).ThePrecipice:ExistentialRiskandtheFutureofHumanity.London:Bloomsbury.Derek Parfit (1984).Reasons and Persons.Oxford:Clarendon Press.DerekParfit(2011).OnWhatMatters.Vol.1.Ed.SamuelScheffler.Oxford:OxfordUniversity Press.PaulPierson(2000).IncreasingReturns,PathDependence,andtheStudyofPolitics.TheAmerican Political Science Review94(2):251-267.Arthur C. Pigou (1932).The Economics of Welfare.Vol. 1 (4th edn). London: Macmillan.DouglasW.Portmore(2011).CommonsenseConsequentialism:WhereinMoralityMeetsRationality. Oxford: Oxford University Press.Richard Posner (2004).Catastrophe: Risk and Response.Oxford: Oxford University Press.JohnW.Pratt(1964).RiskAversionintheSmallandintheLarge.Econometrica32(1–2):122–136.H.A.Prichard(1932)[2002].DutyandIgnoranceofFact.InJimMcAdam(ed.),MoralWritings. Oxford: Oxford University Press, 85–110.JohnQuiggin(1982).ATheoryofAnticipatedUtility.JournalofEconomicBehaviorandOrganization3:323–343.JohnQuigginandPeterWakker(1994).TheAxiomaticBasisofAnticipatedUtility:AClarification.Journal of Economic Theory64:486–499.MatthewRabin(2000).RiskAversionandExpected-UtilityTheory:ACalibrationTheorem.Econometrica68(5):1281–92.FrankP.Ramsey(1928).Amathematicaltheoryofsaving.EconomicJournal38:543-559.(ReprintedinFrank.P.Ramsey,Foundations:EssaysinPhilosophy,Logic,Mathematics, and Economics, ed. D. H. Mellor.)MartinRees(2018).OntheFuture:ProspectsforHumanity.Princeton,NJ:PrincetonUniversity Press.JacyReese(2018).WhyIprioritizemoralcircleexpansionoverartificialintelligencealignment.EffectiveAltruismForum.Availableathttps://forum.effectivealtruism.org/posts/BY8gXSpGijypbGitT/why-i-prioritize-moral-circle-expansion-over-artificial. Accessed 9 February2021.Jess Riedel (MS). Value Lock-In Feasibility.39
MaxRoser,HannahRitchieandEstebanOrtiz-Ospina(2013).WorldPopulationGrowth.OurWorld in Data.Available at https://ourworldindata.org/world-population-growth.W.D. Ross (1930).The Right and the Good. Oxford:Oxford University Press.W.D. Ross (1939).The Foundations of Ethics. Oxford:Clarendon Press.Thomas Rowe and Alex Voorhoeve (2018). Egalitarianismunder Severe Uncertainty.Philosophy & Public Affairs, 46(3):239–268.BertrandRussell(1966).TheElementsofEthics.InPhilosophicalEssays.NewYork:Simonand Schuster, 13–59.StuartRussell(2019).HumanCompatible:ArtificialIntelligenceandtheProblemofControl.New York: Viking.Inge-JulianaSackmann,ArnoldI.BoothroydandKathleenE.Kraemer(1993).OurSun.III.Present and Future.The Astrophysics Journal418:457–468.AndersSandberg(2013).Feasibilityofwholebrainemulation.InVincentC.Müller(ed.),Theory and Philosophy of Artificial Intelligence.Berlin: Springer, 251–64.AndersSandberg(2014).Ethicsofbrainemulations.JournalofExperimentalandTheoretical Artificial Intelligence26(3):439–457.AndersSandbergandNickBostrom(2008).GlobalCatastrophicRisksSurvey.FutureofHumanityInstituteTechnicalReport#2008-1.Availableathttps://www.fhi.ox.ac.uk/reports/2008-1.pdf.AndersSandbergandNickBostrom(2008).WholeBrainEmulation:ARoadmap.FutureofHumanityInstituteTechnicalReport#2008‐3.Availableathttps://www.fhi.ox.ac.uk/reports/2008-3.pdf.IoanSarbuandCalinSebarchievici(2017).SolarHeatingandCoolingSystems:Fundamentals, Experiments and Applications.London:Elsevier.T.M.Scanlon(1998).WhatWeOwetoEachOther.Cambridge,Mass.:HarvardUniversityPress.Klaus-PeterSchröderandRobertConnonSmith(2008).DistantfutureoftheSunandEarthrevisited.Monthly Notices of the Royal AstronomicalSociety386(1):155-163.Kieran Setiya (2014). The Ethics of Existence.PhilosophicalPerspectives28(1), 291–301.CarlShulman(2020).Whatdohistoricalstatisticsteachusabouttheaccidentalreleaseofpandemicbioweapons?ReflectiveDisequilibrium.Availableathttp://reflectivedisequilibrium.blogspot.com/2020/10/what-do-historical-statistics-teach-us.html.Henry Sidgwick. (1890).The Methods of Ethics. London:Macmillan.JulianSimon(1998).TheUltimateResource2(Rev.ed.).Princeton:PrincetonUniversityPress.40
PeterSinger(1972).Famine,AffluenceandMorality.Philosophy&PublicAffairs1(3):229–243.AndrewSnyder-Beattie,TobyOrdandMichaelB.Bonsall(2019).Anupperboundforthebackground rate of human extinction.Scientific Reports9:11054.RobertM.Solow(1974).Theeconomicsofresourcesortheresourcesofeconomics.American Economic Review Papers and Proceedings64:1–14.KajSotalaandLukasGloor(2017).SuperintelligenceasaCauseorCureforRisksofAstronomical Suffering.Informatica41:501–505.NickStatt(2015).BillGatesIsWorriedaboutArtificialIntelligenceToo.CNET,28January2015.https://www.cnet.com/news/bill-gates-is-worried-about-artificial-intelligence-too/.H. Orri Stefánsson and Richard Bradley (2019). WhatIs Risk Aversion?The British Journalfor the Philosophy of Science70(1):77–102.H.OrriStefánssonandKatieSteele(forthcoming).BeyondUncertainty:ReasoningwithUnknown Possibilities. Cambridge: Cambridge UniversityPress.NicholasStern(2007).TheEconomicsofClimateChange.Cambridge:CambridgeUniversity Press.Michael Stix (2002).The Sun: An Introduction. Berlin:Springer.ChristianTarsney(2019).TheEpistemicChallengetoLongtermism.GPIWorkingPaper10-2019.https://globalprioritiesinstitute.org/christian-tarsney-the-epistemic-challenge-to-longtermism/MaxTegmark(2017).Life3.0:BeingHumanintheAgeofArtificialIntelligence.NewYork:Knopf.TexasEducationAgency(2020).TexasPermanentSchoolFund.Availableathttps://tea.texas.gov/finance-and-grants/texas-permanent-school-fund.TerujiThomas(2019).TheAsymmetryandtheLongTerm.GPIWorkingPaper11-2019.https://globalprioritiesinstitute.org/teruji-thomas-the-asymmetry-uncertainty-and-the-long-term/JudithJarvisThomson(1986).ImposingRisks.InRights,Restitution,andRisks,editedbyWilliam Parent. Cambridge, MA: Harvard UniversityPress, 173-91.Judith Jarvis Thomson (2008).Normativity.Chicago:Open Court.PhilipTrammell(2020).PatienceandPhilanthropy.https://philiptrammell.com/static/PatienceAndPhilanthropy.pdfStefan T. Trautmann and Gijs van de Kuilen (2015).Ambiguity Attitudes. In Gideon Keren &George Wu (Eds.),The Wiley Blackwell Handbook ofJudgment and Decision Making,I. Chichester: Wiley, 89-116.Alan Turing (1951). Can digital computers think?BBCRadioMay 15, 1951.41
UnitedNations,DepartmentofEconomicandSocialAffairs,PopulationDivision(2019).WorldPopulationProspects2019:Highlights.https://population.un.org/wpp/Publications/Files/WPP2019\_Highlights.pdfUniversityofTexasSystem(2021).ThePermanentUniversityFund(PUF).Availableathttps://www.utsystem.edu/puf.JeroenC.J.M.VanDenBerghandPietRietveld(2004).ReconsideringtheLimitstoWorldPopulation: Meta-analysis and Meta-prediction.BioScience54(3):195-204.Alex Voorhoeve (2014). How Should We Aggregate CompetingClaims?Ethics125: 64-87.EricWolfandOwenBrianToon(2015).Theevolutionofhabitableclimatesunderthebrightening Sun.Journal of Geophysical Research:Atmospheres120:5775–5794.EliezerYudkowsky(2013).IntelligenceExplosionMicroeconomics.TechnicalReport2013-1. Berkeley, CA: Machine Intelligence ResearchInstitute.MichaelJ.Zimmerman(2006).IsMoralObligationObjectiveorSubjective?Utilitas18:329-61.MichaelJ.Zimmerman(2008).LivingwithUncertainty:TheMoralSignificanceofIgnorance.Cambridge: Cambridge University Press.
42